English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recurrent neural networks with a gating mechanism such as an LSTM or GRU arepowerful tools to model sequential data. In the mechanism, a forget gate, whichwas introduced to control information flow in a hidden state in the RNN, hasrecently been re-interpreted as a representative of the time scale of thestate, i.e., a measure how long the RNN retains information on inputs. On thebasis of this interpretation, several parameter initialization methods toexploit prior knowledge on temporal dependencies in data have been proposed toimprove learnability. However, the interpretation relies on various unrealisticassumptions, such as that there are no inputs after a certain time point. Inthis work, we reconsider this interpretation of the forget gate in a morerealistic setting. We first generalize the existing theory on gated RNNs sothat we can consider the case where inputs are successively given. We thenargue that the interpretation of a forget gate as a temporal representation isvalid when the gradient of loss with respect to the state decreasesexponentially as time goes back. We empirically demonstrate that existing RNNssatisfy this gradient condition at the initial training phase on several tasks,which is in good agreement with previous initialization methods. On the basisof this finding, we propose an approach to construct new RNNs that canrepresent a longer time scale than conventional models, which will improve thelearnability for long-term sequential data. We verify the effectiveness of ourmethod by experiments with real-world datasets.

Recurrent Neural Networks for Learning Long-term Temporal Dependencies with Reanalysis of Time Scale Representation