English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Training recurrent neural networks (RNNs) on long sequence tasks is plaguedwith difficulties arising from the exponential explosion or vanishing ofsignals as they propagate forward or backward through the network. Manytechniques have been proposed to ameliorate these issues, including variousalgorithmic and architectural modifications. Two of the most successful RNNarchitectures, the LSTM and the GRU, do exhibit modest improvements overvanilla RNN cells, but they still suffer from instabilities when trained onvery long sequences. In this work, we develop a mean field theory of signalpropagation in LSTMs and GRUs that enables us to calculate the time scales forsignal propagation as well as the spectral properties of the state-to-stateJacobians. By optimizing these quantities in terms of the initializationhyperparameters, we derive a novel initialization scheme that eliminates orreduces training instabilities. We demonstrate the efficacy of ourinitialization scheme on multiple sequence tasks, on which it enablessuccessful training while a standard initialization either fails completely oris orders of magnitude slower. We also observe a beneficial effect ongeneralization performance using this new initialization.

Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs