English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recurrent neural networks have gained widespread use in modeling sequencedata across various domains. While many successful recurrent architecturesemploy a notion of gating, the exact mechanism that enables such remarkableperformance is not well understood. We develop a theory for signal propagationin recurrent networks after random initialization using a combination of meanfield theory and random matrix theory. To simplify our discussion, we introducea new RNN cell with a simple gating mechanism that we call the minimalRNN andcompare it with vanilla RNNs. Our theory allows us to define a maximumtimescale over which RNNs can remember an input. We show that this theorypredicts trainability for both recurrent architectures. We show that gatedrecurrent networks feature a much broader, more robust, trainable region thanvanilla RNNs, which corroborates recent experimental findings. Finally, wedevelop a closed-form critical initialization scheme that achieves dynamicalisometry in both vanilla RNNs and minimalRNNs. We show that this results insignificantly improvement in training dynamics. Finally, we demonstrate thatthe minimalRNN achieves comparable performance to its more complexcounterparts, such as LSTMs or GRUs, on a language modeling task.

Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks