English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The exploding and vanishing gradient problem has been the major conceptualprinciple behind most architecture and training improvements in recurrentneural networks (RNNs) during the last decade. In this paper, we argue thatthis principle, while powerful, might need some refinement to explain recentdevelopments. We refine the concept of exploding gradients by reformulating theproblem in terms of the cost function smoothness, which gives insight intohigher-order derivatives and the existence of regions with many close localminima. We also clarify the distinction between vanishing gradients and theneed for the RNN to learn attractors to fully use its expressive power. Throughthe lens of these refinements, we shed new light on recent developments in theRNN field, namely stable RNN and unitary (or orthogonal) RNNs.

Beyond exploding and vanishing gradients: analysing RNN training using attractors and smoothness