English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vanishing and exploding gradients are two of the main obstacles in trainingdeep neural networks, especially in capturing long range dependencies inrecurrent neural networks~(RNNs). In this paper, we present an efficientparametrization of the transition matrix of an RNN that allows us to stabilizethe gradients that arise in its training. Specifically, we parameterize thetransition matrix by its singular value decomposition(SVD), which allows us toexplicitly track and control its singular values. We attain efficiency by usingtools that are common in numerical linear algebra, namely Householderreflectors for representing the orthogonal matrices that arise in the SVD. Byexplicitly controlling the singular values, our proposed Spectral-RNN methodallows us to easily solve the exploding gradient problem and we observe that itempirically solves the vanishing gradient issue to a large extent. We note thatthe SVD parameterization can be used for any rectangular weight matrix, henceit can be easily extended to any deep neural network, such as a multi-layerperceptron. Theoretically, we demonstrate that our parameterization does notlose any expressive power, and show how it controls generalization of RNN forthe classification task. %, and show how it potentially makes the optimizationprocess easier. Our extensive experimental results also demonstrate that theproposed framework converges faster, and has good generalization, especially incapturing long range dependencies, as shown on the synthetic addition and copytasks, as well as on MNIST and Penn Tree Bank data sets.

Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization