English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study discrete time dynamical systems governed by the state equation$h_{t+1}=\phi(Ah_t+Bu_t)$. Here $A,B$ are weight matrices, $\phi$ is anactivation function, and $u_t$ is the input data. This relation is the backboneof recurrent neural networks (e.g. LSTMs) which have broad applications insequential learning tasks. We utilize stochastic gradient descent to learn theweight matrices from a finite input/state trajectory $(u_t,h_t)_{t=0}^N$. Weprove that SGD estimate linearly converges to the ground truth weights whileusing near-optimal sample size. Our results apply to increasing activationswhose derivatives are bounded away from zero. The analysis is based on i) anovel SGD convergence result with nonlinear activations and ii) carefulstatistical characterization of the state vector. Numerical experiments verifythe fast convergence of SGD on ReLU and leaky ReLU in consistence with ourtheory.

Stochastic Gradient Descent Learns State Equations with Nonlinear Activations