English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Momentum methods, including heavy-ball~(HB) and Nesterov's acceleratedgradient~(NAG), are widely used in training neural networks for their fastconvergence. However, there is a lack of theoretical guarantees for theirconvergence and acceleration since the optimization landscape of the neuralnetwork is non-convex. Nowadays, some works make progress towards understandingthe convergence of momentum methods in an over-parameterized regime, where thenumber of the parameters exceeds that of the training instances. Nonetheless,current results mainly focus on the two-layer neural network, which are farfrom explaining the remarkable success of the momentum methods in training deepneural networks. Motivated by this, we investigate the convergence of NAG withconstant learning rate and momentum parameter in training two architectures ofdeep linear networks: deep fully-connected linear neural networks and deeplinear ResNets. Based on the over-parameterization regime, we first analyze theresidual dynamics induced by the training trajectory of NAG for a deepfully-connected linear neural network under the random Gaussian initialization.Our results show that NAG can converge to the global minimum at a $(1 -\mathcal{O}(1/\sqrt{\kappa}))^t$ rate, where $t$ is the iteration number and$\kappa > 1$ is a constant depending on the condition number of the featurematrix. Compared to the $(1 - \mathcal{O}(1/{\kappa}))^t$ rate of GD, NAGachieves an acceleration over GD. To the best of our knowledge, this is thefirst theoretical guarantee for the convergence of NAG to the global minimum intraining deep neural networks. Furthermore, we extend our analysis to deeplinear ResNets and derive a similar convergence result.

A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networks