English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Due to its simplicity and efficiency, the first-order gradient method hasbeen extensively employed in training neural networks. Although theoptimization problem of the neural network is non-convex, recent research hasproved that the first-order method is capable of attaining a global minimumduring training over-parameterized neural networks, where the number ofparameters is significantly larger than that of training instances. Momentummethods, including the heavy ball (HB) method and Nesterov's acceleratedgradient (NAG) method, are the workhorse of first-order gradient methods owningto their accelerated convergence. In practice, NAG often exhibits superiorperformance than HB. However, current theoretical works fail to distinguishtheir convergence difference in training neural networks. To fill this gap, weconsider the training problem of the two-layer ReLU neural network underover-parameterization and random initialization. Leveraging high-resolutiondynamical systems and neural tangent kernel (NTK) theory, our result not onlyestablishes tighter upper bounds of the convergence rate for both HB and NAG,but also provides the first theoretical guarantee for the acceleration of NAGover HB in training neural networks. Finally, we validate our theoreticalresults on three benchmark datasets.

Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks