English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Heavy-ball momentum with decaying learning rates is widely used with SGD foroptimizing deep learning models. In contrast to its empirical popularity, theunderstanding of its theoretical property is still quite limited, especiallyunder the standard anisotropic gradient noise condition for quadraticregression problems. Although it is widely conjectured that heavy-ball momentummethod can provide accelerated convergence and should work well in large batchsettings, there is no rigorous theoretical analysis. In this paper, we fillthis theoretical gap by establishing a non-asymptotic convergence bound forstochastic heavy-ball methods with step decay scheduler on quadraticobjectives, under the anisotropic gradient noise condition. As a directimplication, we show that heavy-ball momentum can provide$\tilde{\mathcal{O}}(\sqrt{\kappa})$ accelerated convergence of the bias termof SGD while still achieving near-optimal convergence rate with respect to thestochastic variance term. The combined effect implies an overall convergencerate within log factors from the statistical minimax rate. This means SGD withheavy-ball momentum is useful in the large-batch settings such as distributedmachine learning or federated learning, where a smaller number of iterationscan significantly reduce the number of communication rounds, leading toacceleration in practice.

Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise