English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

In this paper we study several classes of stochastic optimization algorithmsenriched with heavy ball momentum. Among the methods studied are: stochasticgradient descent, stochastic Newton, stochastic proximal point and stochasticdual subspace ascent. This is the first time momentum variants of several ofthese methods are studied. We choose to perform our analysis in a setting inwhich all of the above methods are equivalent. We prove global nonassymptoticlinear convergence rates for all methods and various measures of success,including primal function values, primal iterates (in L2 sense), and dualfunction values. We also show that the primal iterates converge at anaccelerated linear rate in the L1 sense. This is the first time a linear rateis shown for the stochastic heavy ball method (i.e., stochastic gradientdescent method with momentum). Under somewhat weaker conditions, we establish asublinear convergence rate for Cesaro averages of primal iterates. Moreover, wepropose a novel concept, which we call stochastic momentum, aimed at decreasingthe cost of performing the momentum step. We prove linear convergence ofseveral stochastic methods with stochastic momentum, and show that in somesparse data regimes and for sufficiently small momentum parameters, thesemethods enjoy better overall complexity than methods with deterministicmomentum. Finally, we perform extensive numerical testing on artificial andreal datasets, including data coming from average consensus problems.

Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methods