English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The adaptive stochastic gradient descent (SGD) with momentum has been widelyadopted in deep learning as well as convex optimization. In practice, the lastiterate is commonly used as the final solution to make decisions. However, theavailable regret analysis and the setting of constant momentum parameters onlyguarantee the optimal convergence of the averaged solution. In this paper, wefill this theory-practice gap by investigating the convergence of the lastiterate (referred to as individual convergence), which is a more difficult taskthan convergence analysis of the averaged solution. Specifically, in theconstrained convex cases, we prove that the adaptive Polyak's Heavy-ball (HB)method, in which only the step size is updated using the exponential movingaverage strategy, attains an optimal individual convergence rate of$O(\frac{1}{\sqrt{t}})$, as opposed to the optimality of $O(\frac{\log t}{\sqrt{t}})$ of SGD, where $t$ is the number of iterations. Our new analysis not onlyshows how the HB momentum and its time-varying weight help us to achieve theacceleration in convex optimization but also gives valuable hints how themomentum parameters should be scheduled in deep learning. Empirical results onoptimizing convex functions and training deep networks validate the correctnessof our convergence analysis and demonstrate the improved performance of theadaptive HB methods.

The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methods