English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In this paper, we study the stochastic optimization problem from acontinuous-time perspective. We propose a stochastic first-order algorithm,called Stochastic Gradient Descent with Momentum (SGDM), and show that thesequence $\{x_k\}$ generated by SGDM, despite its stochastic nature, convergesto a deterministic second-order Ordinary Differential Equation (ODE) in$L_2$-norm, as the stepsize goes to zero. The connection between the ODE andthe algorithm results in delightful patterns in the discrete-time convergenceanalysis. More specifically, we develop convergence results for the ODE througha Lyapunov function, and translate the whole argument to the discrete-timecase. This approach yields an $\widetilde{\mathcal{O}}\left(\frac{\log\frac{1}{\beta}}{\sqrt{k}}\right)$ anytime last-iterate convergence guaranteefor SGDM, where $k$ is the number of iterates and $1-\beta$ is the desiredsuccess probability. Notably, the Lyapunov argument helps us to remove both theprojection step and the bounded gradient assumption, and our algorithm does notrely on quantities that cannot be known in general. Additionally, we prove thata subsequence of $\{x_k\}$ generated by SGDM converges in expectation at rate$o\left(\frac{1}{\sqrt{k}\log\log k}\right)$. To the best of our knowledge,both of these results, enabled by the continuous-time perspective, improveexisting works in the field.

The Last Iterate Convergence of Stochastic Gradient Descent with Momentum: From a Continuous-Time Perspective