English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The classical analysis of Stochastic Gradient Descent (SGD) with polynomiallydecaying stepsize $\eta_t = \eta/\sqrt{t}$ relies on well-tuned $\eta$depending on problem parameters such as Lipschitz smoothness constant, which isoften unknown in practice. In this work, we prove that SGD with arbitrary $\eta> 0$, referred to as untuned SGD, still attains an order-optimal convergencerate $\widetilde{O}(T^{-1/4})$ in terms of gradient norm for minimizing smoothobjectives. Unfortunately, it comes at the expense of a catastrophicexponential dependence on the smoothness constant, which we show is unavoidablefor this scheme even in the noiseless setting. We then examine three familiesof adaptive methods $\unicode{x2013}$ Normalized SGD (NSGD), AMSGrad, andAdaGrad $\unicode{x2013}$ unveiling their power in preventing such exponentialdependency in the absence of information about the smoothness parameter andboundedness of stochastic gradients. Our results provide theoreticaljustification for the advantage of adaptive methods over untuned SGD inalleviating the issue with large gradients.

Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods