English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Adaptive gradient methods such as AdaGrad and its variants update thestepsize in stochastic gradient descent on the fly according to the gradientsreceived along the way; such methods have gained widespread use in large-scaleoptimization for their ability to converge robustly, without the need tofine-tune the stepsize schedule. Yet, the theoretical guarantees to date forAdaGrad are for online and convex optimization. We bridge this gap by providingtheoretical guarantees for the convergence of AdaGrad for smooth, nonconvexfunctions. We show that the norm version of AdaGrad (AdaGrad-Norm) converges toa stationary point at the $\mathcal{O}(\log(N)/\sqrt{N})$ rate in thestochastic setting, and at the optimal $\mathcal{O}(1/N)$ rate in the batch(non-stochastic) setting -- in this sense, our convergence guarantees are'sharp'. In particular, the convergence of AdaGrad-Norm is robust to the choiceof all hyper-parameters of the algorithm, in contrast to stochastic gradientdescent whose convergence depends crucially on tuning the step-size to the(generally unknown) Lipschitz smoothness constant and level of stochastic noiseon the gradient. Extensive numerical experiments are provided to corroborateour theory; moreover, the experiments suggest that the robustness ofAdaGrad-Norm extends to state-of-the-art models in deep learning, withoutsacrificing generalization.

AdaGrad stepsizes: Sharp convergence over nonconvex landscapes