English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Adaptive gradient methods have attracted much attention of machine learningcommunities due to the high efficiency. However their acceleration effect inpractice, especially in neural network training, is hard to analyze,theoretically. The huge gap between theoretical convergence results andpractical performances prevents further understanding of existing optimizersand the development of more advanced optimization methods. In this paper, weprovide adaptive gradient methods a novel analysis with an additional mildassumption, and revise AdaGrad to \radagrad for matching a better provableconvergence rate. To find an $\epsilon$-approximate first-order stationarypoint in non-convex objectives, we prove random shuffling \radagrad achieves a$\tilde{O}(T^{-1/2})$ convergence rate, which is significantly improved byfactors $\tilde{O}(T^{-1/4})$ and $\tilde{O}(T^{-1/6})$ compared with existingadaptive gradient methods and random shuffling SGD, respectively. To the bestof our knowledge, it is the first time to demonstrate that adaptive gradientmethods can deterministically be faster than SGD after finite epochs.Furthermore, we conduct comprehensive experiments to validate the additionalmild assumption and the acceleration effect benefited from second moments andrandom shuffling.

Adaptive Gradient Methods Can Be Provably Faster than SGD after Finite Epochs