English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study the application of variance reduction (VR) techniques to generalnon-convex stochastic optimization problems. In this setting, the recent workSTORM [Cutkosky-Orabona '19] overcomes the drawback of having to computegradients of "mega-batches" that earlier VR methods rely on. There, STORMutilizes recursive momentum to achieve the VR effect and is then later madefully adaptive in STORM+ [Levy et al., '21], where full-adaptivity removes therequirement for obtaining certain problem-specific parameters such as thesmoothness of the objective and bounds on the variance and norm of thestochastic gradients in order to set the step size. However, STORM+ cruciallyrelies on the assumption that the function values are bounded, excluding alarge class of useful functions. In this work, we propose META-STORM, ageneralized framework of STORM+ that removes this bounded function valuesassumption while still attaining the optimal convergence rate for non-convexoptimization. META-STORM not only maintains full-adaptivity, removing the needto obtain problem specific parameters, but also improves the convergence rate'sdependency on the problem parameters. Furthermore, META-STORM can utilize alarge range of parameter settings that subsumes previous methods allowing formore flexibility in a wider range of settings. Finally, we demonstrate theeffectiveness of META-STORM through experiments across common deep learningtasks. Our algorithm improves upon the previous work STORM+ and is competitivewith widely used algorithms after the addition of per-coordinate update andexponential moving average heuristics.

META-STORM: Generalized Fully-Adaptive Variance Reduced SGD for Unbounded Functions