English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

For finite-sum optimization, variance-reduced gradient methods (VR) computeat each iteration the gradient of a single function (or of a mini-batch), andyet achieve faster convergence than SGD thanks to a carefully craftedlower-variance stochastic gradient estimator that reuses past gradients.Another important line of research of the past decade in continuousoptimization is the adaptive algorithms such as AdaGrad, that dynamicallyadjust the (possibly coordinate-wise) learning rate to past gradients andthereby adapt to the geometry of the objective function. Variants such asRMSprop and Adam demonstrate outstanding practical performance that havecontributed to the success of deep learning. In this work, we present AdaVR,which combines the AdaGrad algorithm with variance-reduced gradient estimatorssuch as SAGA or L-SVRG. We assess that AdaVR inherits both good convergenceproperties from VR methods and the adaptive nature of AdaGrad: in the case of$L$-smooth convex functions we establish a gradient complexity of$O(n+(L+\sqrt{nL})/\varepsilon)$ without prior knowledge of $L$. Numericalexperiments demonstrate the superiority of AdaVR over state-of-the-art methods.Moreover, we empirically show that the RMSprop and Adam algorithm combined withvariance-reduced gradients estimators achieve even faster convergence.

Finite-sum optimization: Adaptivity to smoothness and loopless variance reduction