English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Among the very first variance reduced stochastic methods for solving theempirical risk minimization problem was the SVRG method (Johnson & Zhang 2013).SVRG is an inner-outer loop based method, where in the outer loop a referencefull gradient is evaluated, after which $m \in \mathbb{N}$ steps of an innerloop are executed where the reference gradient is used to build a variancereduced estimate of the current gradient. The simplicity of the SVRG method andits analysis have led to multiple extensions and variants for even non-convexoptimization. We provide a more general analysis of SVRG than had beenpreviously done by using arbitrary sampling, which allows us to analysevirtually all forms of mini-batching through a single theorem. Furthermore, ouranalysis is focused on more practical variants of SVRG including a new variantof the loopless SVRG (Hofman et al 2015, Kovalev et al 2019, Kulunchakov andMairal 2019) and a variant of k-SVRG (Raj and Stich 2018) where $m=n$ and where$n$ is the number of data points. Since our setup and analysis reflect what isdone in practice, we are able to set the parameters such as the mini-batch sizeand step size using our theory in such a way that produces a more efficientalgorithm in practice, as we show in extensive numerical experiments.

Towards closing the gap between the theory and practice of SVRG