English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we propose a simple variant of the original stochasticvariance reduction gradient (SVRG), where hereafter we refer to as the variancereduced stochastic gradient descent (VR-SGD). Different from the choices of thesnapshot point and starting point in SVRG and its proximal variant, Prox-SVRG,the two vectors of each epoch in VR-SGD are set to the average and last iterateof the previous epoch, respectively. This setting allows us to use much largerlearning rates or step sizes than SVRG, e.g., 3/(7L) for VR-SGD vs 1/(10L) forSVRG, and also makes our convergence analysis more challenging. In fact, alarger learning rate enjoyed by VR-SGD means that the variance of itsstochastic gradient estimator asymptotically approaches zero more rapidly.Unlike common stochastic methods such as SVRG and proximal stochastic methodssuch as Prox-SVRG, we design two different update rules for smooth andnon-smooth objective functions, respectively. In other words, VR-SGD can tacklenon-smooth and/or non-strongly convex problems directly without using anyreduction techniques such as quadratic regularizers. Moreover, we analyze theconvergence properties of VR-SGD for strongly convex problems, which show thatVR-SGD attains a linear convergence rate. We also provide the convergenceguarantees of VR-SGD for non-strongly convex problems. Experimental resultsshow that the performance of VR-SGD is significantly better than itscounterparts, SVRG and Prox-SVRG, and it is also much better than the bestknown stochastic method, Katyusha.

Larger is Better: The Effect of Learning Rates Enjoyed by Stochastic Optimization with Progressive Variance Reduction