English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Stagewise training strategy is widely used for learning neural networks,which runs a stochastic algorithm (e.g., SGD) starting with a relatively largestep size (aka learning rate) and geometrically decreasing the step size aftera number of iterations. It has been observed that the stagewise SGD has muchfaster convergence than the vanilla SGD with a polynomially decaying step sizein terms of both training error and testing error. {\it But how to explain thisphenomenon has been largely ignored by existing studies.} This paper providessome theoretical evidence for explaining this faster convergence. Inparticular, we consider a stagewise training strategy for minimizing empiricalrisk that satisfies the Polyak-\L ojasiewicz (PL) condition, which has beenobserved/proved for neural networks and also holds for a broad family of convexfunctions. For convex loss functions and two classes of "nice-behaviored"non-convex objectives that are close to a convex function, we establish fasterconvergence of stagewise training than the vanilla SGD under the PL conditionon both training error and testing error. Experiments on stagewise learning ofdeep residual networks exhibits that it satisfies one type of non-convexityassumption and therefore can be explained by our theory. Of independentinterest, the testing error bounds for the considered non-convex loss functionsare dimensionality and norm independent.

Stagewise Training Accelerates Convergence of Testing Error Over SGD