English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Although stochastic gradient descent (SGD) method and its variants (e.g.,stochastic momentum methods, AdaGrad) are the choice of algorithms for solvingnon-convex problems (especially deep learning), there still remain big gapsbetween the theory and the practice with many questions unresolved. Forexample, there is still a lack of theories of convergence for SGD and itsvariants that use stagewise step size and return an averaged solution inpractice. In addition, theoretical insights of why adaptive step size ofAdaGrad could improve non-adaptive step size of {\sgd} is still missing fornon-convex optimization. This paper aims to address these questions and fillthe gap between theory and practice. We propose a universal stagewiseoptimization framework for a broad family of {\bf non-smooth non-convex}(namely weakly convex) problems with the following key features: (i) at eachstage any suitable stochastic convex optimization algorithms (e.g., SGD orAdaGrad) that return an averaged solution can be employed for minimizing aregularized convex problem; (ii) the step size is decreased in a stagewisemanner; (iii) an averaged solution is returned as the final solution that isselected from all stagewise averaged solutions with sampling probabilities {\itincreasing} as the stage number. Our theoretical results of stagewise AdaGradexhibit its adaptive convergence, therefore shed insights on its fasterconvergence for problems with sparse stochastic gradients than stagewise SGD.To the best of our knowledge, these new results are the first of their kind foraddressing the unresolved issues of existing theories mentioned earlier.Besides theoretical contributions, our empirical studies show that ourstagewise SGD and ADAGRAD improve the generalization performance of existingvariants/implementations of SGD and ADAGRAD.

Universal Stagewise Learning for Non-Convex Problems with Convergence on Averaged Solutions