English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we give a sharp analysis for Stochastic Gradient Descent (SGD)and prove that SGD is able to efficiently escape from saddle points and find an$(\epsilon, O(\epsilon^{0.5}))$-approximate second-order stationary point in$\tilde{O}(\epsilon^{-3.5})$ stochastic gradient computations for genericnonconvex optimization problems, when the objective function satisfiesgradient-Lipschitz, Hessian-Lipschitz, and dispersive noise assumptions. Thisresult subverts the classical belief that SGD requires at least$O(\epsilon^{-4})$ stochastic gradient computations for obtaining an$(\epsilon,O(\epsilon^{0.5}))$-approximate second-order stationary point. SuchSGD rate matches, up to a polylogarithmic factor of problem-dependentparameters, the rate of most accelerated nonconvex stochastic optimizationalgorithms that adopt additional techniques, such as Nesterov's momentumacceleration, negative curvature search, as well as quadratic and cubicregularization tricks. Our novel analysis gives new insights into nonconvex SGDand can be potentially generalized to a broad class of stochastic optimizationalgorithms.

Sharp Analysis for Nonconvex SGD Escaping from Saddle Points