English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Stochastically controlled stochastic gradient (SCSG) methods have been provedto converge efficiently to first-order stationary points which, however, can besaddle points in nonconvex optimization. It has been observed that a stochasticgradient descent (SGD) step introduces anistropic noise around saddle pointsfor deep learning and non-convex half space learning problems, which indicatesthat SGD satisfies the correlated negative curvature (CNC) condition for theseproblems. Therefore, we propose to use a separate SGD step to help the SCSGmethod escape from strict saddle points, resulting in the CNC-SCSG method. TheSGD step plays a role similar to noise injection but is more stable. We provethat the resultant algorithm converges to a second-order stationary point witha convergence rate of $\tilde{O}( \epsilon^{-2} log( 1/\epsilon))$ where$\epsilon$ is the pre-specified error tolerance. This convergence rate isindependent of the problem dimension, and is faster than that of CNC-SGD. Amore general framework is further designed to incorporate the proposed CNC-SCSGinto any first-order method for the method to escape saddle points. Simulationstudies illustrate that the proposed algorithm can escape saddle points in muchfewer epochs than the gradient descent methods perturbed by either noiseinjection or a SGD step.

Escaping Saddle Points with Stochastically Controlled Stochastic Gradient Methods