English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Traditional analyses of gradient descent show that when the largesteigenvalue of the Hessian, also known as the sharpness $S(\theta)$, is boundedby $2/\eta$, training is "stable" and the training loss decreasesmonotonically. Recent works, however, have observed that this assumption doesnot hold when training modern neural networks with full batch or large batchgradient descent. Most recently, Cohen et al. (2021) observed two importantphenomena. The first, dubbed progressive sharpening, is that the sharpnesssteadily increases throughout training until it reaches the instability cutoff$2/\eta$. The second, dubbed edge of stability, is that the sharpness hovers at$2/\eta$ for the remainder of training while the loss continues decreasing,albeit non-monotonically. We demonstrate that, far from being chaotic, thedynamics of gradient descent at the edge of stability can be captured by acubic Taylor expansion: as the iterates diverge in direction of the topeigenvector of the Hessian due to instability, the cubic term in the localTaylor expansion of the loss function causes the curvature to decrease untilstability is restored. This property, which we call self-stabilization, is ageneral property of gradient descent and explains its behavior at the edge ofstability. A key consequence of self-stabilization is that gradient descent atthe edge of stability implicitly follows projected gradient descent (PGD) underthe constraint $S(\theta) \le 2/\eta$. Our analysis provides precisepredictions for the loss, sharpness, and deviation from the PGD trajectorythroughout training, which we verify both empirically in a number of standardsettings and theoretically under mild conditions. Our analysis uncovers themechanism for gradient descent's implicit bias towards stability.

Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability