English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we study the implicit regularization of stochastic gradientdescent (SGD) through the lens of {\em dynamical stability} (Wu et al., 2018).We start by revising existing stability analyses of SGD, showing how theFrobenius norm and trace of Hessian relate to different notions of stability.Notably, if a global minimum is linearly stable for SGD, then the trace ofHessian must be less than or equal to $2/\eta$, where $\eta$ denotes thelearning rate. By contrast, for gradient descent (GD), the stability imposes asimilar constraint but only on the largest eigenvalue of Hessian. We then turnto analyze the generalization properties of these stable minima, focusingspecifically on two-layer ReLU networks and diagonal linear networks. Notably,we establish the {\em equivalence} between these metrics of sharpness andcertain parameter norms for the two models, which allows us to show that thestable minima of SGD provably generalize well. By contrast, thestability-induced regularization of GD is provably too weak to ensuresatisfactory generalization. This discrepancy provides an explanation of whySGD often generalizes better than GD. Note that the learning rate (LR) plays apivotal role in the strength of stability-induced regularization. As the LRincreases, the regularization effect becomes more pronounced, elucidating whySGD with a larger LR consistently demonstrates superior generalizationcapabilities. Additionally, numerical experiments are provided to support ourtheoretical findings.

The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent