English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Stochastic Gradient Descent (SGD) stands as a cornerstone optimizationalgorithm with proven real-world empirical successes but relatively limitedtheoretical understanding. Recent research has illuminated a key factorcontributing to its practical efficacy: the implicit regularization itinstigates. Several studies have investigated the linear stability property ofSGD in the vicinity of a stationary point as a predictive proxy for sharpnessand generalization error in overparameterized neural networks (Wu et al., 2022;Jastrzebski et al., 2019; Cohen et al., 2021). In this paper, we delve deeperinto the relationship between linear stability and sharpness. Morespecifically, we meticulously delineate the necessary and sufficient conditionsfor linear stability, contingent on hyperparameters of SGD and the sharpness atthe optimum. Towards this end, we introduce a novel coherence measure of theloss Hessian that encapsulates pertinent geometric properties of the lossfunction that are relevant to the linear stability of SGD. It enables us toprovide a simplified sufficient condition for identifying linear instability atan optimum. Notably, compared to previous works, our analysis relies onsignificantly milder assumptions and is applicable for a broader class of lossfunctions than known before, encompassing not only mean-squared error but alsocross-entropy loss.

A Precise Characterization of SGD Stability Using Loss Surface Geometry