English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The dynamical stability of optimization methods at the vicinity of minima ofthe loss has recently attracted significant attention. For gradient descent(GD), stable convergence is possible only to minima that are sufficiently flatw.r.t. the step size, and those have been linked with favorable properties ofthe trained model. However, while the stability threshold of GD is well-known,to date, no explicit expression has been derived for the exact threshold ofstochastic GD (SGD). In this paper, we derive such a closed-form expression.Specifically, we provide an explicit condition on the step size $\eta$ that isboth necessary and sufficient for the stability of SGD in the mean squaresense. Our analysis sheds light on the precise role of the batch size $B$.Particularly, we show that the stability threshold is a monotonicallynon-decreasing function of the batch size, which means that reducing the batchsize can only hurt stability. Furthermore, we show that SGD's stabilitythreshold is equivalent to that of a process which takes in each iteration afull batch gradient step w.p. $1-p$, and a single sample gradient step w.p.$p$, where $p \approx 1/B $. This indicates that even with moderate batchsizes, SGD's stability threshold is very close to that of GD's. Finally, weprove simple necessary conditions for stability, which depend on the batchsize, and are easier to compute than the precise threshold. We demonstrate ourtheoretical findings through experiments on the MNIST dataset.

Exact Mean Square Linear Stability Analysis for SGD