English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Understanding when the noise in stochastic gradient descent (SGD) affectsgeneralization of deep neural networks remains a challenge, complicated by thefact that networks can operate in distinct training regimes. Here we study howthe magnitude of this noise $T$ affects performance as the size of the trainingset $P$ and the scale of initialization $\alpha$ are varied. For gradientdescent, $\alpha$ is a key parameter that controls if the network is`lazy'($\alpha\gg1$) or instead learns features ($\alpha\ll1$). Forclassification of MNIST and CIFAR10 images, our central results are: (i)obtaining phase diagrams for performance in the $(\alpha,T)$ plane. They showthat SGD noise can be detrimental or instead useful depending on the trainingregime. Moreover, although increasing $T$ or decreasing $\alpha$ both allow thenet to escape the lazy regime, these changes can have opposite effects onperformance. (ii) Most importantly, we find that the characteristic temperature$T_c$ where the noise of SGD starts affecting the trained model (and eventuallyperformance) is a power law of $P$. We relate this finding with the observationthat key dynamical quantities, such as the total variation of weights duringtraining, depend on both $T$ and $P$ as power laws. These results indicate thata key effect of SGD noise occurs late in training by affecting the stoppingprocess whereby all data are fitted. Indeed, we argue that due to SGD noise,nets must develop a stronger `signal', i.e. larger informative weights, to fitthe data, leading to a longer training time. A stronger signal and a longertraining time are also required when the size of the training set $P$increases. We confirm these views in the perceptron model, where signal andnoise can be precisely measured. Interestingly, exponents characterizing theeffect of SGD depend on the density of data near the decision boundary, as weexplain.

Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning