English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

In this work we explore the limiting dynamics of deep neural networks trainedwith stochastic gradient descent (SGD). As observed previously, long afterperformance has converged, networks continue to move through parameter space bya process of anomalous diffusion in which distance travelled grows as a powerlaw in the number of gradient updates with a nontrivial exponent. We reveal anintricate interaction between the hyperparameters of optimization, thestructure in the gradient noise, and the Hessian matrix at the end of trainingthat explains this anomalous diffusion. To build this understanding, we firstderive a continuous-time model for SGD with finite learning rates and batchsizes as an underdamped Langevin equation. We study this equation in thesetting of linear regression, where we can derive exact, analytic expressionsfor the phase space dynamics of the parameters and their instantaneousvelocities from initialization to stationarity. Using the Fokker-Planckequation, we show that the key ingredient driving these dynamics is not theoriginal training loss, but rather the combination of a modified loss, whichimplicitly regularizes the velocity, and probability currents, which causeoscillations in phase space. We identify qualitative and quantitativepredictions of this theory in the dynamics of a ResNet-18 model trained onImageNet. Through the lens of statistical physics, we uncover a mechanisticorigin for the anomalous limiting dynamics of deep neural networks trained withSGD.

The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion