English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

This manuscript investigates the one-pass stochastic gradient descent (SGD)dynamics of a two-layer neural network trained on Gaussian data and labelsgenerated by a similar, though not necessarily identical, target function. Werigorously analyse the limiting dynamics via a deterministic andlow-dimensional description in terms of the sufficient statistics for thepopulation risk. Our unifying analysis bridges different regimes of interest,such as the classical gradient-flow regime of vanishing learning rate, thehigh-dimensional regime of large input dimension, and the overparameterised"mean-field" regime of large network width, covering as well the intermediateregimes where the limiting dynamics is determined by the interplay betweenthese behaviours. In particular, in the high-dimensional limit, theinfinite-width dynamics is found to remain close to a low-dimensional subspacespanned by the target principal directions. Our results therefore provide aunifying picture of the limiting SGD dynamics with synthetic data.

From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks