English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We investigate the stationary (late-time) training regime of single- andtwo-layer linear neural networks within the continuum limit of stochasticgradient descent (SGD) for synthetic Gaussian data. In the case of asingle-layer network in the weakly oversampled regime, the spectrum of thenoise covariance matrix deviates notably from the Hessian, which can beattributed to the broken detailed balance of SGD dynamics. The weightfluctuations are in this case generally anisotropic, but experience anisotropic loss. For a two-layer network, we obtain the stochastic dynamics ofthe weights in each layer and analyze the associated stationary covariances. Weidentify the inter-layer coupling as a new source of anisotropy for the weightfluctuations. In contrast to the single-layer case, the weight fluctuationsexperience an anisotropic loss, the flatness of which is inversely related tothe fluctuation variance. We thereby provide an analytical derivation of therecently observed inverse variance-flatness relation in a deep linear networkmodel.

Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation