English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We systematically analyze optimization dynamics in deep neural networks(DNNs) trained with stochastic gradient descent (SGD) and study the effect oflearning rate $\eta$, depth $d$, and width $w$ of the neural network. Byanalyzing the maximum eigenvalue $\lambda^H_t$ of the Hessian of the loss,which is a measure of sharpness of the loss landscape, we find that thedynamics can show four distinct regimes: (i) an early time transient regime,(ii) an intermediate saturation regime, (iii) a progressive sharpening regime,and (iv) a late time ``edge of stability" regime. The early and intermediateregimes (i) and (ii) exhibit a rich phase diagram depending on $\eta \equiv c /\lambda_0^H $, $d$, and $w$. We identify several critical values of $c$, whichseparate qualitatively distinct phenomena in the early time dynamics oftraining loss and sharpness. Notably, we discover the opening up of a``sharpness reduction" phase, where sharpness decreases at early times, as $d$and $1/w$ are increased.

Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width