English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Despite the non-convex optimization landscape, over-parametrized shallownetworks are able to achieve global convergence under gradient descent. Thepicture can be radically different for narrow networks, which tend to get stuckin badly-generalizing local minima. Here we investigate the cross-over betweenthese two regimes in the high-dimensional setting, and in particularinvestigate the connection between the so-called mean-field/hydrodynamic regimeand the seminal approach of Saad & Solla. Focusing on the case of Gaussiandata, we study the interplay between the learning rate, the time scale, and thenumber of hidden units in the high-dimensional dynamics of stochastic gradientdescent (SGD). Our work builds on a deterministic description of SGD inhigh-dimensions from statistical physics, which we extend and for which weprovide rigorous convergence rates.

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks