English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Existing analyses of neural network training often operate under theunrealistic assumption of an extremely small learning rate. This lies in starkcontrast to practical wisdom and empirical studies, such as the work of J.Cohen et al. (ICLR 2021), which exhibit startling new phenomena (the "edge ofstability" or "unstable convergence") and potential benefits for generalizationin the large learning rate regime. Despite a flurry of recent works on thistopic, however, the latter effect is still poorly understood. In this paper, wetake a step towards understanding genuinely non-convex training dynamics withlarge learning rates by performing a detailed analysis of gradient descent forsimplified models of two-layer neural networks. For these models, we provablyestablish the edge of stability phenomenon and discover a sharp phasetransition for the step size below which the neural network fails to learn"threshold-like" neurons (i.e., neurons with a non-zero first-layer bias). Thiselucidates one possible mechanism by which the edge of stability can in factlead to better generalization, as threshold neurons are basic building blockswith useful inductive bias for many tasks.

Learning threshold neurons via the "edge of stability"