English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

A fundamental property of deep learning normalization techniques, such asbatch normalization, is making the pre-normalization parameters scaleinvariant. The intrinsic domain of such parameters is the unit sphere, andtherefore their gradient optimization dynamics can be represented via sphericaloptimization with varying effective learning rate (ELR), which was studiedpreviously. However, the varying ELR may obscure certain characteristics of theintrinsic loss landscape structure. In this work, we investigate the propertiesof training scale-invariant neural networks directly on the sphere using afixed ELR. We discover three regimes of such training depending on the ELRvalue: convergence, chaotic equilibrium, and divergence. We study these regimesin detail both on a theoretical examination of a toy example and on a thoroughempirical analysis of real scale-invariant deep learning models. Each regimehas unique features and reflects specific properties of the intrinsic losslandscape, some of which have strong parallels with previous research on bothregular and scale-invariant neural networks training. Finally, we demonstratehow the discovered regimes are reflected in conventional training of normalizednetworks and how they can be leveraged to achieve better optima.

Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes