English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Normalization Layers (NLs) are widely used in modern deep-learningarchitectures. Despite their apparent simplicity, their effect on optimizationis not yet fully understood. This paper introduces a spherical framework tostudy the optimization of neural networks with NLs from a geometricperspective. Concretely, the radial invariance of groups of parameters, such asfilters for convolutional neural networks, allows to translate the optimizationsteps on the $L_2$ unit hypersphere. This formulation and the associatedgeometric interpretation shed new light on the training dynamics. Firstly, thefirst effective learning rate expression of Adam is derived. Then thedemonstration that, in the presence of NLs, performing Stochastic GradientDescent (SGD) alone is actually equivalent to a variant of Adam constrained tothe unit hypersphere, stems from the framework. Finally, this analysis outlinesphenomena that previous variants of Adam act on and their importance in theoptimization process are experimentally validated.

Spherical Perspective on Learning with Normalization Layers