English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Symmetries have proven to be important ingredients in the analysis of neuralnetworks. So far their use has mostly been implicit or seemingly coincidental.  We undertake a systematic study of the role that symmetry plays. Inparticular, we clarify how symmetry interacts with the learning algorithm. Thekey ingredient in our study is played by Noether's celebrated theorem which,informally speaking, states that symmetry leads to conserved quantities (e.g.,conservation of energy or conservation of momentum). In the realm of neuralnetworks under gradient descent, model symmetries imply restrictions on thegradient path. E.g., we show that symmetry of activation functions leads toboundedness of weight matrices, for the specific case of linear activations itleads to balance equations of consecutive layers, data augmentation leads togradient paths that have "momentum"-type restrictions, and time symmetry leadsto a version of the Neural Tangent Kernel.  Symmetry alone does not specify the optimization path, but the moresymmetries are contained in the model the more restrictions are imposed on thepath. Since symmetry also implies over-parametrization, this in effect impliesthat some part of this over-parametrization is cancelled out by the existenceof the conserved quantities.  Symmetry can therefore be thought of as one further important tool inunderstanding the performance of neural networks under gradient descent.

Noether: The More Things Change, the More Stay the Same