English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We develop a mathematically rigorous framework for multilayer neural networksin the mean field regime. As the network's widths increase, the network'slearning trajectory is shown to be well captured by a meaningful anddynamically nonlinear limit (the \textit{mean field} limit), which ischaracterized by a system of ODEs. Our framework applies to a broad range ofnetwork architectures, learning dynamics and network initializations. Centralto the framework is the new idea of a \textit{neuronal embedding}, whichcomprises of a non-evolving probability space that allows to embed neuralnetworks of arbitrary widths.  Using our framework, we prove several properties of large-width multilayerneural networks. Firstly we show that independent and identically distributedinitializations cause strong degeneracy effects on the network's learningtrajectory when the network's depth is at least four. Secondly we obtainseveral global convergence guarantees for feedforward multilayer networks undera number of different setups. These include two-layer and three-layer networkswith independent and identically distributed initializations, and multilayernetworks of arbitrary depths with a special type of correlated initializationsthat is motivated by the new concept of \textit{bidirectional diversity}.Unlike previous works that rely on convexity, our results admit non-convexlosses and hinge on a certain universal approximation property, which is adistinctive feature of infinite-width neural networks and is shown to holdthroughout the training process. Aside from being the first known results forglobal convergence of multilayer networks in the mean field regime, theydemonstrate flexibility of our framework and incorporate several new ideas andinsights that depart from the conventional convex optimization wisdom.

A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks