English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Several recent trends in machine learning theory and practice, from thedesign of state-of-the-art Gaussian Process to the convergence analysis of deepneural nets (DNNs) under stochastic gradient descent (SGD), have found itfruitful to study wide random neural networks. Central to these approaches arecertain scaling limits of such networks. We unify these results by introducinga notion of a straightline \emph{tensor program} that can express most neuralnetwork computations, and we characterize its scaling limit when its tensorsare large and randomized. From our framework follows (1) the convergence ofrandom neural networks to Gaussian processes for architectures such asrecurrent neural networks, convolutional neural networks, residual networks,attention, and any combination thereof, with or without batch normalization;(2) conditions under which the \emph{gradient independence assumption} -- thatweights in backpropagation can be assumed to be independent from weights in theforward pass -- leads to correct computation of gradient dynamics, andcorrections when it does not; (3) the convergence of the Neural Tangent Kernel,a recently proposed kernel used to predict training dynamics of neural networksunder gradient descent, at initialization for all architectures in (1) withoutbatch normalization. Mathematically, our framework is general enough torederive classical random matrix results such as the semicircle and theMarchenko-Pastur laws, as well as recent results in neural network Jacobiansingular values. We hope our work opens a way toward design of even strongerGaussian Processes, initialization schemes to avoid gradientexplosion/vanishing, and deeper understanding of SGD dynamics in modernarchitectures.

Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation