English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent developments in applications of artificial neural networks with over$n=10^{14}$ parameters make it extremely important to study the large $n$behaviour of such networks. Most works studying wide neural networks havefocused on the infinite width $n \to +\infty$ limit of such networks and haveshown that, at initialization, they correspond to Gaussian processes. In thiswork we will study their behavior for large, but finite $n$. Our maincontributions are the following:  (1) The computation of the corrections to Gaussianity in terms of anasymptotic series in $n^{-\frac{1}{2}}$. The coefficients in this expansion aredetermined by the statistics of parameter initialization and by the activationfunction.  (2) Controlling the evolution of the outputs of finite width $n$ networks,during training, by computing deviations from the limiting infinite width case(in which the network evolves through a linear flow). This improves previousestimates and yields sharper decay rates for the (finite width) NTK in terms of$n$, valid during the entire training procedure. As a corollary, we also provethat, with arbitrarily high probability, the training of sufficiently wideneural networks converges to a global minimum of the corresponding quadraticloss function.  (3) Estimating how the deviations from Gaussianity evolve with training interms of $n$. In particular, using a certain metric in the space of measures wefind that, along training, the resulting measure is within$n^{-\frac{1}{2}}(\log n)^{1+}$ of the time dependent Gaussian processcorresponding to the infinite width network (which is explicitly given byprecomposing the initial Gaussian process with the linear flow corresponding totraining in the infinite width limit).

Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training