English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This article considers fully connected neural networks with Gaussian randomweights and biases as well as $L$ hidden layers, each of width proportional toa large parameter $n$. For polynomially bounded non-linearities we give sharpestimates in powers of $1/n$ for the joint cumulants of the network output andits derivatives. Moreover, we show that network cumulants form a perturbativelysolvable hierarchy in powers of $1/n$ in that $k$-th order cumulants in onelayer have recursions that depend to leading order in $1/n$ only on $j$-thorder cumulants at the previous layer with $j\leq k$. By solving a variety ofsuch recursions, however, we find that the depth-to-width ratio $L/n$ plays therole of an effective network depth, controlling both the scale of fluctuationsat individual neurons and the size of inter-neuron correlations. Thus, whilethe cumulant recursions we derive form a hierarchy in powers of $1/n$,contributions of order $1/n^k$ often grow like $L^k$ and are hencenon-negligible at positive $L/n$. We use this to study a somewhat simplifiedversion of the exploding and vanishing gradient problem, proving that thisparticular variant occurs if and only if $L/n$ is large. Several key ideas inthis article were first developed at a physics level of rigor in a recentmonograph of Daniel A. Roberts, Sho Yaida, and the author. This article notonly makes these ideas mathematically precise but also significantly extendsthem, opening the way to obtaining corrections to all orders in $1/n$.

Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies