English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In modern deep learning, there is a recent and growing literature on theinterplay between large-width asymptotic properties of deep Gaussian neuralnetworks (NNs), i.e. deep NNs with Gaussian-distributed weights, and Gaussianstochastic processes (SPs). Such an interplay has proved to be critical inBayesian inference under Gaussian SP priors, kernel regression for infinitelywide deep NNs trained via gradient descent, and information propagation withininfinitely wide NNs. Motivated by empirical analyses that show the potential ofreplacing Gaussian distributions with Stable distributions for the NN'sweights, in this paper we present a rigorous analysis of the large-widthasymptotic behaviour of (fully connected) feed-forward deep Stable NNs, i.e.deep NNs with Stable-distributed weights. We show that as the width goes toinfinity jointly over the NN's layers, i.e. the ``joint growth" setting, arescaled deep Stable NN converges weakly to a Stable SP whose distribution ischaracterized recursively through the NN's layers. Because of thenon-triangular structure of the NN, this is a non-standard asymptotic problem,to which we propose an inductive approach of independent interest. Then, weestablish sup-norm convergence rates of the rescaled deep Stable NN to theStable SP, under the ``joint growth" and a ``sequential growth" of the widthover the NN's layers. Such a result provides the difference between the ``jointgrowth" and the ``sequential growth" settings, showing that the former leads toa slower rate than the latter, depending on the depth of the layer and thenumber of inputs of the NN. Our work extends some recent results on infinitelywide limits for deep Gaussian NNs to the more general deep Stable NNs,providing the first result on convergence rates in the ``joint growth" setting.

Deep Stable neural networks: large-width asymptotics and convergence rates