English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

There is a growing literature on the study of large-width properties of deepGaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributedparameters or weights, and Gaussian stochastic processes. Motivated by someempirical and theoretical studies showing the potential of replacing Gaussiandistributions with Stable distributions, namely distributions with heavy tails,in this paper we investigate large-width properties of deep Stable NNs, i.e.deep NNs with Stable-distributed parameters. For sub-linear activationfunctions, a recent work has characterized the infinitely wide limit of asuitable rescaled deep Stable NN in terms of a Stable stochastic process, bothunder the assumption of a ``joint growth" and under the assumption of a``sequential growth" of the width over the NN's layers. Here, assuming a``sequential growth" of the width, we extend such a characterization to ageneral class of activation functions, which includes sub-linear,asymptotically linear and super-linear functions. As a novelty with respect toprevious works, our results rely on the use of a generalized central limittheorem for heavy tails distributions, which allows for an interesting unifiedtreatment of infinitely wide limits for deep Stable NNs. Our study shows thatthe scaling of Stable NNs and the stability of their infinitely wide limits maydepend on the choice of the activation function, bringing out a criticaldifference with respect to the Gaussian setting.

Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functions