English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Stochastic regularisation is an important weapon in the arsenal of a deeplearning practitioner. However, despite recent theoretical advances, ourunderstanding of how noise influences signal propagation in deep neuralnetworks remains limited. By extending recent work based on mean field theory,we develop a new framework for signal propagation in stochastic regularisedneural networks. Our noisy signal propagation theory can incorporate severalcommon noise distributions, including additive and multiplicative Gaussiannoise as well as dropout. We use this framework to investigate initialisationstrategies for noisy ReLU networks. We show that no critical initialisationstrategy exists using additive noise, with signal propagation explodingregardless of the selected noise distribution. For multiplicative noise (e.g.dropout), we identify alternative critical initialisation strategies thatdepend on the second moment of the noise distribution. Simulations andexperiments on real-world data confirm that our proposed initialisation is ableto stably propagate signals in deep networks, while using an initialisationdisregarding noise fails to do so. Furthermore, we analyse correlation dynamicsbetween inputs. Stronger noise regularisation is shown to reduce the depth towhich discriminatory information about the inputs to a noisy ReLU network isable to propagate, even when initialised at criticality. We support ourtheoretical predictions for these trainable depths with simulations, as well aswith experiments on MNIST and CIFAR-10

Critical initialisation for deep signal propagation in noisy rectifier neural networks