English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent work in signal propagation theory has shown that dropout limits thedepth to which information can propagate through a neural network. In thispaper, we investigate the effect of initialisation on training speed andgeneralisation for ReLU networks within this depth limit. We ask the followingresearch question: given that critical initialisation is crucial for trainingat large depth, if dropout limits the depth at which networks are trainable,does initialising critically still matter? We conduct a large-scale controlledexperiment, and perform a statistical analysis of over $12000$ trainednetworks. We find that (1) trainable networks show no statistically significantdifference in performance over a wide range of non-critical initialisations;(2) for initialisations that show a statistically significant difference, thenet effect on performance is small; (3) only extreme initialisations (verysmall or very large) perform worse than criticality. These findings also applyto standard ReLU networks of moderate depth as a special case of zero dropout.Our results therefore suggest that, in the shallow-to-moderate depth setting,critical initialisation provides zero performance gains when compared tooff-critical initialisations and that searching for off-criticalinitialisations that might improve training speed or generalisation, is likelyto be a fruitless endeavour.

If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks