English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Corrupting the input and hidden layers of deep neural networks (DNNs) withmultiplicative noise, often drawn from the Bernoulli distribution (or'dropout'), provides regularization that has significantly contributed to deeplearning's success. However, understanding how multiplicative corruptionsprevent overfitting has been difficult due to the complexity of a DNN'sfunctional form. In this paper, we show that when a Gaussian prior is placed ona DNN's weights, applying multiplicative noise induces a Gaussian scalemixture, which can be reparameterized to circumvent the problematic likelihoodfunction. Analysis can then proceed by using a type-II maximum likelihoodprocedure to derive a closed-form expression revealing how regularizationevolves as a function of the network's weights. Results show thatmultiplicative noise forces weights to become either sparse or invariant torescaling. We find our analysis has implications for model compression as itnaturally reveals a weight pruning rule that starkly contrasts with thecommonly used signal-to-noise ratio (SNR). While the SNR prunes weights withlarge variances, seeing them as noisy, our approach recognizes their robustnessand retains them. We empirically demonstrate our approach has a strongadvantage over the SNR heuristic and is competitive to retraining with softtargets produced from a teacher model.

A Scale Mixture Perspective of Multiplicative Noise in Neural Networks