English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Noise injection (NI) is an efficient technique to mitigate over-fitting inneural networks (NNs). The Bernoulli NI procedure as implemented in dropout andshakeout has connections with $l_1$ and $l_2$ regularization for the NN modelparameters. We propose whiteout, a family NI regularization techniques (NIRT)through injecting adaptive Gaussian noises during the training of NNs. Whiteoutis the first NIRT than imposes a broad range of the $l_{\gamma}$ sparsityregularization $(\gamma\in(0,2))$ without having to involving the $l_2$regularization. Whiteout can also be extended to offer regularizations similarto the adaptive lasso and group lasso. We establish the regularization effectof whiteout in the framework of generalized linear models with closed-formpenalty terms and show that whiteout stabilizes the training of NNs withdecreased sensitivity to small perturbations in the input. We establish thatthe noise-perturbed empirical loss function (pelf) with whiteout convergesalmost surely to the ideal loss function (ilf), and the minimizer of the pelfis consistent for the minimizer of the ilf. We derive the tail bound on thepelf to establish the practical feasibility in its minimization. Thesuperiority of whiteout over the Bernoulli NIRTs, dropout and shakeout, inlearning NNs with relatively small-sized training sets and non-inferiority inlarge-sized training sets is demonstrated in both simulated and real-life datasets. This work represents the first in-depth theoretical, methodological, andpractical examination of the regularization effects of both additive andmultiplicative Gaussian NI in deep NNs.

Whiteout: Gaussian Adaptive Noise Regularization in Deep Neural Networks