English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Random Matrix Theory (RMT) is applied to analyze weight matrices of DeepNeural Networks (DNNs), including both production quality, pre-trained modelssuch as AlexNet and Inception, and smaller models trained from scratch, such asLeNet5 and a miniature-AlexNet. Empirical and theoretical results clearlyindicate that the DNN training process itself implicitly implements a form ofSelf-Regularization. The empirical spectral density (ESD) of DNN layer matricesdisplays signatures of traditionally-regularized statistical models, even inthe absence of exogenously specifying traditional forms of explicitregularization. Building on relatively recent results in RMT, most notably itsextension to Universality classes of Heavy-Tailed matrices, we develop a theoryto identify 5+1 Phases of Training, corresponding to increasing amounts ofImplicit Self-Regularization. These phases can be observed during the trainingprocess as well as in the final learned DNNs. For smaller and/or older DNNs,this Implicit Self-Regularization is like traditional Tikhonov regularization,in that there is a "size scale" separating signal from noise. Forstate-of-the-art DNNs, however, we identify a novel form of Heavy-TailedSelf-Regularization, similar to the self-organization seen in the statisticalphysics of disordered systems. This results from correlations arising at allsize scales, which arises implicitly due to the training process itself. Thisimplicit Self-Regularization can depend strongly on the many knobs of thetraining process. By exploiting the generalization gap phenomena, wedemonstrate that we can cause a small model to exhibit all 5+1 phases oftraining simply by changing the batch size. This demonstrates that---all elsebeing equal---DNN optimization with larger batch sizes leads to less-wellimplicitly-regularized models, and it provides an explanation for thegeneralization gap phenomena.

Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning