English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Substantial experiments have validated the success of Batch Normalization(BN) Layer in benefiting convergence and generalization. However, BN requiresextra memory and float-point calculation. Moreover, BN would be inaccurate onmicro-batch, as it depends on batch statistics. In this paper, we address theseproblems by simplifying BN regularization while keeping two fundamental impactsof BN layers, i.e., data decorrelation and adaptive learning rate. We propose anovel normalization method, named MimicNorm, to improve the convergence andefficiency in network training. MimicNorm consists of only two lightoperations, including modified weight mean operations (subtract mean valuesfrom weight parameter tensor) and one BN layer before loss function (last BNlayer). We leverage the neural tangent kernel (NTK) theory to prove that ourweight mean operation whitens activations and transits network into the chaoticregime like BN layer, and consequently, leads to an enhanced convergence. Thelast BN layer provides autotuned learning rates and also improves accuracy.Experimental results show that MimicNorm achieves similar accuracy for variousnetwork structures, including ResNets and lightweight networks like ShuffleNet,with a reduction of about 20% memory consumption. The code is publiclyavailable at https://github.com/Kid-key/MimicNorm.

MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization