English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Batch Normalization (BatchNorm) is commonly used in Convolutional NeuralNetworks (CNNs) to improve training speed and stability. However, there isstill limited consensus on why this technique is effective. This paper usesconcepts from the traditional adaptive filter domain to provide insight intothe dynamics and inner workings of BatchNorm. First, we show that theconvolution weight updates have natural modes whose stability and convergencespeed are tied to the eigenvalues of the input autocorrelation matrices, whichare controlled by BatchNorm through the convolution layers' channel-wisestructure. Furthermore, our experiments demonstrate that the speed andstability benefits are distinct effects. At low learning rates, it isBatchNorm's amplification of the smallest eigenvalues that improves convergencespeed, while at high learning rates, it is BatchNorm's suppression of thelargest eigenvalues that ensures stability. Lastly, we prove that in the firsttraining step, when normalization is needed most, BatchNorm satisfies the sameoptimization as Normalized Least Mean Square (NLMS), while it continues toapproximate this condition in subsequent steps. The analyses provided in thispaper lay the groundwork for gaining further insight into the operation ofmodern neural network structures using adaptive filter theory.

Separating the Effects of Batch Normalization on CNN Training Speed and Stability Using Classical Adaptive Filter Theory