English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Training neural networks with batch normalization and weight decay has becomea common practice in recent years. In this work, we show that their combineduse may result in a surprising periodic behavior of optimization dynamics: thetraining process regularly exhibits destabilizations that, however, do not leadto complete divergence but cause a new period of training. We rigorouslyinvestigate the mechanism underlying the discovered periodic behavior from bothempirical and theoretical points of view and analyze the conditions in which itoccurs in practice. We also demonstrate that periodic behavior can be regardedas a generalization of two previously opposing perspectives on training withbatch normalization and weight decay, namely the equilibrium presumption andthe instability presumption.

On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay