English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Weight decay is a broadly used technique for training state-of-the-art deepnetworks, including large language models. Despite its widespread usage, itsrole remains poorly understood. In this work, we highlight that the role ofweight decay in modern deep learning is different from its regularizationeffect studied in classical learning theory. For overparameterized deepnetworks, we show how weight decay modifies the optimization dynamics enhancingthe ever-present implicit regularization of SGD via the loss stabilizationmechanism. In contrast, for underparameterized large language models trainedwith nearly online SGD, we describe how weight decay balances the bias-variancetradeoff in stochastic optimization leading to lower training loss. Moreover,we show that weight decay also prevents sudden loss divergences for bfloat16mixed-precision training which is a crucial tool for LLM training. Overall, wepresent a unifying perspective from ResNets on vision tasks to LLMs: weightdecay is never useful as an explicit regularizer but instead changes thetraining dynamics in a desirable way. Our code is available athttps://github.com/tml-epfl/why-weight-decay.

Why Do We Need Weight Decay in Modern Deep Learning?