English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Weight decay is a simple yet powerful regularization technique that has beenvery widely used in training of deep neural networks (DNNs). While weight decayhas attracted much attention, previous studies fail to discover some overlookedpitfalls on large gradient norms resulted by weight decay. In this paper, wediscover that, weight decay can unfortunately lead to large gradient norms atthe final phase (or the terminated solution) of training, which often indicatesbad convergence and poor generalization. To mitigate the gradient-norm-centeredpitfalls, we present the first practical scheduler for weight decay, called theScheduled Weight Decay (SWD) method that can dynamically adjust the weightdecay strength according to the gradient norm and significantly penalize largegradient norms during training. Our experiments also support that SWD indeedmitigates large gradient norms and often significantly outperforms theconventional constant weight decay strategy for Adaptive Moment Estimation(Adam).

On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective