English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Regularization is typically understood as improving generalization byaltering the landscape of local extrema to which the model eventuallyconverges. Deep neural networks (DNNs), however, challenge this view: We showthat removing regularization after an initial transient period has littleeffect on generalization, even if the final loss landscape is the same as ifthere had been no regularization. In some cases, generalization even improvesafter interrupting regularization. Conversely, if regularization is appliedonly after the initial transient, it has no effect on the final solution, whosegeneralization gap is as bad as if regularization never happened. This suggeststhat what matters for training deep networks is not just whether or how, butwhen to regularize. The phenomena we observe are manifest in different datasets(CIFAR-10, CIFAR-100), different architectures (ResNet-18, All-CNN), differentregularization methods (weight decay, data augmentation), different learningrate schedules (exponential, piece-wise constant). They collectively suggestthat there is a ``critical period'' for regularizing deep networks that isdecisive of the final performance. More analysis should, therefore, focus onthe transient rather than asymptotic behavior of learning.

Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence