English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The merits of fast convergence and potentially better performance of theweight normalization family have drawn increasing attention in recent years.These methods use standardization or normalization that changes the weight$\boldsymbol{W}$ to $\boldsymbol{W}'$, which makes $\boldsymbol{W}'$independent to the magnitude of $\boldsymbol{W}$. Surprisingly,$\boldsymbol{W}$ must be decayed during gradient descent, otherwise we willobserve a severe under-fitting problem, which is very counter-intuitive sinceweight decay is widely known to prevent deep networks from over-fitting. Inthis paper, we \emph{theoretically} prove that the weight decay term$\frac{1}{2}\lambda||{\boldsymbol{W}}||^2$ merely modulates the effectivelearning rate for improving objective optimization, and has no influence ongeneralization when the weight normalization family is compositely employed.Furthermore, we also expose several critical problems when introducing weightdecay term to weight normalization family, including the missing of globalminimum and training instability. To address these problems, we propose an$\epsilon-$shifted $L_2$ regularizer, which shifts the $L_2$ objective by apositive constant $\epsilon$. Such a simple operation can theoreticallyguarantee the existence of global minimum, while preventing the network weightsfrom being too small and thus avoiding gradient float overflow. Itsignificantly improves the training stability and can achieve slightly betterperformance in our practice. The effectiveness of $\epsilon-$shifted $L_2$regularizer is comprehensively validated on the ImageNet, CIFAR-100, and COCOdatasets. Our codes and pretrained models will be released inhttps://github.com/implus/PytorchInsight.

Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon-$shifted $L_2$ Regularizer