English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Deep neural networks with remarkably strong generalization performances areusually over-parameterized. Despite explicit regularization strategies are usedfor practitioners to avoid over-fitting, the impacts are often small. Sometheoretical studies have analyzed the implicit regularization effect ofstochastic gradient descent (SGD) on simple machine learning models withcertain assumptions. However, how it behaves practically in state-of-the-artmodels and real-world datasets is still unknown. To bridge this gap, we studythe role of SGD implicit regularization in deep learning systems. We show pureSGD tends to converge to minimas that have better generalization performancesin multiple natural language processing (NLP) tasks. This phenomenon coexistswith dropout, an explicit regularizer. In addition, neural network's finitelearning capability does not impact the intrinsic nature of SGD's implicitregularization effect. Specifically, under limited training samples or withcertain corrupted labels, the implicit regularization effect remains strong. Wefurther analyze the stability by varying the weight initialization range. Wecorroborate these experimental findings with a decision boundary visualizationusing a 3-layer neural network for interpretation. Altogether, our work enablesa deepened understanding on how implicit regularization affects the deeplearning model and sheds light on the future study of the over-parameterizedmodel's generalization ability.

Implicit Regularization of Stochastic Gradient Descent in Natural Language Processing: Observations and Implications