English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We perform an experimental study of the dynamics of Stochastic GradientDescent (SGD) in learning deep neural networks for several real and syntheticclassification tasks. We show that in the initial epochs, almost all of theperformance improvement of the classifier obtained by SGD can be explained by alinear classifier. More generally, we give evidence for the hypothesis that, asiterations progress, SGD learns functions of increasing complexity. Thishypothesis can be helpful in explaining why SGD-learned classifiers tend togeneralize well even in the over-parameterized regime. We also show that thelinear classifier learned in the initial stages is "retained" throughout theexecution even if training is continued to the point of zero training error,and complement this with a theoretical result in a simplified model. Key to ourwork is a new measure of how well one classifier explains the performance ofanother, based on conditional mutual information.

SGD on Neural Networks Learns Functions of Increasing Complexity