English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The learning dynamics of deep neural networks are not well understood. Theinformation bottleneck (IB) theory proclaimed separate fitting and compressionphases. But they have since been heavily debated. We comprehensively analyzethe learning dynamics by investigating a layer's reconstruction ability of theinput and prediction performance based on the evolution of parameters duringtraining. We empirically show the existence of three phases using commondatasets and architectures such as ResNet and VGG: (i) near constantreconstruction loss, (ii) decrease, and (iii) increase. We also derive anempirically grounded data model and prove the existence of phases forsingle-layer networks. Technically, our approach leverages classical complexityanalysis. It differs from IB by relying on measuring reconstruction loss ratherthan information theoretic measures to relate information of intermediatelayers and inputs. Our work implies a new best practice for transfer learning:We show empirically that the pre-training of a classifier should stop wellbefore its performance is optimal.

Understanding and Leveraging the Learning Phases of Neural Networks