English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In suitably initialized wide networks, small learning rates transform deepneural networks (DNNs) into neural tangent kernel (NTK) machines, whosetraining dynamics is well-approximated by a linear weight expansion of thenetwork at initialization. Standard training, however, diverges from itslinearization in ways that are poorly understood. We study the relationshipbetween the training dynamics of nonlinear deep networks, the geometry of theloss landscape, and the time evolution of a data-dependent NTK. We do sothrough a large-scale phenomenological analysis of training, synthesizingdiverse measures characterizing loss landscape geometry and NTK dynamics. Inmultiple neural architectures and datasets, we find these diverse measuresevolve in a highly correlated manner, revealing a universal picture of the deeplearning process. In this picture, deep network training exhibits a highlychaotic rapid initial transient that within 2 to 3 epochs determines the finallinearly connected basin of low loss containing the end point of training.During this chaotic transient, the NTK changes rapidly, learning usefulfeatures from the training data that enables it to outperform the standardinitial NTK by a factor of 3 in less than 3 to 4 epochs. After this rapidchaotic transient, the NTK changes at constant velocity, and its performancematches that of full network training in 15% to 45% of training time. Overall,our analysis reveals a striking correlation between a diverse set of metricsover training time, governed by a rapid chaotic to stable transition in thefirst few epochs, that together poses challenges and opportunities for thedevelopment of more accurate theories of deep learning.

Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel