English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent work by Jacot et al. (2018) has shown that training a neural networkusing gradient descent in parameter space is related to kernel gradient descentin function space with respect to the Neural Tangent Kernel (NTK). Lee et al.(2019) built on this result by establishing that the output of a neural networktrained using gradient descent can be approximated by a linear model when thenetwork width is large. Indeed, under regularity conditions, the NTK convergesto a time-independent kernel in the infinite-width limit. This regime is oftencalled the NTK regime. In parallel, recent works on signal propagation (Pooleet al., 2016; Schoenholz et al., 2017; Hayou et al., 2019a) studied the impactof the initialization and the activation function on signal propagation in deepneural networks. In this paper, we connect these two theories by quantifyingthe impact of the initialization and the activation function on the NTK whenthe network depth becomes large. In particular, we provide a comprehensiveanalysis of the convergence rates of the NTK regime to the infinite depthregime.

Exact Convergence Rates of the Neural Tangent Kernel in the Large Depth Limit