English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Neural Tangent Kernel (NTK) theory is widely used to study the dynamics ofinfinitely-wide deep neural networks (DNNs) under gradient descent. But do theresults for infinitely-wide networks give us hints about the behavior of realfinite-width ones? In this paper, we study empirically when NTK theory is validin practice for fully-connected ReLU and sigmoid DNNs. We find out that whethera network is in the NTK regime depends on the hyperparameters of randominitialization and the network's depth. In particular, NTK theory does notexplain the behavior of sufficiently deep networks initialized so that theirgradients explode as they propagate through the network's layers: the kernel israndom at initialization and changes significantly during training in thiscase, contrary to NTK theory. On the other hand, in the case of vanishinggradients, DNNs are in the the NTK regime but become untrainable rapidly withdepth. We also describe a framework to study generalization properties of DNNs,in particular the variance of network's output function, by means of NTK theoryand discuss its limits.

Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?