English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

We analyze architectural features of Deep Neural Networks (DNNs) using theso-called Neural Tangent Kernel (NTK), which describes the training andgeneralization of DNNs in the infinite-width setting. In this setting, we showthat for fully-connected DNNs, as the depth grows, two regimes appear: "order",where the (scaled) NTK converges to a constant, and "chaos", where it convergesto a Kronecker delta. Extreme order slows down training while extreme chaoshinders generalization. Using the scaled ReLU as a nonlinearity, we end up inthe ordered regime. In contrast, Layer Normalization brings the network intothe chaotic regime. We observe a similar effect for Batch Normalization (BN)applied after the last nonlinearity. We uncover the same order and chaos modesin Deep Deconvolutional Networks (DC-NNs). Our analysis explains the appearanceof so-called checkerboard patterns and border artifacts. Moving the networkinto the chaotic regime prevents checkerboard patterns; we propose agraph-based parametrization which eliminates border artifacts; finally, weintroduce a new layer-dependent learning rate to improve the convergence ofDC-NNs. We illustrate our findings on DCGANs: the ordered regime leads to acollapse of the generator to a checkerboard mode, which can be avoided bytuning the nonlinearity to reach the chaotic regime. As a result, we are ableto obtain good quality samples for DCGANs without BN.

Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts