English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Using an extended and formalized version of the Q/C map analysis of Poole etal. (2016), along with Neural Tangent Kernel theory, we identify the mainpathologies present in deep networks that prevent them from training fast andgeneralizing to unseen data, and show how these can be avoided by carefullycontrolling the "shape" of the network's initialization-time kernel function.We then develop a method called Deep Kernel Shaping (DKS), which accomplishesthis using a combination of precise parameter initialization, activationfunction transformations, and small architectural tweaks, all of which preservethe model class. In our experiments we show that DKS enables SGD training ofresidual networks without normalization layers on Imagenet and CIFAR-10classification tasks at speeds comparable to standard ResNetV2 and Wide-ResNetmodels, with only a small decrease in generalization performance. And whenusing K-FAC as the optimizer, we achieve similar results for networks withoutskip connections. Our results apply for a large variety of activationfunctions, including those which traditionally perform very badly, such as thelogistic sigmoid. In addition to DKS, we contribute a detailed analysis of skipconnections, normalization layers, special activation functions like RELU andSELU, and various initialization schemes, explaining their effectiveness asalternative (and ultimately incomplete) ways of "shaping" the network'sinitialization-time kernel.

Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping