English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Neural tangent kernel (NTK) is a powerful tool to analyze training dynamicsof neural networks and their generalization bounds. The study on NTK has beendevoted to typical neural network architectures, but it is incomplete forneural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomialneural networks (PNNs). In this work, we derive the finite-width NTKformulation for a special class of NNs-Hp, i.e., polynomial neural networks. Weprove their equivalence to the kernel regression predictor with the associatedNTK, which expands the application scope of NTK. Based on our results, weelucidate the separation of PNNs over standard neural networks with respect toextrapolation and spectral bias. Our two key insights are that when compared tostandard neural networks, PNNs can fit more complicated functions in theextrapolation regime and admit a slower eigenvalue decay of the respective NTK,leading to a faster learning towards high-frequency functions. Besides, ourtheoretical results can be extended to other types of NNs-Hp, which expand thescope of our work. Our empirical results validate the separations in broaderclasses of NNs-Hp, which provide a good justification for a deeperunderstanding of neural architectures.

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study