English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

As its width tends to infinity, a deep neural network's behavior undergradient descent can become simplified and predictable (e.g. given by theNeural Tangent Kernel (NTK)), if it is parametrized appropriately (e.g. the NTKparametrization). However, we show that the standard and NTK parametrizationsof a neural network do not admit infinite-width limits that can learn features,which is crucial for pretraining and transfer learning such as with BERT. Wepropose simple modifications to the standard parametrization to allow forfeature learning in the limit. Using the *Tensor Programs* technique, we deriveexplicit formulas for such limits. On Word2Vec and few-shot learning onOmniglot via MAML, two canonical tasks that rely crucially on feature learning,we compute these limits exactly. We find that they outperform both NTKbaselines and finite-width networks, with the latter approaching theinfinite-width feature learning performance as width increases.  More generally, we classify a natural space of neural networkparametrizations that generalizes standard, NTK, and Mean Fieldparametrizations. We show 1) any parametrization in this space either admitsfeature learning or has an infinite-width training dynamics given by kernelgradient descent, but not both; 2) any such infinite-width limit can becomputed using the Tensor Programs technique. Code for our experiments can befound at github.com/edwardjhu/TP4.

Feature Learning in Infinite-Width Neural Networks