English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study how neural networks compress uninformative input space in modelswhere data lie in $d$ dimensions, but whose label only vary within a linearmanifold of dimension $d_\parallel < d$. We show that for a one-hidden layernetwork initialized with infinitesimal weights (i.e. in the feature learningregime) trained with gradient descent, the first layer of weights evolve tobecome nearly insensitive to the $d_\perp=d-d_\parallel$ uninformativedirections. These are effectively compressed by a factor $\lambda\sim\sqrt{p}$, where $p$ is the size of the training set. We quantify the benefitof such a compression on the test error $\epsilon$. For large initialization ofthe weights (the lazy training regime), no compression occurs and for regularboundaries separating labels we find that $\epsilon \sim p^{-\beta}$, with$\beta_\text{Lazy} = d / (3d-2)$. Compression improves the learning curves sothat $\beta_\text{Feature} = (2d-1)/(3d-2)$ if $d_\parallel = 1$ and$\beta_\text{Feature} = (d + d_\perp/2)/(3d-2)$ if $d_\parallel > 1$. We testthese predictions for a stripe model where boundaries are parallel interfaces($d_\parallel=1$) as well as for a cylindrical boundary ($d_\parallel=2$). Nextwe show that compression shapes the Neural Tangent Kernel (NTK) evolution intime, so that its top eigenvectors become more informative and display a largerprojection on the labels. Consequently, kernel learning with the frozen NTK atthe end of training outperforms the initial NTK. We confirm these predictionsboth for a one-hidden layer FC network trained on the stripe model and for a16-layers CNN trained on MNIST, for which we also find$\beta_\text{Feature}>\beta_\text{Lazy}$.

Geometric compression of invariant manifolds in neural nets