English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study the effect of width on the dynamics of feature-learning neuralnetworks across a variety of architectures and datasets. Early in training,wide neural networks trained on online data have not only identical loss curvesbut also agree in their point-wise test predictions throughout training. Forsimple tasks such as CIFAR-5m this holds throughout training for networks ofrealistic widths. We also show that structural properties of the models,including internal representations, preactivation distributions, edge ofstability phenomena, and large learning rate effects are consistent acrosslarge widths. This motivates the hypothesis that phenomena seen in realisticmodels can be captured by infinite-width, feature-learning limits. For hardertasks (such as ImageNet and language modeling), and later training times,finite-width deviations grow systematically. Two distinct effects cause thesedeviations across widths. First, the network output hasinitialization-dependent variance scaling inversely with width, which can beremoved by ensembling networks. We observe, however, that ensembles of narrowernetworks perform worse than a single wide network. We call this the bias ofnarrower width. We conclude with a spectral perspective on the origin of thisfinite-width bias.

Feature-Learning Networks Are Consistent Across Widths At Realistic Scales