English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The ability of deep neural networks to generalise well even when theyinterpolate their training data has been explained using various "simplicitybiases". These theories postulate that neural networks avoid overfitting byfirst learning simple functions, say a linear classifier, before learning morecomplex, non-linear functions. Meanwhile, data structure is also recognised asa key ingredient for good generalisation, yet its role in simplicity biases isnot yet understood. Here, we show that neural networks trained using stochasticgradient descent initially classify their inputs using lower-order inputstatistics, like mean and covariance, and exploit higher-order statistics onlylater during training. We first demonstrate this distributional simplicity bias(DSB) in a solvable model of a neural network trained on synthetic data. Weempirically demonstrate DSB in a range of deep convolutional networks andvisual transformers trained on CIFAR10, and show that it even holds in networkspre-trained on ImageNet. We discuss the relation of DSB to other simplicitybiases and consider its implications for the principle of Gaussian universalityin learning.

Neural networks trained with SGD learn distributions of increasing complexity