English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Understanding the reasons for the success of deep neural networks trainedusing stochastic gradient-based methods is a key open problem for the nascenttheory of deep learning. The types of data where these networks are mostsuccessful, such as images or sequences of speech, are characterised byintricate correlations. Yet, most theoretical work on neural networks does notexplicitly model training data, or assumes that elements of each data sampleare drawn independently from some factorised probability distribution. Theseapproaches are thus by construction blind to the correlation structure ofreal-world data sets and their impact on learning in neural networks. Here, weintroduce a generative model for structured data sets that we call the hiddenmanifold model (HMM). The idea is to construct high-dimensional inputs that lieon a lower-dimensional manifold, with labels that depend only on their positionwithin this manifold, akin to a single layer decoder or generator in agenerative adversarial network. We demonstrate that learning of the hiddenmanifold model is amenable to an analytical treatment by proving a "GaussianEquivalence Property" (GEP), and we use the GEP to show how the dynamics oftwo-layer neural networks trained using one-pass stochastic gradient descent iscaptured by a set of integro-differential equations that track the performanceof the network at all times. This permits us to analyse in detail how a neuralnetwork learns functions of increasing complexity during training, how itsperformance depends on its size and how it is impacted by parameters such asthe learning rate or the dimension of the hidden manifold.

Modelling the influence of data structure on learning in neural networks: the hidden manifold model