English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Autoencoders are among the earliest introduced nonlinear models forunsupervised learning. Although they are widely adopted beyond research, it hasbeen a longstanding open problem to understand mathematically the featureextraction mechanism that trained nonlinear autoencoders provide.  In this work, we make progress in this problem by analyzing a class oftwo-layer weight-tied nonlinear autoencoders in the mean field framework. Upona suitable scaling, in the regime of a large number of neurons, the modelstrained with stochastic gradient descent are shown to admit a mean fieldlimiting dynamics. This limiting description reveals an asymptotically precisepicture of feature learning by these models: their training dynamics exhibitdifferent phases that correspond to the learning of different principalsubspaces of the data, with varying degrees of nonlinear shrinkage dependent onthe $\ell_{2}$-regularization and stopping time. While we prove these resultsunder an idealized assumption of (correlated) Gaussian data, experiments onreal-life data demonstrate an interesting match with the theory.  The autoencoder setup of interests poses a nontrivial mathematical challengeto proving these results. In this setup, the "Lipschitz" constants of themodels grow with the data dimension $d$. Consequently an adaptation of previousanalyses requires a number of neurons $N$ that is at least exponential in $d$.Our main technical contribution is a new argument which proves that therequired $N$ is only polynomial in $d$. We conjecture that $N\gg d$ issufficient and that $N$ is necessarily larger than a data-dependent intrinsicdimension, a behavior that is fundamentally different from previously studiedsetups.

Analysis of feature learning in weight-tied autoencoders via the mean field lens