English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider the problem of universal approximation of functions by two-layerneural nets with random weights that are "nearly Gaussian" in the sense ofKullback-Leibler divergence. This problem is motivated by recent works on lazytraining, where the weight updates generated by stochastic gradient descent donot move appreciably from the i.i.d. Gaussian initialization. We first considerthe mean-field limit, where the finite population of neurons in the hiddenlayer is replaced by a continual ensemble, and show that our problem can bephrased as global minimization of a free-energy functional on the space ofprobability measures over the weights. This functional trades off the $L^2$approximation risk against the KL divergence with respect to a centeredGaussian prior. We characterize the unique global minimizer and then constructa controlled nonlinear dynamics in the space of probability measures overweights that solves a McKean--Vlasov optimal control problem. This controlproblem is closely related to the Schr\"odinger bridge (or entropic optimaltransport) problem, and its value is proportional to the minimum of the freeenergy. Finally, we show that SGD in the lazy training regime (which can beensured by jointly tuning the variance of the Gaussian prior and the entropicregularization parameter) serves as a greedy approximation to the optimalMcKean--Vlasov distributional dynamics and provide quantitative guarantees onthe $L^2$ approximation error.

A mean-field theory of lazy training in two-layer neural nets: entropic regularization and controlled McKean-Vlasov dynamics