English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Understanding the properties of neural networks trained via stochasticgradient descent (SGD) is at the heart of the theory of deep learning. In thiswork, we take a mean-field view, and consider a two-layer ReLU network trainedvia SGD for a univariate regularized regression problem. Our main result isthat SGD is biased towards a simple solution: at convergence, the ReLU networkimplements a piecewise linear map of the inputs, and the number of "knot"points - i.e., points where the tangent of the ReLU network estimator changes -between two consecutive training inputs is at most three. In particular, as thenumber of neurons of the network grows, the SGD dynamics is captured by thesolution of a gradient flow and, at convergence, the distribution of theweights approaches the unique minimizer of a related free energy, which has aGibbs form. Our key technical contribution consists in the analysis of theestimator resulting from this minimizer: we show that its second derivativevanishes everywhere, except at some specific locations which represent the"knot" points. We also provide empirical evidence that knots at locationsdistinct from the data points might occur, as predicted by our theory.

Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks