English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

We investigate gradient descent training of wide neural networks and thecorresponding implicit bias in function space. For univariate regression, weshow that the solution of training a width-$n$ shallow ReLU network is within$n^{- 1/2}$ of the function which fits the training data and whose differencefrom the initial function has the smallest 2-norm of the second derivativeweighted by a curvature penalty that depends on the probability distributionthat is used to initialize the network parameters. We compute the curvaturepenalty function explicitly for various common initialization procedures. Forinstance, asymmetric initialization with a uniform distribution yields aconstant curvature penalty, and thence the solution function is the naturalcubic spline interpolation of the training data. \hj{For stochastic gradientdescent we obtain the same implicit bias result.} We obtain a similar resultfor different activation functions. For multivariate regression we show ananalogous result, whereby the second derivative is replaced by the Radontransform of a fractional Laplacian. For initialization schemes that yield aconstant penalty function, the solutions are polyharmonic splines. Moreover, weshow that the training trajectories are captured by trajectories of smoothingsplines with decreasing regularization strength.

Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networks