English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Many modern neural network architectures are trained in an overparameterizedregime where the parameters of the model exceed the size of the trainingdataset. Sufficiently overparameterized neural network architectures inprinciple have the capacity to fit any set of labels including random noise.However, given the highly nonconvex nature of the training landscape it is notclear what level and kind of overparameterization is required for first ordermethods to converge to a global optima that perfectly interpolate any labels. Anumber of recent theoretical works have shown that for very wide neuralnetworks where the number of hidden units is polynomially large in the size ofthe training data gradient descent starting from a random initialization doesindeed converge to a global optima. However, in practice much more moderatelevels of overparameterization seems to be sufficient and in many casesoverparameterized models seem to perfectly interpolate the training data assoon as the number of parameters exceed the size of the training data by aconstant factor. Thus there is a huge gap between the existing theoreticalliterature and practical experiments. In this paper we take a step towardsclosing this gap. Focusing on shallow neural nets and smooth activations, weshow that (stochastic) gradient descent when initialized at random converges ata geometric rate to a nearby global optima as soon as the square-root of thenumber of network parameters exceeds the size of the training data. Our resultsalso benefit from a fast convergence rate and continue to hold fornon-differentiable activations such as Rectified Linear Units (ReLUs).

Towards moderate overparameterization: global convergence guarantees for training shallow neural networks