English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study the dynamics of optimization and the generalization properties ofone-hidden layer neural networks with quadratic activation function in theover-parametrized regime where the layer width $m$ is larger than the inputdimension $d$.  We consider a teacher-student scenario where the teacher has the samestructure as the student with a hidden layer of smaller width $m^*\le m$.  We describe how the empirical loss landscape is affected by the number $n$ ofdata samples and the width $m^*$ of the teacher network. In particular wedetermine how the probability that there be no spurious minima on the empiricalloss depends on $n$, $d$, and $m^*$, thereby establishing conditions underwhich the neural network can in principle recover the teacher.  We also show that under the same conditions gradient descent dynamics on theempirical loss converges and leads to small generalization error, i.e. itenables recovery in practice.  Finally we characterize the time-convergence rate of gradient descent in thelimit of a large number of samples.  These results are confirmed by numerical experiments.

Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions