English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Any continuous function $f^*$ can be approximated arbitrarily well by aneural network with sufficiently many neurons $k$. We consider the case when$f^*$ itself is a neural network with one hidden layer and $k$ neurons.Approximating $f^*$ with a neural network with $n< k$ neurons can thus be seenas fitting an under-parameterized "student" network with $n$ neurons to a"teacher" network with $k$ neurons. As the student has fewer neurons than theteacher, it is unclear, whether each of the $n$ student neurons should copy oneof the teacher neurons or rather average a group of teacher neurons. Forshallow neural networks with erf activation function and for the standardGaussian input distribution, we prove that "copy-average" configurations arecritical points if the teacher's incoming vectors are orthonormal and itsoutgoing weights are unitary. Moreover, the optimum among such configurationsis reached when $n-1$ student neurons each copy one teacher neuron and the$n$-th student neuron averages the remaining $k-n+1$ teacher neurons. For thestudent network with $n=1$ neuron, we provide additionally a closed-formsolution of the non-trivial critical point(s) for commonly used activationfunctions through solving an equivalent constrained optimization problem.Empirically, we find for the erf activation function that gradient flowconverges either to the optimal copy-average critical point or to another pointwhere each student neuron approximately copies a different teacher neuron.Finally, we find similar results for the ReLU activation function, suggestingthat the optimal solution of underparameterized networks has a universalstructure.

Should Under-parameterized Student Networks Copy or Average Teacher Weights?