English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

It is well-known that overparametrized neural networks trained usinggradient-based methods quickly achieve small training error with appropriatehyperparameter settings. Recent papers have proved this statement theoreticallyfor highly overparametrized networks under reasonable assumptions. Theseresults either assume that the activation function is ReLU or they cruciallydepend on the minimum eigenvalue of a certain Gram matrix depending on thedata, random initialization and the activation function. In the later case,existing works only prove that this minimum eigenvalue is non-zero and do notprovide quantitative bounds. On the empirical side, a contemporary line ofinvestigations has proposed a number of alternative activation functions whichtend to perform better than ReLU at least in some settings but no clearunderstanding has emerged. This state of affairs underscores the importance oftheoretically understanding the impact of activation functions on training. Inthe present paper, we provide theoretical results about the effect ofactivation function on the training of highly overparametrized 2-layer neuralnetworks. A crucial property that governs the performance of an activation iswhether or not it is smooth. For non-smooth activations such as ReLU, SELU andELU, all eigenvalues of the associated Gram matrix are large under minimalassumptions on the data. For smooth activations such as tanh, swish andpolynomials, the situation is more complex. If the subspace spanned by the datahas small dimension then the minimum eigenvalue of the Gram matrix can be smallleading to slow training. But if the dimension is large and the data satisfiesanother mild condition, then the eigenvalues are large. If we allow deepnetworks, then the small data dimension is not a limitation provided that thedepth is sufficient. We discuss a number of extensions and applications ofthese results.

Effect of Activation Functions on the Training of Overparametrized Neural Nets