English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider the optimization problem associated with training simple ReLUneural networks of the form $\mathbf{x}\mapsto\sum_{i=1}^{k}\max\{0,\mathbf{w}_i^\top \mathbf{x}\}$ with respect to thesquared loss. We provide a computer-assisted proof that even if the inputdistribution is standard Gaussian, even if the dimension is arbitrarily large,and even if the target values are generated by such a network, with orthonormalparameter vectors, the problem can still have spurious local minima once $6\lek\le 20$. By a concentration of measure argument, this implies that in highinput dimensions, \emph{nearly all} target networks of the relevant sizes leadto spurious local minima. Moreover, we conduct experiments which show that theprobability of hitting such local minima is quite high, and increasing with thenetwork size. On the positive side, mild over-parameterization appears todrastically reduce such local minima, indicating that an over-parameterizationassumption is necessary to get a positive result in this setting.

Spurious Local Minima are Common in Two-Layer ReLU Neural Networks