English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study the optimization problem associated with fitting two-layer ReLUneural networks with respect to the squared loss, where labels are generated bya target network. We make use of the rich symmetry structure to develop a novelset of tools for studying families of spurious minima. In contrast to existingapproaches which operate in limiting regimes, our technique directly addressesthe nonconvex loss landscape for a finite number of inputs $d$ and neurons $k$,and provides analytic, rather than heuristic, information. In particular, wederive analytic estimates for the loss at different minima, and prove thatmodulo $O(d^{-1/2})$-terms the Hessian spectrum concentrates near smallpositive constants, with the exception of $\Theta(d)$ eigenvalues which growlinearly with~$d$. We further show that the Hessian spectrum at global andspurious minima coincide to $O(d^{-1/2})$-order, thus challenging our abilityto argue about statistical generalization through local curvature. Lastly, ourtechnique provides the exact \emph{fractional} dimensionality at which familiesof critical points turn from saddles into spurious minima. This makes possiblethe study of the creation and the annihilation of spurious minima usingpowerful tools from equivariant bifurcation theory.

Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II