English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study the loss surface of DNNs with $L_{2}$ regularization. We show thatthe loss in terms of the parameters can be reformulated into a loss in terms ofthe layerwise activations $Z_{\ell}$ of the training set. This reformulationreveals the dynamics behind feature learning: each hidden representations$Z_{\ell}$ are optimal w.r.t. to an attraction/repulsion problem andinterpolate between the input and output representations, keeping as littleinformation from the input as necessary to construct the activation of the nextlayer. For positively homogeneous non-linearities, the loss can be furtherreformulated in terms of the covariances of the hidden representations, whichtakes the form of a partially convex optimization over a convex cone.  This second reformulation allows us to prove a sparsity result forhomogeneous DNNs: any local minimum of the $L_{2}$-regularized loss can beachieved with at most $N(N+1)$ neurons in each hidden layer (where $N$ is thesize of the training set). We show that this bound is tight by giving anexample of a local minimum that requires $N^{2}/4$ hidden neurons. But we alsoobserve numerically that in more traditional settings much less than $N^{2}$neurons are required to reach the minima.

Feature Learning in $L_{2}$-regularized DNNs: Attraction/Repulsion and Sparsity