English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We prove a general Embedding Principle of loss landscape of deep neuralnetworks (NNs) that unravels a hierarchical structure of the loss landscape ofNNs, i.e., loss landscape of an NN contains all critical points of all thenarrower NNs. This result is obtained by constructing a class of criticalembeddings which map any critical point of a narrower NN to a critical point ofthe target NN with the same output function. By discovering a wide class ofgeneral compatible critical embeddings, we provide a gross estimate of thedimension of critical submanifolds embedded from critical points of narrowerNNs. We further prove an irreversiblility property of any critical embeddingthat the number of negative/zero/positive eigenvalues of the Hessian matrix ofa critical point may increase but never decrease as an NN becomes wider throughthe embedding. Using a special realization of general compatible criticalembedding, we prove a stringent necessary condition for being a "truly-bad"critical point that never becomes a strict-saddle point through any criticalembedding. This result implies the commonplace of strict-saddle points in wideNNs, which may be an important reason underlying the easy optimization of wideNNs widely observed in practice.

Embedding Principle: a hierarchical structure of loss landscape of deep neural networks