English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The loss surface of deep neural networks has recently attracted interest inthe optimization and machine learning communities as a prime example ofhigh-dimensional non-convex problem. Some insights were recently gained usingspin glass models and mean-field approximations, but at the expense of stronglysimplifying the nonlinear nature of the model.  In this work, we do not make any such assumption and study conditions on thedata distribution and model architecture that prevent the existence of badlocal minima. Our theoretical work quantifies and formalizes two important\emph{folklore} facts: (i) the landscape of deep linear networks has aradically different topology from that of deep half-rectified ones, and (ii)that the energy landscape in the non-linear case is fundamentally controlled bythe interplay between the smoothness of the data distribution and modelover-parametrization. Our main theoretical contribution is to prove thathalf-rectified single layer networks are asymptotically connected, and weprovide explicit bounds that reveal the aforementioned interplay.  The conditioning of gradient descent is the next challenge we address. Westudy this question through the geometry of the level sets, and we introduce analgorithm to efficiently estimate the regularity of such sets on large-scalenetworks. Our empirical results show that these level sets remain connectedthroughout all the learning phase, suggesting a near convex behavior, but theybecome exponentially more curvy as the energy level decays, in accordance towhat is observed in practice with very low curvature attractors.

Topology and Geometry of Half-Rectified Network Optimization