English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This theoretical paper is devoted to developing a rigorous theory fordemystifying the global convergence phenomenon in a challenging scenario:learning over-parameterized Rectified Linear Unit (ReLU) nets for very highdimensional dataset under very mild assumptions. A major ingredient of ouranalysis is a fine-grained analysis of random activation matrices. Theessential virtue of dissecting activation matrices is that it bridges thedynamics of optimization and angular distribution in high-dimensional dataspace. This angle-based detailed analysis leads to asymptotic characterizationsof gradient norm and directional curvature of objective function at eachgradient descent iteration, revealing that the empirical loss function enjoysnice geometrical properties in the overparameterized setting. Along the way, wesignificantly improve existing theoretical bounds on both over-parameterizationcondition and learning rate with very mild assumptions for learning very highdimensional data. Moreover, we uncover the role of the geometrical and spectralproperties of the input data in determining desired over-parameterization sizeand global convergence rate. All these clues allow us to discover a novelgeometric picture of nonconvex optimization in deep learning: angulardistribution in high-dimensional data space $\mapsto$ spectrums ofoverparameterized activation matrices $\mapsto$ favorable geometricalproperties of empirical loss landscape $\mapsto$ global convergence phenomenon.Furthremore, our theoretical results imply that gradient-based nonconvexoptimization algorithms have much stronger statistical guarantees with muchmilder over-parameterization condition than exisiting theory states forlearning very high dimensional data, which is rarely explored so far.

Demystifying the Global Convergence Puzzle of Learning Over-parameterized ReLU Nets in Very High Dimensions