English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we present some theoretical work to explain why simplegradient descent methods are so successful in solving non-convex optimizationproblems in learning large-scale neural networks (NN). After introducing amathematical tool called canonical space, we have proved that the objectivefunctions in learning NNs are convex in the canonical model space. We furtherelucidate that the gradients between the original NN model space and thecanonical space are related by a pointwise linear transformation, which isrepresented by the so-called disparity matrix. Furthermore, we have proved thatgradient descent methods surely converge to a global minimum of zero lossprovided that the disparity matrices maintain full rank. If this full-rankcondition holds, the learning of NNs behaves in the same way as normal convexoptimization. At last, we have shown that the chance to have singular disparitymatrices is extremely slim in large NNs. In particular, when over-parameterizedNNs are randomly initialized, the gradient decent algorithms converge to aglobal minimum of zero loss in probability.

Why Learning of Large-Scale Neural Networks Behaves Like Convex Optimization