English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Natural gradient descent has proven effective at mitigating the effects ofpathological curvature in neural network optimization, but little is knowntheoretically about its convergence properties, especially for \emph{nonlinear}networks. In this work, we analyze for the first time the speed of convergenceof natural gradient descent on nonlinear neural networks with squared-errorloss. We identify two conditions which guarantee efficient convergence fromrandom initializations: (1) the Jacobian matrix (of network's output for alltraining cases with respect to the parameters) has full row rank, and (2) theJacobian matrix is stable for small perturbations around the initialization.For two-layer ReLU neural networks, we prove that these two conditions do infact hold throughout the training, under the assumptions of nondegenerateinputs and overparameterization. We further extend our analysis to more generalloss functions. Lastly, we show that K-FAC, an approximate natural gradientdescent method, also converges to global minima under the same assumptions, andwe give a bound on the rate of this convergence.

Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks