English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In recent years, stochastic gradient descent (SGD) based techniques hasbecome the standard tools for training neural networks. However, formaltheoretical understanding of why SGD can train neural networks in practice islargely missing.  In this paper, we make progress on understanding this mystery by providing aconvergence analysis for SGD on a rich subset of two-layer feedforward networkswith ReLU activations. This subset is characterized by a special structurecalled "identity mapping". We prove that, if input follows from Gaussiandistribution, with standard $O(1/\sqrt{d})$ initialization of the weights, SGDconverges to the global minimum in polynomial number of steps. Unlike normalvanilla networks, the "identity mapping" makes our network asymmetric and thusthe global minimum is unique. To complement our theory, we are also able toshow experimentally that multi-layer networks with this mapping have betterperformance compared with normal vanilla networks.  Our convergence theorem differs from traditional non-convex optimizationtechniques. We show that SGD converges to optimal in "two phases": In phase I,the gradient points to the wrong direction, however, a potential function $g$gradually decreases. Then in phase II, SGD enters a nice one point convexregion and converges. We also show that the identity mapping is necessary forconvergence, as it moves the initial point to a better place for optimization.Experiment verifies our claims.

Convergence Analysis of Two-layer Neural Networks with ReLU Activation