English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Training deep neural networks with stochastic gradient descent (SGD) canoften achieve zero training loss on real-world tasks although the optimizationlandscape is known to be highly non-convex. To understand the success of SGDfor training deep neural networks, this work presents a mean-field analysis ofdeep residual networks, based on a line of works that interpret the continuumlimit of the deep residual network as an ordinary differential equation whenthe network capacity tends to infinity. Specifically, we propose a newcontinuum limit of deep residual networks, which enjoys a good landscape in thesense that every local minimizer is global. This characterization enables us toderive the first global convergence result for multilayer neural networks inthe mean-field regime. Furthermore, without assuming the convexity of the losslandscape, our proof relies on a zero-loss assumption at the global minimizerthat can be achieved when the model shares a universal approximation property.Key to our result is the observation that a deep residual network resembles ashallow network ensemble, i.e. a two-layer network. We bound the differencebetween the shallow network and our ResNet model via the adjoint sensitivitymethod, which enables us to apply existing mean-field analyses of two-layernetworks to deep networks. Furthermore, we propose several novel trainingschemes based on the new continuous model, including one training procedurethat switches the order of the residual blocks and results in strong empiricalperformance on the benchmark datasets.

A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth