English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

It is not clear yet why ADAM-alike adaptive gradient algorithms suffer fromworse generalization performance than SGD despite their faster training speed.This work aims to provide understandings on this generalization gap byanalyzing their local convergence behaviors. Specifically, we observe the heavytails of gradient noise in these algorithms. This motivates us to analyze thesealgorithms through their Levy-driven stochastic differential equations (SDEs)because of the similar convergence behaviors of an algorithm and its SDE. Thenwe establish the escaping time of these SDEs from a local basin. The resultshows that (1) the escaping time of both SGD and ADAM~depends on the Radonmeasure of the basin positively and the heaviness of gradient noise negatively;(2) for the same basin, SGD enjoys smaller escaping time than ADAM, mainlybecause (a) the geometry adaptation in ADAM~via adaptively scaling eachgradient coordinate well diminishes the anisotropic structure in gradient noiseand results in larger Radon measure of a basin; (b) the exponential gradientaverage in ADAM~smooths its gradient and leads to lighter gradient noise tailsthan SGD. So SGD is more locally unstable than ADAM~at sharp minima defined asthe minima whose local basins have small Radon measure, and can better escapefrom them to flatter ones with larger Radon measure. As flat minima here whichoften refer to the minima at flat or asymmetric basins/valleys often generalizebetter than sharp ones , our result explains the better generalizationperformance of SGD over ADAM. Finally, experimental results confirm ourheavy-tailed gradient noise assumption and theoretical affirmation.

Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning