English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Adam is one of the most influential adaptive stochastic algorithms fortraining deep neural networks, which has been pointed out to be divergent evenin the simple convex setting via a few simple counterexamples. Many attempts,such as decreasing an adaptive learning rate, adopting a big batch size,incorporating a temporal decorrelation technique, seeking an analogoussurrogate, \textit{etc.}, have been tried to promote Adam-type algorithms toconverge. In contrast with existing approaches, we introduce an alternativeeasy-to-check sufficient condition, which merely depends on the parameters ofthe base learning rate and combinations of historical second-order moments, toguarantee the global convergence of generic Adam for solving large-scalenon-convex stochastic optimization. This observation, coupled with thissufficient condition, gives much deeper interpretations on the divergence ofAdam. On the other hand, in practice, mini-Adam and distributed-Adam are widelyused without any theoretical guarantee. We further give an analysis on how thebatch size or the number of nodes in the distributed system affects theconvergence of Adam, which theoretically shows that mini-batch and distributedAdam can be linearly accelerated by using a larger mini-batch size or a largernumber of nodes.At last, we apply the generic Adam and mini-batch Adam with thesufficient condition for solving the counterexample and training several neuralnetworks on various real-world datasets. Experimental results are exactly inaccord with our theoretical analysis.

Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration