English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Ever since Reddi et al. 2018 pointed out the divergence issue of Adam, manynew variants have been designed to obtain convergence. However, vanilla Adamremains exceptionally popular and it works well in practice. Why is there a gapbetween theory and practice? We point out there is a mismatch between thesettings of theory and practice: Reddi et al. 2018 pick the problem afterpicking the hyperparameters of Adam, i.e., $(\beta_1, \beta_2)$; whilepractical applications often fix the problem first and then tune $(\beta_1,\beta_2)$. Due to this observation, we conjecture that the empiricalconvergence can be theoretically justified, only if we change the order ofpicking the problem and hyperparameter. In this work, we confirm thisconjecture. We prove that, when $\beta_2$ is large and $\beta_1 <\sqrt{\beta_2}<1$, Adam converges to the neighborhood of critical points. Thesize of the neighborhood is propositional to the variance of stochasticgradients. Under an extra condition (strong growth condition), Adam convergesto critical points. It is worth mentioning that our results cover a wide rangeof hyperparameters: as $\beta_2$ increases, our convergence result can coverany $\beta_1 \in [0,1)$ including $\beta_1=0.9$, which is the default settingin deep learning libraries. To our knowledge, this is the first result showingthat Adam can converge without any modification on its update rules. Further,our analysis does not require assumptions of bounded gradients or bounded2nd-order momentum. When $\beta_2$ is small, we further point out a largeregion of $(\beta_1,\beta_2)$ where Adam can diverge to infinity. Ourdivergence result considers the same setting as our convergence result,indicating a phase transition from divergence to convergence when increasing$\beta_2$. These positive and negative results can provide suggestions on howto tune Adam hyperparameters.

Adam Can Converge Without Any Modification On Update Rules