English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Adaptive gradient methods (AGMs) have become popular in optimizing thenonconvex problems in deep learning area. We revisit AGMs and identify that theadaptive learning rate (A-LR) used by AGMs varies significantly across thedimensions of the problem over epochs (i.e., anisotropic scale), which may leadto issues in convergence and generalization. All existing modified AGMsactually represent efforts in revising the A-LR. Theoretically, we provide anew way to analyze the convergence of AGMs and prove that the convergence rateof \textsc{Adam} also depends on its hyper-parameter $\epsilon$, which has beenoverlooked previously. Based on these two facts, we propose a new AGM bycalibrating the A-LR with an activation ({\em softplus}) function, resulting inthe \textsc{Sadam} and \textsc{SAMSGrad} methods \footnote{Code is available athttps://github.com/neilliang90/Sadam.git.}. We further prove that thesealgorithms enjoy better convergence speed under nonconvex, non-strongly convex,and Polyak-{\L}ojasiewicz conditions compared with \textsc{Adam}. Empiricalstudies support our observation of the anisotropic A-LR and show that theproposed methods outperform existing AGMs and generalize even better thanS-Momentum in multiple deep learning tasks.

Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM