English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Adaptive gradient methods such as Adam have gained extreme popularity due totheir success in training complex neural networks and less sensitivity tohyperparameter tuning compared to SGD. However, it has been recently shown thatAdam can fail to converge and might cause poor generalization -- this lead tothe design of new, sophisticated adaptive methods which attempt to generalizewell while being theoretically reliable. In this technical report we focus onAdaBound, a promising, recently proposed optimizer. We present a stochasticconvex problem for which AdaBound can provably take arbitrarily long toconverge in terms of a factor which is not accounted for in the convergencerate guarantee of Luo et al. (2019). We present a new $O(\sqrt T)$ regretguarantee under different assumptions on the bound functions, and provideempirical results on CIFAR suggesting that a specific form of momentum SGD canmatch AdaBound's performance while having less hyperparameters and lowercomputational costs.

On the Convergence of AdaBound and its Connection to SGD