English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Sharpness aware minimization (SAM) optimizer has been extensively explored asit can generalize better for training deep neural networks via introducingextra perturbation steps to flatten the landscape of deep learning models.Integrating SAM with adaptive learning rate and momentum acceleration, dubbedAdaSAM, has already been explored empirically to train large-scale deep neuralnetworks without theoretical guarantee due to the triple difficulties inanalyzing the coupled perturbation step, adaptive learning rate and momentumstep. In this paper, we try to analyze the convergence rate of AdaSAM in thestochastic non-convex setting. We theoretically show that AdaSAM admits a$\mathcal{O}(1/\sqrt{bT})$ convergence rate, which achieves linear speedupproperty with respect to mini-batch size $b$. Specifically, to decouple thestochastic gradient steps with the adaptive learning rate and perturbedgradient, we introduce the delayed second-order momentum term to decompose themto make them independent while taking an expectation during the analysis. Thenwe bound them by showing the adaptive learning rate has a limited range, whichmakes our analysis feasible. To the best of our knowledge, we are the first toprovide the non-trivial convergence rate of SAM with an adaptive learning rateand momentum acceleration. At last, we conduct several experiments on severalNLP tasks, which show that AdaSAM could achieve superior performance comparedwith SGD, AMSGrad, and SAM optimizers.

AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks