English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We make contributions towards improving adaptive-optimizer performance. Ourimprovements are based on suppression of the range of adaptive stepsizes in theAdaBelief optimizer. Firstly, we show that the particular placement of theparameter epsilon within the update expressions of AdaBelief reduces the rangeof the adaptive stepsizes, making AdaBelief closer to SGD with momentum.Secondly, we extend AdaBelief by further suppressing the range of the adaptivestepsizes. To achieve the above goal, we perform mutual layerwise vectorprojections between the gradient g_t and its first momentum m_t before usingthem to estimate the second momentum. The new optimization method is referredto as Aida. Thirdly, extensive experimental results show that Aida outperformsnine optimizers when training transformers and LSTMs for NLP, and VGG andResNet for image classification over CIAF10 and CIFAR100 while matching thebest performance of the nine methods when training WGAN-GP models for imagegeneration tasks. Furthermore, Aida produces higher validation accuracies thanAdaBelief for training ResNet18 over ImageNet. Code is available <ahref="https://github.com/guoqiang-x-zhang/AidaOptimizer">at this URL</a>

A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range