English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider gradient descent with `momentum', a widely used method for lossfunction minimization in machine learning. This method is often used with`Nesterov acceleration', meaning that the gradient is evaluated not at thecurrent position in parameter space, but at the estimated position after onestep. In this work, we show that the algorithm can be improved by extendingthis `acceleration' --- by using the gradient at an estimated position severalsteps ahead rather than just one step ahead. How far one looks ahead in this`super-acceleration' algorithm is determined by a new hyperparameter.Considering a one-parameter quadratic loss function, the optimal value of thesuper-acceleration can be exactly calculated and analytically estimated. Weshow explicitly that super-accelerating the momentum algorithm is beneficial,not only for this idealized problem, but also for several synthetic losslandscapes and for the MNIST classification task with neural networks.Super-acceleration is also easy to incorporate into adaptive algorithms likeRMSProp or Adam, and is shown to improve these algorithms.

Gradient descent with momentum --- to accelerate or to super-accelerate?