English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The optimization step in many machine learning problems rarely relies onvanilla gradient descent but it is common practice to use momentum-basedaccelerated methods. Despite these algorithms being widely applied to arbitraryloss functions, their behaviour in generically non-convex, high dimensionallandscapes is poorly understood. In this work, we use dynamical mean fieldtheory techniques to describe analytically the average dynamics of thesemethods in a prototypical non-convex model: the (spiked) matrix-tensor model.We derive a closed set of equations that describe the behaviour of heavy-ballmomentum and Nesterov acceleration in the infinite dimensional limit. Bynumerical integration of these equations, we observe that these methods speedup the dynamics but do not improve the algorithmic threshold with respect togradient descent in the spiked model.

Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic High-Dimensional Non-Convex Problems