English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Heavy Ball (HB) nowadays is one of the most popular momentum methods innon-convex optimization. It has been widely observed that incorporating theHeavy Ball dynamic in gradient-based methods accelerates the training processof modern machine learning models. However, the progress on establishing itstheoretical foundation of acceleration is apparently far behind its empiricalsuccess. Existing provable acceleration results are of the quadratic orclose-to-quadratic functions, as the current techniques of showing HB'sacceleration are limited to the case when the Hessian is fixed. In this work,we develop some new techniques that help show acceleration beyond quadratics,which is achieved by analyzing how the change of the Hessian at two consecutivetime points affects the convergence speed. Based on our technical results, aclass of Polyak-\L{}ojasiewicz (PL) optimization problems for which provableacceleration can be achieved via HB is identified. Moreover, our analysisdemonstrates a benefit of adaptively setting the momentum parameter.  (Update: 08/29/2023) Erratum is added in Appendix J. This is an updatedversion that fixes an issue in the previous version. An additional conditionneeds to be satisfied for the acceleration result of HB beyond quadratics inthis work, which naturally holds when the dimension is one or, more broadly,when the Hessian is diagonal. We elaborate on the issue in Appendix J.

Provable Acceleration of Heavy Ball beyond Quadratics for a Class of Polyak-\L{}ojasiewicz Functions when the Non-Convexity is Averaged-Out