English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We analyze the dynamics of large batch stochastic gradient descent withmomentum (SGD+M) on the least squares problem when both the number of samplesand dimensions are large. In this setting, we show that the dynamics of SGD+Mconverge to a deterministic discrete Volterra equation as dimension increases,which we analyze. We identify a stability measurement, the implicitconditioning ratio (ICR), which regulates the ability of SGD+M to acceleratethe algorithm. When the batch size exceeds this ICR, SGD+M converges linearlyat a rate of $\mathcal{O}(1/\sqrt{\kappa})$, matching optimal full-batchmomentum (in particular performing as well as a full-batch but with a fractionof the size). For batch sizes smaller than the ICR, in contrast, SGD+M hasrates that scale like a multiple of the single batch SGD rate. We give explicitchoices for the learning rate and momentum parameter in terms of the Hessianspectra that achieve this performance.

Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions