English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We propose a new framework, inspired by random matrix theory, for analyzingthe dynamics of stochastic gradient descent (SGD) when both number of samplesand dimensions are large. This framework applies to any fixed stepsize and thefinite sum setting. Using this new framework, we show that the dynamics of SGDon a least squares problem with random data become deterministic in the largesample and dimensional limit. Furthermore, the limiting dynamics are governedby a Volterra integral equation. This model predicts that SGD undergoes a phasetransition at an explicitly given critical stepsize that ultimately affects itsconvergence rate, which we also verify experimentally. Finally, when input datais isotropic, we provide explicit expressions for the dynamics and average-caseconvergence rates (i.e., the complexity of an algorithm averaged over allpossible inputs). These rates show significant improvement over the worst-casecomplexities.

SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality