English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper we aim to formally explain the phenomenon of fast convergenceof SGD observed in modern machine learning. The key observation is that mostmodern learning architectures are over-parametrized and are trained tointerpolate the data by driving the empirical loss (classification andregression) close to zero. While it is still unclear why these interpolatedsolutions perform well on test data, we show that these regimes allow for fastconvergence of SGD, comparable in number of iterations to full gradientdescent.  For convex loss functions we obtain an exponential convergence bound for {\itmini-batch} SGD parallel to that for full gradient descent. We show that thereis a critical batch size $m^*$ such that: (a) SGD iteration with mini-batchsize $m\leq m^*$ is nearly equivalent to $m$ iterations of mini-batch size $1$(\emph{linear scaling regime}). (b) SGD iteration with mini-batch $m> m^*$ isnearly equivalent to a full gradient descent iteration (\emph{saturationregime}).  Moreover, for the quadratic loss, we derive explicit expressions for theoptimal mini-batch and step size and explicitly characterize the two regimesabove. The critical mini-batch size can be viewed as the limit for effectivemini-batch parallelization. It is also nearly independent of the data size,implying $O(n)$ acceleration over GD per unit of computation. We giveexperimental evidence on real data which closely follows our theoreticalanalyses.  Finally, we show how our results fit in the recent developments in trainingdeep neural networks and discuss connections to adaptive rates for SGD andvariance reduction.

The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning