English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Stochastic gradient descent (SGD) exhibits strong algorithmic regularizationeffects in practice, which has been hypothesized to play an important role inthe generalization of modern machine learning approaches. In this work, we seekto understand these issues in the simpler setting of linear regression(including both underparameterized and overparameterized regimes), where ourgoal is to make sharp instance-based comparisons of the implicit regularizationafforded by (unregularized) average SGD with the explicit regularization ofridge regression. For a broad class of least squares problem instances (thatare natural in high-dimensional settings), we show: (1) for every probleminstance and for every ridge parameter, (unregularized) SGD, when provided withlogarithmically more samples than that provided to the ridge algorithm,generalizes no worse than the ridge solution (provided SGD uses a tunedconstant stepsize); (2) conversely, there exist instances (in this wide problemclass) where optimally-tuned ridge regression requires quadratically moresamples than SGD in order to have the same generalization performance. Takentogether, our results show that, up to the logarithmic factors, thegeneralization performance of SGD is always no worse than that of ridgeregression in a wide range of overparameterized problems, and, in fact, couldbe much better for some problem instances. More generally, our results show howalgorithmic regularization has important consequences even in simpler(overparameterized) convex settings.

The Benefits of Implicit Regularization from SGD in Least Squares Problems