English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We give a new separation result between the generalization performance ofstochastic gradient descent (SGD) and of full-batch gradient descent (GD) inthe fundamental stochastic convex optimization model. While for SGD it iswell-known that $O(1/\epsilon^2)$ iterations suffice for obtaining a solutionwith $\epsilon$ excess expected risk, we show that with the same number ofsteps GD may overfit and emit a solution with $\Omega(1)$ generalization error.Moreover, we show that in fact $\Omega(1/\epsilon^4)$ iterations are necessaryfor GD to match the generalization performance of SGD, which is also tight dueto recent work by Bassily et al. (2020). We further discuss how regularizingthe empirical risk minimized by GD essentially does not change the aboveresult, and revisit the concepts of stability, implicit bias and the role ofthe learning algorithm in generalization.

SGD Generalizes Better Than GD (And Regularization Doesn't Help)