English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Stochastic gradient descent (SGD) has been shown to generalize well in manydeep learning applications. In practice, one often runs SGD with ageometrically decaying stepsize, i.e., a constant initial stepsize followed bymultiple geometric stepsize decay, and uses the last iterate as the output.This kind of SGD is known to be nearly minimax optimal for classicalfinite-dimensional linear regression problems (Ge et al., 2019). However, asharp analysis for the last iterate of SGD in the overparameterized setting isstill open. In this paper, we provide a problem-dependent analysis on the lastiterate risk bounds of SGD with decaying stepsize, for (overparameterized)linear regression problems. In particular, for last iterate SGD with (tail)geometrically decaying stepsize, we prove nearly matching upper and lowerbounds on the excess risk. Moreover, we provide an excess risk lower bound forlast iterate SGD with polynomially decaying stepsize and demonstrate theadvantage of geometrically decaying stepsize in an instance-wise manner, whichcomplements the minimax rate comparison made in prior works.

Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regression