English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Understanding the algorithmic bias of \emph{stochastic gradient descent}(SGD) is one of the key challenges in modern machine learning and deep learningtheory. Most of the existing works, however, focus on \emph{very small or eveninfinitesimal} learning rate regime, and fail to cover practical scenarioswhere the learning rate is \emph{moderate and annealing}. In this paper, wemake an initial attempt to characterize the particular regularization effect ofSGD in the moderate learning rate regime by studying its behavior foroptimizing an overparameterized linear regression problem. In this case, SGDand GD are known to converge to the unique minimum-norm solution; however, withthe moderate and annealing learning rate, we show that they exhibit different\emph{directional bias}: SGD converges along the large eigenvalue directions ofthe data matrix, while GD goes after the small eigenvalue directions.Furthermore, we show that such directional bias does matter when early stoppingis adopted, where the SGD output is nearly optimal but the GD output issuboptimal. Finally, our theory explains several folk arts in practice used forSGD hyperparameter tuning, such as (1) linearly scaling the initial learningrate with batch size; and (2) overrunning SGD with high learning rate even whenthe loss stops decreasing.

Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate