English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In this paper, we investigate the impact of stochasticity and large stepsizeson the implicit regularisation of gradient descent (GD) and stochastic gradientdescent (SGD) over diagonal linear networks. We prove the convergence of GD andSGD with macroscopic stepsizes in an overparametrised regression setting andcharacterise their solutions through an implicit regularisation problem. Ourcrisp characterisation leads to qualitative insights about the impact ofstochasticity and stepsizes on the recovered solution. Specifically, we showthat large stepsizes consistently benefit SGD for sparse regression problems,while they can hinder the recovery of sparse solutions for GD. These effectsare magnified for stepsizes in a tight window just below the divergencethreshold, in the "edge of stability" regime. Our findings are supported byexperimental results.

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability