English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Understanding the implicit bias of training algorithms is of crucialimportance in order to explain the success of overparametrised neural networks.In this paper, we study the dynamics of stochastic gradient descent overdiagonal linear networks through its continuous time version, namely stochasticgradient flow. We explicitly characterise the solution chosen by the stochasticflow and prove that it always enjoys better generalisation properties than thatof gradient flow. Quite surprisingly, we show that the convergence speed of thetraining loss controls the magnitude of the biasing effect: the slower theconvergence, the better the bias. To fully complete our analysis, we provideconvergence guarantees for the dynamics. We also give experimental resultswhich support our theoretical claims. Our findings highlight the fact thatstructured noise can induce better generalisation and they help explain thegreater performances observed in practice of stochastic gradient descent overgradient descent.

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity