English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We showcase important features of the dynamics of the Stochastic GradientDescent (SGD) in the training of neural networks. We present empiricalobservations that commonly used large step sizes (i) lead the iterates to jumpfrom one side of a valley to the other causing loss stabilization, and (ii)this stabilization induces a hidden stochastic dynamics orthogonal to thebouncing directions that biases it implicitly toward sparse predictors.Furthermore, we show empirically that the longer large step sizes keep SGD highin the loss landscape valleys, the better the implicit regularization canoperate and find sparse representations. Notably, no explicit regularization isused so that the regularization effect comes solely from the SGD trainingdynamics influenced by the step size schedule. Therefore, these observationsunveil how, through the step size schedules, both gradient and noise drivetogether the SGD dynamics through the loss landscape of neural networks. Wejustify these findings theoretically through the study of simple neural networkmodels as well as qualitative arguments inspired from stochastic processes.Finally, this analysis allows us to shed a new light on some common practiceand observed phenomena when training neural networks. The code of ourexperiments is available at https://github.com/tml-epfl/sgd-sparse-features.

SGD with Large Step Sizes Learns Sparse Features