English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Stochastic gradient descent is a workhorse for training deep neural networksdue to its excellent generalization performance. Several studies demonstratedthis success is attributed to the implicit bias of the method that prefers aflat minimum and developed new methods based on this perspective. Recently,Izmailov et al. (2018) empirically observed that an averaged stochasticgradient descent with a large step size can bring out the implicit bias moreeffectively and can converge more stably to a flat minimum than the vanillastochastic gradient descent. In our work, we theoretically justify thisobservation by showing that the averaging scheme improves the bias-optimizationtradeoff coming from the stochastic gradient noise: a large step size amplifiesthe bias but makes convergence unstable, and vice versa. Specifically, we showthat the averaged stochastic gradient descent can get closer to a solution of apenalized objective on the sharpness than the vanilla stochastic gradientdescent using the same step size under certain conditions. In experiments, weverify our theory and show this learning scheme significantly improvesperformance.

Parameter Averaging for SGD Stabilizes the Implicit Bias towards Flat Regions