English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

For infinitesimal learning rates, stochastic gradient descent (SGD) followsthe path of gradient flow on the full batch loss function. However moderatelylarge learning rates can achieve higher test accuracies, and thisgeneralization benefit is not explained by convergence bounds, since thelearning rate which maximizes test accuracy is often larger than the learningrate which minimizes training loss. To interpret this phenomenon we prove thatfor SGD with random shuffling, the mean SGD iterate also stays close to thepath of gradient flow if the learning rate is small and finite, but on amodified loss. This modified loss is composed of the original loss function andan implicit regularizer, which penalizes the norms of the minibatch gradients.Under mild assumptions, when the batch size is small the scale of the implicitregularization term is proportional to the ratio of the learning rate to thebatch size. We verify empirically that explicitly including the implicitregularizer in the loss can enhance the test accuracy when the learning rate issmall.

On the Origin of Implicit Regularization in Stochastic Gradient Descent