English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Stochastic differential equations (SDEs) have been shown recently to wellcharacterize the dynamics of training machine learning models with SGD. Thisprovides two opportunities for better understanding the generalizationbehaviour of SGD through its SDE approximation. First, under the SDEcharacterization, SGD may be regarded as the full-batch gradient descent withGaussian gradient noise. This allows the application of the generalizationbounds developed by Xu & Raginsky (2017) to analyzing the generalizationbehaviour of SGD, resulting in upper bounds in terms of the mutual informationbetween the training set and the training trajectory. Second, under mildassumptions, it is possible to obtain an estimate of the steady-state weightdistribution of SDE. Using this estimate, we apply the PAC-Bayes-likeinformation-theoretic bounds developed in both Xu & Raginsky (2017) and Negreaet al. (2019) to obtain generalization upper bounds in terms of the KLdivergence between the steady-state weight distribution of SGD with respect toa prior distribution. Among various options, one may choose the prior as thesteady-state weight distribution obtained by SGD on the same training set butwith one example held out. In this case, the bound can be elegantly expressedusing the influence function (Koh & Liang, 2017), which suggests that thegeneralization of the SGD is related to the stability of SGD. Various insightsare presented along the development of these bounds, which are subsequentlyvalidated numerically.

Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States