English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, the information-theoretical framework has been proven to be able toobtain non-vacuous generalization bounds for large models trained by StochasticGradient Langevin Dynamics (SGLD) with isotropic noise. In this paper, weoptimize the information-theoretical generalization bound by manipulating thenoise structure in SGLD. We prove that with constraint to guarantee lowempirical risk, the optimal noise covariance is the square root of the expectedgradient covariance if both the prior and the posterior are jointly optimized.This validates that the optimal noise is quite close to the empirical gradientcovariance. Technically, we develop a new information-theoretical bound thatenables such an optimization analysis. We then apply matrix analysis to derivethe form of optimal noise covariance. Presented constraint and results arevalidated by the empirical observations.

Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD