English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent works have shown that high probability metrics with stochasticgradient descent (SGD) exhibit informativeness and in some cases advantage overthe commonly adopted mean-square error-based ones. In this work we provide aformal framework for the study of general high probability bounds with SGD,based on the theory of large deviations. The framework allows for a generic(not-necessarily bounded) gradient noise satisfying mild technical assumptions,allowing for the dependence of the noise distribution on the current iterate.Under the preceding assumptions, we find an upper large deviations bound forSGD with strongly convex functions. The corresponding rate function capturesanalytical dependence on the noise distribution and other problem parameters.This is in contrast with conventional mean-square error analysis that capturesonly the noise dependence through the variance and does not capture the effectof higher order moments nor interplay between the noise geometry and the shapeof the cost function. We also derive exact large deviation rates for the casewhen the objective function is quadratic and show that the obtained functionmatches the one from the general upper bound hence showing the tightness of thegeneral upper bound. Numerical examples illustrate and corroborate theoreticalfindings.

Large deviations rates for stochastic gradient descent with strongly convex functions