English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We introduce a general framework for nonlinear stochastic gradient descent(SGD) for the scenarios when gradient noise exhibits heavy tails. The proposedframework subsumes several popular nonlinearity choices, like clipped,normalized, signed or quantized gradient, but we also consider novelnonlinearity choices. We establish for the considered class of methods strongconvergence guarantees assuming a strongly convex cost function with Lipschitzcontinuous gradients under very general assumptions on the gradient noise. Mostnotably, we show that, for a nonlinearity with bounded outputs and for thegradient noise that may not have finite moments of order greater than one, thenonlinear SGD's mean squared error (MSE), or equivalently, the expected costfunction's optimality gap, converges to zero at rate~$O(1/t^\zeta)$, $\zeta \in(0,1)$. In contrast, for the same noise setting, the linear SGD generates asequence with unbounded variances. Furthermore, for the nonlinearities that canbe decoupled component wise, like, e.g., sign gradient or component-wiseclipping, we show that the nonlinear SGD asymptotically (locally) achieves a$O(1/t)$ rate in the weak convergence sense and explicitly quantify thecorresponding asymptotic variance. Experiments show that, while our frameworkis more general than existing studies of SGD under heavy-tail noise, severaleasy-to-implement nonlinearities from our framework are competitive with stateof the art alternatives on real data sets with heavy tail noises.

Nonlinear gradient mappings and stochastic optimization: A general framework with applications to heavy-tail noise