English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Random label noises (or observational noises) widely exist in practicalmachine learning settings. While previous studies primarily focus on theaffects of label noises to the performance of learning, our work intends toinvestigate the implicit regularization effects of the label noises, undermini-batch sampling settings of stochastic gradient descent (SGD), withassumptions that label noises are unbiased. Specifically, we analyze thelearning dynamics of SGD over the quadratic loss with unbiased label noises,where we model the dynamics of SGD as a stochastic differentiable equation(SDE) with two diffusion terms (namely a Doubly Stochastic Model). While thefirst diffusion term is caused by mini-batch sampling over the(label-noiseless) loss gradients as many other works on SGD, our modelinvestigates the second noise term of SGD dynamics, which is caused bymini-batch sampling over the label noises, as an implicit regularizer. Ourtheoretical analysis finds such implicit regularizer would favor someconvergence points that could stabilize model outputs against perturbation ofparameters (namely inference stability). Though similar phenomenon have beeninvestigated, our work doesn't assume SGD as an Ornstein-Uhlenbeck like processand achieve a more generalizable result with convergence of approximationproved. To validate our analysis, we design two sets of empirical studies toanalyze the implicit regularizer of SGD with unbiased random label noises fordeep neural networks training and linear regression.

Doubly Stochastic Models: Learning with Unbiased Label Noises and Inference Stability