English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper we study the problem of convergence and generalization errorbound of stochastic momentum for deep learning from the perspective ofregularization. To do so, we first interpret momentum as solving an$\ell_2$-regularized minimization problem to learn the offsets betweenarbitrary two successive model parameters. We call this {\em time-delaymomentum} because the model parameter is updated after a few iterations towardsfinding the minimizer. We then propose our learning algorithm, \ie stochasticgradient descent (SGD) with time-delay momentum. We show that our algorithm canbe interpreted as solving a sequence of strongly convex optimization problemsusing SGD. We prove that under mild conditions our algorithm can converge to astationary point with rate of $O(\frac{1}{\sqrt{K}})$ and generalization errorbound of $O(\frac{1}{\sqrt{n\delta}})$ with probability at least $1-\delta$,where $K,n$ are the numbers of model updates and training samples,respectively. We demonstrate the empirical superiority of our algorithm in deeplearning in comparison with the state-of-the-art deep learning solvers.

Time-Delay Momentum: A Regularization Perspective on the Convergence and Generalization of Stochastic Momentum for Deep Learning