English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We introduce a hybrid stochastic estimator to design stochastic gradientalgorithms for solving stochastic optimization problems. Such a hybridestimator is a convex combination of two existing biased and unbiasedestimators and leads to some useful property on its variance. We limit ourconsideration to a hybrid SARAH-SGD for nonconvex expectation problems.However, our idea can be extended to handle a broader class of estimators inboth convex and nonconvex settings. We propose a new single-loop stochasticgradient descent algorithm that can achieve$O(\max\{\sigma^3\varepsilon^{-1},\sigma\varepsilon^{-3}\})$-complexity boundto obtain an $\varepsilon$-stationary point under smoothness and$\sigma^2$-bounded variance assumptions. This complexity is better than$O(\sigma^2\varepsilon^{-4})$ often obtained in state-of-the-art SGDs when$\sigma < O(\varepsilon^{-3})$. We also consider different extensions of ourmethod, including constant and adaptive step-size with single-loop,double-loop, and mini-batch variants. We compare our algorithms with existingmethods on several datasets using two nonconvex models.

Hybrid Stochastic Gradient Descent Algorithms for Stochastic Nonconvex Optimization