English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study stochastic decentralized optimization for the problem of trainingmachine learning models with large-scale distributed data. We extend the widelyused EXTRA and DIGing methods with variance reduction (VR), and propose twomethods: VR-EXTRA and VR-DIGing. The proposed VR-EXTRA requires the time of$O((\kappa_s+n)\log\frac{1}{\epsilon})$ stochastic gradient evaluations and$O((\kappa_b+\kappa_c)\log\frac{1}{\epsilon})$ communication rounds to reachprecision $\epsilon$, which are the best complexities among the non-acceleratedgradient-type methods, where $\kappa_s$ and $\kappa_b$ are the stochasticcondition number and batch condition number for strongly convex and smoothproblems, respectively, $\kappa_c$ is the condition number of the communicationnetwork, and $n$ is the sample size on each distributed node. The proposedVR-DIGing has a little higher communication cost of$O((\kappa_b+\kappa_c^2)\log\frac{1}{\epsilon})$. Our stochastic gradientcomputation complexities are the same as the ones of single-machine VR methods,such as SAG, SAGA, and SVRG, and our communication complexities keep the sameas those of EXTRA and DIGing, respectively. To further speed up theconvergence, we also propose the accelerated VR-EXTRA and VR-DIGing with boththe optimal $O((\sqrt{n\kappa_s}+n)\log\frac{1}{\epsilon})$ stochastic gradientcomputation complexity and $O(\sqrt{\kappa_b\kappa_c}\log\frac{1}{\epsilon})$communication complexity. Our stochastic gradient computation complexity isalso the same as the ones of single-machine accelerated VR methods, such asKatyusha, and our communication complexity keeps the same as those ofaccelerated full batch decentralized methods, such as MSDA.

Variance Reduced EXTRA and DIGing and Their Optimal Acceleration for Strongly Convex Decentralized Optimization