English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Recently, local SGD has got much attention and been extensively studied inthe distributed learning community to overcome the communication bottleneckproblem. However, the superiority of local SGD to minibatch SGD only holds inquite limited situations. In this paper, we study a new local algorithm calledBias-Variance Reduced Local SGD (BVR-L-SGD) for nonconvex distributedoptimization. Algorithmically, our proposed bias and variance reduced localgradient estimator fully utilizes small second-order heterogeneity of localobjectives and suggests randomly picking up one of the local models instead oftaking the average of them when workers are synchronized. Theoretically, undersmall heterogeneity of local objectives, we show that BVR-L-SGD achieves bettercommunication complexity than both the previous non-local and local methodsunder mild conditions, and particularly BVR-L-SGD is the first method thatbreaks the barrier of communication complexity $\Theta(1/\varepsilon)$ forgeneral nonconvex smooth objectives when the heterogeneity is small and thelocal computation budget is large. Numerical results are given to verify thetheoretical findings and give empirical evidence of the superiority of ourmethod.

Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning