English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Synchronous mini-batch SGD is state-of-the-art for large-scale distributedmachine learning. However, in practice, its convergence is bottlenecked by slowcommunication rounds between worker nodes. A natural solution to reducecommunication is to use the \emph{`local-SGD'} model in which the workers traintheir model independently and synchronize every once in a while. This algorithmimproves the computation-communication trade-off but its convergence is notunderstood very well. We propose a non-asymptotic error analysis, which enablescomparison to \emph{one-shot averaging} i.e., a single communication roundamong independent workers, and \emph{mini-batch averaging} i.e., communicatingat every step. We also provide adaptive lower bounds on the communicationfrequency for large step-sizes ($ t^{-\alpha} $, $ \alpha\in (1/2 , 1 ) $) andshow that \emph{Local-SGD} reduces communication by a factor of$O\Big(\frac{\sqrt{T}}{P^{3/2}}\Big)$, with $T$ the total number of gradientsand $P$ machines.

Communication trade-offs for synchronized distributed SGD with large step size