English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Gradient compression is a recent and increasingly popular technique forreducing the communication cost in distributed training of large-scale machinelearning models. In this work we focus on developing efficient distributedmethods that can work for any compressor satisfying a certain contractionproperty, which includes both unbiased (after appropriate scaling) and biasedcompressors such as RandK and TopK. Applied naively, gradient compressionintroduces errors that either slow down convergence or lead to divergence. Apopular technique designed to tackle this issue is error compensation/errorfeedback. Due to the difficulties associated with analyzing biased compressors,it is not known whether gradient compression with error compensation can becombined with Nesterov's acceleration. In this work, we show for the first timethat error compensated gradient compression methods can be accelerated. Inparticular, we propose and study the error compensated loopless Katyushamethod, and establish an accelerated linear convergence rate under standardassumptions. We show through numerical experiments that the proposed methodconverges with substantially fewer communication rounds than previous errorcompensated algorithms.

Error Compensated Distributed SGD Can Be Accelerated