English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Distributed optimization methods are often applied to solving huge-scaleproblems like training neural networks with millions and even billions ofparameters. In such applications, communicating full vectors, e.g.,(stochastic) gradients, iterates, is prohibitively expensive, especially whenthe number of workers is large. Communication compression is a powerfulapproach to alleviating this issue, and, in particular, methods with biasedcompression and error compensation are extremely popular due to their practicalefficiency. Sahu et al. (2021) propose a new analysis of Error Compensated SGD(EC-SGD) for the class of absolute compression operators showing that in acertain sense, this class contains optimal compressors for EC-SGD. However, theanalysis was conducted only under the so-called $(M,\sigma^2)$-bounded noiseassumption. In this paper, we generalize the analysis of EC-SGD with absolutecompression to the arbitrary sampling strategy and propose the first analysisof Error Compensated Loopless Stochastic Variance Reduced Gradient method(EC-LSVRG) with absolute compression for (strongly) convex problems. Our ratesimprove upon the previously known ones in this setting. Numerical experimentscorroborate our theoretical findings.

Distributed Methods with Absolute Compression and Error Compensation