English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In distributed or federated optimization and learning, communication betweenthe different computing units is often the bottleneck and gradient compressionis widely used to reduce the number of bits sent within each communicationround of iterative methods. There are two classes of compression operators andseparate algorithms making use of them. In the case of unbiased randomcompressors with bounded variance (e.g., rand-k), the DIANA algorithm ofMishchenko et al. (2019), which implements a variance reduction technique forhandling the variance introduced by compression, is the current state of theart. In the case of biased and contractive compressors (e.g., top-k), the EF21algorithm of Richt\'arik et al. (2021), which instead implements anerror-feedback mechanism, is the current state of the art. These two classes ofcompression schemes and algorithms are distinct, with different analyses andproof techniques. In this paper, we unify them into a single framework andpropose a new algorithm, recovering DIANA and EF21 as particular cases. Ourgeneral approach works with a new, larger class of compressors, which has twoparameters, the bias and the variance, and includes unbiased and biasedcompressors as particular cases. This allows us to inherit the best of the twoworlds: like EF21 and unlike DIANA, biased compressors, like top-k, whose goodperformance in practice is recognized, can be used. And like DIANA and unlikeEF21, independent randomness at the compressors allows to mitigate the effectsof compression, with the convergence rate improving when the number of parallelworkers is large. This is the first time that an algorithm with all thesefeatures is proposed. We prove its linear convergence under certain conditions.Our approach takes a step towards better understanding of two so-far distinctworlds of communication-efficient distributed learning.

EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimization