English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

One of the main focuses in distributed learning is communication efficiency,since model aggregation at each round of training can consist of millions tobillions of parameters. Several model compression methods, such as gradientquantization and sparsification, have been proposed to improve thecommunication efficiency of model aggregation. However, theinformation-theoretic minimum communication cost for a given distortion ofgradient estimators is still unknown. In this paper, we study the fundamentallimit of communication cost of model aggregation in distributed learning from arate-distortion perspective. By formulating the model aggregation as a vectorGaussian CEO problem, we derive the rate region bound and sum-rate-distortionfunction for the model aggregation problem, which reveals the minimumcommunication rate at a particular gradient distortion upper bound. We alsoanalyze the communication cost at each iteration and total communication costbased on the sum-rate-distortion function with the gradient statistics ofreal-world datasets. It is found that the communication gain by exploiting thecorrelation between worker nodes is significant for SignSGD, and a highdistortion of gradient estimator can achieve low total communication cost ingradient compression.

Fundamental Limits of Communication Efficiency for Model Aggregation in Distributed Learning: A Rate-Distortion Approach