English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In federated learning (FL), a global model is trained at a Parameter Server(PS) by aggregating model updates obtained from multiple remote learners.Generally, the communication between the remote users and the PS israte-limited, while the transmission from the PS to the remote users areunconstrained. The FL setting gives rise to the distributed learning scenarioin which the updates from the remote learners have to be compressed so as tomeet communication rate constraints in the uplink transmission toward the PS.For this problem, one wishes to compress the model updates so as to minimizethe loss in accuracy resulting from the compression error. In this paper, wetake a rate-distortion approach to address the compressor design problem forthe distributed training of deep neural networks (DNNs). In particular, wedefine a measure of the compression performance under communication-rateconstraints -- the \emph{per-bit accuracy} -- which addresses the ultimateimprovement of accuracy that a bit of communication brings to the centralizedmodel. In order to maximize the per-bit accuracy, we consider modeling the DNNgradient updates at remote learners as a generalized normal distribution. Underthis assumption on the DNN gradient distribution, we propose a class ofdistortion measures to aid the design of quantizers for the compression of themodel updates. We argue that this family of distortion measures, which we referto as "$M$-magnitude weighted $L_2$" norm, captures the practitioner'sintuition in the choice of gradient compressor. Numerical simulations areprovided to validate the proposed approach for the CIFAR-10 dataset.

Lossy Gradient Compression: How Much Accuracy Can One Bit Buy?