English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Communication overhead has become one of the major bottlenecks in thedistributed training of deep neural networks. To alleviate the concern, variousgradient compression methods have been proposed, and sign-based algorithms areof surging interest. However, SIGNSGD fails to converge in the presence of dataheterogeneity, which is commonly observed in the emerging federated learning(FL) paradigm. Error feedback has been proposed to address the non-convergenceissue. Nonetheless, it requires the workers to locally keep track of thecompression errors, which renders it not suitable for FL since the workers maynot participate in the training throughout the learning process. In this paper,we propose a magnitude-driven sparsification scheme, which addresses thenon-convergence issue of SIGNSGD while further improving communicationefficiency. Moreover, the local update scheme is further incorporated toimprove the learning performance, and the convergence of the proposed method isestablished. The effectiveness of the proposed scheme is validated throughexperiments on Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets.

Magnitude Matters: Fixing SIGNSGD Through Magnitude-Aware Sparsification in the Presence of Data Heterogeneity