English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Although the distributed machine learning methods can speed up the trainingof large deep neural networks, the communication cost has become thenon-negligible bottleneck to constrain the performance. To address thischallenge, the gradient compression based communication-efficient distributedlearning methods were designed to reduce the communication cost, and morerecently the local error feedback was incorporated to compensate for thecorresponding performance loss. However, in this paper, we will show that a new"gradient mismatch" problem is raised by the local error feedback incentralized distributed training and can lead to degraded performance comparedwith full-precision training. To solve this critical problem, we propose twonovel techniques, 1) step ahead and 2) error averaging, with rigoroustheoretical analysis. Both our theoretical and empirical results show that ournew methods can handle the "gradient mismatch" problem. The experimentalresults show that we can even train faster with common gradient compressionschemes than both the full-precision training and local error feedbackregarding the training epochs and without performance loss.

Step-Ahead Error Feedback for Distributed Training with Compressed Gradient