English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

As a crucial scheme to accelerate the deep neural network (DNN) training,distributed stochastic gradient descent (DSGD) is widely adopted in manyreal-world applications. In most distributed deep learning (DL) frameworks,DSGD is implemented with Ring-AllReduce architecture (Ring-SGD) and uses acomputation-communication overlap strategy to address the overhead of themassive communications required by DSGD. However, we observe that although$O(1)$ gradients are needed to be communicated per worker in Ring-SGD, the$O(n)$ handshakes required by Ring-SGD limits its usage when training with manyworkers or in high latency network. In this paper, we propose Shuffle-ExchangeSGD (SESGD) to solve the dilemma of Ring-SGD. In the cluster of 16 workers with0.1ms Ethernet latency, SESGD can accelerate the DNN training to $1.7 \times$without losing model accuracy. Moreover, the process can be accelerated up to$5\times$ in high latency networks (5ms).

Shuffle-Exchange Brings Faster: Reduce the Idle Time During Communication for Decentralized Neural Network Training