English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

With huge amounts of training data, deep learning has made greatbreakthroughs in many artificial intelligence (AI) applications. However, suchlarge-scale data sets present computational challenges, requiring training tobe distributed on a cluster equipped with accelerators like GPUs. With the fastincrease of GPU computing power, the data communications among GPUs have becomea potential bottleneck on the overall training performance. In this paper, wefirst propose a general directed acyclic graph (DAG) model to describe thedistributed synchronous stochastic gradient descent (S-SGD) algorithm, whichhas been widely used in distributed deep learning frameworks. To understand thepractical impact of data communications on training performance, we conductextensive empirical studies on four state-of-the-art distributed deep learningframeworks (i.e., Caffe-MPI, CNTK, MXNet and TensorFlow) over multi-GPU andmulti-node environments with different data communication techniques, includingPCIe, NVLink, 10GbE, and InfiniBand. Through both analytical and experimentalstudies, we identify the potential bottlenecks and overheads that could befurther optimized. At last, we make the data set of our experimental tracespublicly available, which could be used to support simulation-based studies.

A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning