English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Availability of high performance computing infrastructures such as clustersof GPUs and CPUs have fueled the growth of distributed learning systems. DeepLearning frameworks express neural nets as DAGs and execute these DAGs oncomputation resources such as GPUs. In this paper, we propose efficient designsof embedding MPI collective operations into data parallel DAGs. Incorrectdesigns can easily lead to deadlocks or program crashes. In particular, wedemonstrate three designs: Funneled, Concurrent communication and Dependencychaining of using MPI collectives with DAGs. These designs automatically enableoverlap of computation with communication by allowing for concurrent executionwith the other tasks. We directly implement these designs into the KVStore APIof the MXNET. This allows us to directly leverage the rest of theinfrastructure. Using ImageNet and CIFAR data sets, we show the potential ofour designs. In particular, our designs scale to 256 GPUs with as low as 50seconds of epoch times for ImageNet 1K datasets.

Efficient Embedding of MPI Collectives in MXNET DAGs for scaling Deep Learning