English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent advances in deep learning base on growing model sizes and thenecessary scaling of compute power. Training such large-scale models requiresan intricate combination of data-, operator-, and pipeline parallelism incomplex distributed systems. We show how to use OneFlow's Split, Broadcast, andPartial Sum (SBP) tensor formulations to enable new distributed trainingmethods with asymptotically optimal communication overheads. Using theseinsights, we develop AutoDDL, a distributed training framework that combines anexhaustive performance model and automated configuration search to finddistributions with near-optimal communication overheads. We conduct evaluationson Multi-Node-Single-GPU and Multi-Node-Multi-GPU machines using differentmodels, including VGG and Transformer. Compared to expert-optimizedimplementations, AutoDDL reduces the end-to-end training time by up to 31.1\%and 10\% for Transformer and up to 17.7\% and 71.5\% for VGG on the twodifferent systems, respectively.

AutoDDL: Automatic Distributed Deep Learning with Asymptotically Optimal Communication