English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

In order to satisfy their ever increasing capacity and compute requirements,machine learning models are distributed across multiple nodes using numerousparallelism strategies. As a result, collective communications are often on thecritical path, and hiding their latency by overlapping kernel-granularcommunication and computation is difficult due to the absence of independentcomputation. In this work, we propose fusing computation with dependentcollective communication by leveraging GPUs' massive parallelism andGPU-initiated communication. We have developed self-contained GPU kernels whereworkgroups (WGs) immediately communicate their results to remote GPUs when theycomplete their computation. Meanwhile, other WGs within the same kernel performoverlapping computation, maintaining high ALU utilization.  We demonstrate our approach by creating three prototype fused operators(embedding + All-to-All, GEMV + AllReduce, and GEMM + All-to-All) to addressthe pervasive communication overheads observed in DLRM, Transformers and MoEmodel architectures. In order to demonstrate that our approach can beintegrated into ML frameworks for wide adoption in production environments, weexpose our fused operators as new PyTorch operators as well as extend theTriton framework to enable them. Our evaluations show that our approach caneffectively overlap communication with computations, subsequently reducingtheir combined execution time than the current collective library-basedapproaches. Our scale-up GEMV + AllReduce and GEMM + All-to-All implementationsachieve up to 22% and 20% lower execution time, while our fused embedding +All-to-All reduces execution time by 20% and 31% for intra-node and inter-nodeconfigurations. Large scale-out simulations indicate that our approach reducesDLRM execution time by 21% for 128 node system.

Optimizing Distributed ML Communication with Fused Computation-Collective Operations