English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

We evaluate the current state of collective communication on GPU-basedsupercomputers for large language model (LLM) training at scale. Existinglibraries such as RCCL and Cray-MPICH exhibit critical limitations on systemssuch as Frontier -- Cray-MPICH underutilizes network and compute resources,while RCCL suffers from severe scalability issues. To address these challenges,we introduce PCCL, a communication library with highly optimizedimplementations of all-gather and reduce-scatter operations tailored fordistributed deep learning workloads. PCCL is designed to maximally utilize allavailable network and compute resources and to scale efficiently to thousandsof GPUs. It achieves substantial performance improvements, delivering 6-33xspeedups over RCCL and 28-70x over Cray-MPICH for all-gather on 2048 GCDs ofFrontier. These gains translate directly to end-to-end performance: inlarge-scale GPT-3-style training, PCCL provides up to 60% and 40% speedups overRCCL for 7B and 13B parameter models, respectively.

The Big Send-off: High Performance Collectives on GPU-based Supercomputers