English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

To efficiently scale large model (LM) training, researchers transition fromdata parallelism (DP) to hybrid parallelism (HP) on GPU clusters, whichfrequently experience hardware and software failures. Existing works introducein-memory checkpointing optimizations that snapshot parameters to device memoryfor rapid failure recovery. However, these methods introduce severe resourcecompetition between checkpointing and training, which can work under DP but canhardly scale under resource-intensive HP. To ensure low checkpointing overheadfor hybrid-parallel training, this paper introduces a distributed in-memorycheckpointing system with near-zero in-memory saving overhead. It strives fromtwo aspects to mitigate the on-host resource competition caused by in-memorycheckpointing: (1) It introduces Hierarchical Asynchronous SnapshottingCoordination in the checkpoint saving stage. This approach uses three-levelasynchronous on-device scheduling to enhance parallelism between snapshottingand training, thereby minimizing snapshotting overhead. (2) It proposes HybridIn-memory Checkpoint Protection to enhance checkpoint completeness duringhardware failures. Unlike methods that require inter-node communications, whichmay block training under HP, it creates intra-node redundancy with efficientresource utilization, protecting training against hardware failures withminimal overhead. With these methods, this work enables fast restart for failedHP training with Distributed In-memory Checkpoint Loading, bypassinginefficiencies in NFS reads. In our evaluation, we achieve zero in-memorycheckpoint saving overhead on Frontier while training Llama-2-34B on 256 MI250Xdevices (512 GPUs).

Fault-Tolerant Hybrid-Parallel Training at Scale with Reliable and Efficient In-memory Checkpointing