English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent years have seen an increase in the development of large deep learning(DL) models, which makes training efficiency crucial. Common practice isstruggling with the trade-off between usability and performance. On one hand,DL frameworks such as PyTorch use dynamic graphs to facilitate model developersat a price of sub-optimal model training performance. On the other hand,practitioners propose various approaches to improving the training efficiencyby sacrificing some of the flexibility, ranging from making the graph staticfor more thorough optimization (e.g., XLA) to customizing optimization towardslarge-scale distributed training (e.g., DeepSpeed and Megatron-LM). In thispaper, we aim to address the tension between usability and training efficiencythrough separation of concerns. Inspired by DL compilers that decouple theplatform-specific optimizations of a tensor-level operator from its arithmeticdefinition, this paper proposes a schedule language, Slapo, to decouple modelexecution from definition. Specifically, Slapo works on a PyTorch model anduses a set of schedule primitives to convert the model for common modeltraining optimizations such as high-performance kernels, effective 3Dparallelism, and efficient activation checkpointing. Compared to existingoptimization solutions, Slapo progressively optimizes the model "as-needed"through high-level primitives, and thus preserving programmability anddebuggability for users to a large extent. Our evaluation results show that byscheduling the existing hand-crafted optimizations in a systematic way usingSlapo, we are able to improve training throughput by up to 2.92x on a singlemachine with 8 NVIDIA V100 GPUs, and by up to 1.41x on multiple machines withup to 64 GPUs, when compared to the out-of-the-box performance of DeepSpeed andMegatron-LM.

Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Training