English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The size of Transformer models is growing at an unprecedented pace. It hasonly taken less than one year to reach trillion-level parameters after therelease of GPT-3 (175B). Training such models requires both substantialengineering efforts and enormous computing resources, which are luxuries mostresearch teams cannot afford. In this paper, we propose PipeTransformer, whichleverages automated and elastic pipelining and data parallelism for efficientdistributed training of Transformer models. PipeTransformer automaticallyadjusts the pipelining and data parallelism by identifying and freezing somelayers during the training, and instead allocates resources for training of theremaining active layers. More specifically, PipeTransformer dynamicallyexcludes converged layers from the pipeline, packs active layers into fewerGPUs, and forks more replicas to increase data-parallel width. We evaluatePipeTransformer using Vision Transformer (ViT) on ImageNet and BERT on GLUE andSQuAD datasets. Our results show that PipeTransformer attains a 2.4 foldspeedup compared to the state-of-the-art baseline. We also provide variousperformance analyses for a more comprehensive understanding of our algorithmicand system-wise design. We also develop open-sourced flexible APIs forPipeTransformer, which offer a clean separation among the freeze algorithm,model definitions, and training accelerations, hence allowing it to be appliedto other algorithms that require similar freezing strategies.

PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers