English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Training deep learning models can be computationally expensive. Prior workshave shown that increasing the batch size can potentially lead to betteroverall throughput. However, the batch size is frequently limited by theaccelerator memory capacity due to the activations/feature maps stored for thetraining backward pass, as larger batch sizes require larger feature maps to bestored. Transformer-based models, which have recently seen a surge inpopularity due to their good performance and applicability to a variety oftasks, have a similar problem. To remedy this issue, we propose Tempo, a newapproach to efficiently use accelerator (e.g., GPU) memory resources fortraining Transformer-based models. Our approach provides drop-in replacementsfor the GELU, LayerNorm, and Attention layers, reducing the memory usage andultimately leading to more efficient training. We implement Tempo and evaluatethe throughput, memory usage, and accuracy/loss on the BERT Large pre-trainingtask. We demonstrate that Tempo enables up to 2x higher batch sizes and 16%higher training throughput over the state-of-the-art baseline. We also evaluateTempo on GPT2 and RoBERTa models, showing 19% and 26% speedup over thebaseline.

Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction