English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Increasing the context length of large language models (LLMs) unlocksfundamentally new capabilities, but also significantly increases the memoryfootprints of training. Previous model-parallel systems such as Megatron-LMpartition and compute different attention heads in parallel, resulting in largecommunication volumes, so they cannot scale beyond the number of attentionheads, thereby hindering its adoption. In this paper, we introduce a newapproach, LightSeq, for long-context LLMs training. LightSeq has many notableadvantages. First, LightSeq partitions over the sequence dimension, hence isagnostic to model architectures and readily applicable for models with varyingnumbers of attention heads, such as Multi-Head, Multi-Query and Grouped-Queryattention. Second, LightSeq not only requires up to 4.7x less communicationthan Megatron-LM on popular LLMs but also overlaps the communication withcomputation. To further reduce the training time, LightSeq features a novelgradient checkpointing scheme to bypass an forward computation formemory-efficient attention. We evaluate LightSeq on Llama-7B and its variantswith sequence lengths from 32K to 512K. Through comprehensive experiments onsingle and cross-node training, we show that LightSeq achieves up to 1.24-2.01xend-to-end speedup, and a 2-8x longer sequence length on models with fewerheads, compared to Megatron-LM. Codes will be available athttps://github.com/RulinShao/LightSeq.

LightSeq: Sequence Level Parallelism for Distributed Training of Long Context Transformers