English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Computation in a typical Transformer-based large language model (LLM) can becharacterized by batch size, hidden dimension, number of layers, and sequencelength. Until now, system works for accelerating LLM training have focused onthe first three dimensions: data parallelism for batch size, tensor parallelismfor hidden size and pipeline parallelism for model depth or layers. Thesewidely studied forms of parallelism are not targeted or optimized for longsequence Transformer models. Given practical application needs for longsequence LLM, renewed attentions are being drawn to sequence parallelism.However, existing works in sequence parallelism are constrained bymemory-communication inefficiency, limiting their scalability to long sequencelarge models. In this work, we introduce DeepSpeed-Ulysses, a novel, portableand effective methodology for enabling highly efficient and scalable LLMtraining with extremely long sequence length. DeepSpeed-Ulysses at its corepartitions input data along the sequence dimension and employs an efficientall-to-all collective communication for attention computation. Theoreticalcommunication analysis shows that whereas other methods incur communicationoverhead as sequence length increases, DeepSpeed-Ulysses maintains constantcommunication volume when sequence length and compute devices are increasedproportionally. Furthermore, experimental evaluations show thatDeepSpeed-Ulysses trains 2.5x faster with 4x longer sequence length than theexisting method SOTA baseline.

DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models