English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The past several years have witnessed the success of transformer-basedmodels, and their scale and application scenarios continue to growaggressively. The current landscape of transformer models is increasinglydiverse: the model size varies drastically with the largest being ofhundred-billion parameters; the model characteristics differ due to thesparsity introduced by the Mixture-of-Experts; the target application scenarioscan be latency-critical or throughput-oriented; the deployment hardware couldbe single- or multi-GPU systems with different types of memory and storage,etc. With such increasing diversity and the fast-evolving pace of transformermodels, designing a highly performant and efficient inference system isextremely challenging. In this paper, we present DeepSpeed Inference, acomprehensive system solution for transformer model inference to address theabove-mentioned challenges. DeepSpeed Inference consists of (1) a multi-GPUinference solution to minimize latency while maximizing the throughput of bothdense and sparse transformer models when they fit in aggregate GPU memory, and(2) a heterogeneous inference solution that leverages CPU and NVMe memory inaddition to the GPU memory and compute to enable high inference throughput withlarge models which do not fit in aggregate GPU memory. DeepSpeed Inferencereduces latency by up to 7.3X over the state-of-the-art for latency-orientedscenarios and increases throughput by over 1.5x for throughput-orientedscenarios. Moreover, it enables trillion parameter scale inference underreal-time latency constraints by leveraging hundreds of GPUs, an unprecedentedscale for inference. It can inference 25x larger models than with GPU-onlysolutions, while delivering a high throughput of 84 TFLOPS (over $50\%$ ofA6000 peak).

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale