English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

With the widespread deployment of long-context large language models (LLMs),there has been a growing demand for efficient support of high-throughputinference. However, as the key-value (KV) cache expands with the sequencelength, the increasing memory footprint and the need to access it for eachtoken generation both result in low throughput when serving long-context LLMs.While various dynamic sparse attention methods have been proposed to speed upinference while maintaining generation quality, they either fail tosufficiently reduce GPU memory consumption or introduce significant decodinglatency by offloading the KV cache to the CPU. We present ShadowKV, ahigh-throughput long-context LLM inference system that stores the low-rank keycache and offloads the value cache to reduce the memory footprint for largerbatch sizes and longer sequences. To minimize decoding latency, ShadowKVemploys an accurate KV selection strategy that reconstructs minimal sparse KVpairs on-the-fly. By evaluating ShadowKV on a broad range of benchmarks,including RULER, LongBench, and Needle In A Haystack, and models likeLlama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M, Yi-9B-200K, Phi-3-Mini-128K, andQwen2-7B-128K, we demonstrate that it can support up to 6$\times$ larger batchsizes and boost throughput by up to 3.04$\times$ on an A100 GPU withoutsacrificing accuracy, even surpassing the performance achievable with infinitebatch size under the assumption of infinite GPU memory. The code is availableat https://github.com/bytedance/ShadowKV.

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference