English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this study, we introduce adaptive KV cache compression, a plug-and-playmethod that reduces the memory footprint of generative inference for LargeLanguage Models (LLMs). Different from the conventional KV cache that retainskey and value vectors for all context tokens, we conduct targeted profiling todiscern the intrinsic structure of attention modules. Based on the recognizedstructure, we then construct the KV cache in an adaptive manner: evictinglong-range contexts on attention heads emphasizing local contexts, discardingnon-special tokens on attention heads centered on special tokens, and onlyemploying the standard KV cache for attention heads that broadly attend to alltokens. Moreover, with the lightweight attention profiling used to guide theconstruction of the adaptive KV cache, FastGen can be deployed withoutresource-intensive fine-tuning or re-training. In our experiments acrossvarious asks, FastGen demonstrates substantial reduction on GPU memoryconsumption with negligible generation quality loss. We will release our codeand the compatible CUDA kernel for reproducibility.

Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs