English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large language models (LLMs) with hundreds of billions of parameters havesparked a new wave of exciting AI applications. However, they arecomputationally expensive at inference time. Sparsity is a natural approach toreduce this cost, but existing methods either require costly retraining, haveto forgo LLM's in-context learning ability, or do not yield wall-clock timespeedup on modern hardware. We hypothesize that contextual sparsity, which aresmall, input-dependent sets of attention heads and MLP parameters that yieldapproximately the same output as the dense model for a given input, can addressthese issues. We show that contextual sparsity exists, that it can beaccurately predicted, and that we can exploit it to speed up LLM inference inwall-clock time without compromising LLM's quality or in-context learningability. Based on these insights, we propose DejaVu, a system that uses alow-cost algorithm to predict contextual sparsity on the fly given inputs toeach layer, along with an asynchronous and hardware-aware implementation thatspeeds up LLM inference. We validate that DejaVu can reduce the inferencelatency of OPT-175B by over 2X compared to the state-of-the-artFasterTransformer, and over 6X compared to the widely used Hugging Faceimplementation, without compromising model quality. The code is available athttps://github.com/FMInference/DejaVu.

Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time