English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper introduces PowerInfer, a high-speed Large Language Model (LLM)inference engine on a personal computer (PC) equipped with a singleconsumer-grade GPU. The key underlying the design of PowerInfer is exploitingthe high locality inherent in LLM inference, characterized by a power-lawdistribution in neuron activation. This distribution indicates that a smallsubset of neurons, termed hot neurons, are consistently activated acrossinputs, while the majority, cold neurons, vary based on specific inputs.PowerInfer exploits such an insight to design a GPU-CPU hybrid inferenceengine: hot-activated neurons are preloaded onto the GPU for fast access, whilecold-activated neurons are computed on the CPU, thus significantly reducing GPUmemory demands and CPU-GPU data transfers. PowerInfer further integratesadaptive predictors and neuron-aware sparse operators, optimizing theefficiency of neuron activation and computational sparsity. Evaluation showsthat PowerInfer attains an average token generation rate of 13.20 tokens/s,with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on asingle NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tierserver-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69xwhile retaining model accuracy.

PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU