English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large language models (LLMs) have demonstrated impressive abilities invarious domains while the inference cost is expensive. The state-of-the-artmethods use 2-bit quantization for mainstream LLMs. However, challenges stillexist: (1) Nonnegligible accuracy loss for 2-bit quantization. Weights arequantized by groups, while the ranges of weights are large in some groups,resulting in large quantization errors and nonnegligible accuracy loss (e.g.>3% for Llama2-7b with 2-bit quantization in GPTQ and Greenbit). (2) Limitedaccuracy improvement by adding 4-bit weights. Increasing 10% extra average bitmore 4-bit weights only leads to <0.5% accuracy improvement on a quantizedLlama2-7b. (3) Time-consuming dequantization operations on GPUs. Thedequantization operations lead to >50% execution time, hindering the potentialof reducing LLM inference cost. To tackle these challenges, we propose thefollowing techniques: (1) We only quantize a small fraction of groups with thelarger range using 4-bit with memory alignment consideration on GPUs.(2) Wedesign the asynchronous dequantization on GPUs, leading to up to 3.92X speedup.We conduct extensive experiments on different model sizes. We achieve 2.85-bitfor each weight and the end-to-end speedup for Llama2-7b is 1.74X over theoriginal model, and we reduce both runtime cost and hardware cost by up to2.70X and 2.81X with less GPU requirements.

Enabling Fast 2-bit LLM on GPUs: Memory Alignment and Asynchronous Dequantization