English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large Language Models (LLMs) pose significant hardware challenges related tomemory requirements and computational ability. There are two mainstreamquantization schemes for LLMs: coarse-grained ($\textit{e.g.,}$ channel-wise)quantization and fine-grained ($\textit{e.g.,}$ group-wise) quantization.Fine-grained quantization has smaller quantization loss, consequently achievingsuperior performance. However, when applied to weight-activation quantization,it disrupts continuous integer matrix multiplication, leading to inefficientinference. In this paper, we introduce Dual Grained Quantization (DGQ), a novelA8W4 quantization for LLM that maintains superior performance while ensuringfast inference speed. DSQ dequantizes the fine-grained INT4 weight intocoarse-grained INT8 representation and preform matrix multiplication using INT8kernels. Besides, we develop a two-phase grid search algorithm to simplify thedetermination of fine-grained and coarse-grained quantization scales. We alsodevise a percentile clipping schema for smoothing the activation outlierswithout the need for complex optimization techniques. Experimental resultsdemonstrate that DGQ consistently outperforms prior methods across various LLMarchitectures and a wide range of tasks. Remarkably, by our implementedefficient CUTLASS kernel, we achieve $\textbf{1.12}$ $\times$ memory reductionand $\textbf{3.24}$ $\times$ speed gains comparing A16W4 implementation. Theseadvancements enable efficient deployment of A8W4 LLMs for real-worldapplications.

Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM