English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We propose LLM-FP4 for quantizing both weights and activations in largelanguage models (LLMs) down to 4-bit floating-point values, in a post-trainingmanner. Existing post-training quantization (PTQ) solutions are primarilyinteger-based and struggle with bit widths below 8 bits. Compared to integerquantization, floating-point (FP) quantization is more flexible and can betterhandle long-tail or bell-shaped distributions, and it has emerged as a defaultchoice in many hardware platforms. One characteristic of FP quantization isthat its performance largely depends on the choice of exponent bits andclipping range. In this regard, we construct a strong FP-PTQ baseline bysearching for the optimal quantization parameters. Furthermore, we observe ahigh inter-channel variance and low intra-channel variance pattern inactivation distributions, which adds activation quantization difficulty. Werecognize this pattern to be consistent across a spectrum of transformer modelsdesigned for diverse tasks, such as LLMs, BERT, and Vision Transformer models.To tackle this, we propose per-channel activation quantization and show thatthese additional scaling factors can be reparameterized as exponential biasesof weights, incurring a negligible cost. Our method, for the first time, canquantize both weights and activations in the LLaMA-13B to only 4-bit andachieves an average score of 63.1 on the common sense zero-shot reasoningtasks, which is only 5.8 lower than the full-precision model, significantlyoutperforming the previous state-of-the-art by 12.7 points. Code is availableat: https://github.com/nbasyl/LLM-FP4.

LLM-FP4: 4-Bit Floating-Point Quantized Transformers