English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In the complex domain of large language models (LLMs), striking a balancebetween computational efficiency and maintaining model quality is a formidablechallenge. Navigating the inherent limitations of uniform quantization,particularly when dealing with outliers, and motivated by the launch ofNVIDIA's H100 hardware, this study delves into the viability of floating-point(FP) quantization, particularly focusing on FP8 and FP4, as a potentialsolution. Our comprehensive investigation reveals that for LLMs, FP8 activationconsistently outshines its integer (INT8) equivalent, with the performance edgebecoming more noticeable in models possessing parameters beyond one billion.For weight quantization, our findings indicate that FP4 exhibits comparable, ifnot superior, performance to INT4, simplifying deployment on FP-supportedhardware like H100. To mitigate the overhead from precision alignment caused bythe disparity between weights and activations, we propose two scalingconstraints for weight quantization that negligibly impact the performancecompared to the standard W4A8 model. We additionally enhance our quantizationmethods by integrating the Low Rank Compensation (LoRC) strategy, yieldingimprovements especially in smaller models. The results of our investigationemphasize the immense potential of FP quantization for LLMs, paving the way forhigh-efficiency deployment in resource-limited settings.

ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats