English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large language models (LLMs) have shown remarkable capabilities in varioustasks. However their huge model size and the consequent demand forcomputational and memory resources also pose challenges to model deployment.Currently, 4-bit post-training quantization (PTQ) has achieved some success inLLMs, reducing the memory footprint by approximately 75% compared to FP16models, albeit with some accuracy loss. In this paper, we propose SmoothQuant+,an accurate and efficient 4-bit weight-only PTQ that requires no additionaltraining, which enables lossless in accuracy for LLMs for the first time. Basedon the fact that the loss of weight quantization is amplified by the activationoutliers, SmoothQuant+ smoothes the activation outliers by channel beforequantization, while adjusting the corresponding weights for mathematicalequivalence, and then performs group-wise 4-bit weight quantization for linearlayers. We have integrated SmoothQuant+ into the vLLM framework, an advancedhigh-throughput inference engine specially developed for LLMs, and equipped itwith an efficient W4A16 CUDA kernels, so that vLLM can seamlessly supportSmoothQuant+ 4-bit weight quantization. Our results show that, withSmoothQuant+, the Code Llama-34B model can be quantized and deployed on a A10040GB GPU, achieving lossless accuracy and a throughput increase of 1.9 to 4.0times compared to the FP16 model deployed on two A100 40GB GPUs. Moreover, thelatency per token is only 68% of the FP16 model deployed on two A100 40GB GPUs.This is the state-of-the-art 4-bit weight quantization for LLMs as we know.

SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM