English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Large Language Models (LLMs) have achieved state-of-the-art performanceacross various language tasks but pose challenges for practical deployment dueto their substantial memory requirements. Furthermore, the latest generativemodels suffer from high inference costs caused by the memory bandwidthbottleneck in the auto-regressive decoding process. To address these issues, wepropose an efficient weight-only quantization method that reduces memoryconsumption and accelerates inference for LLMs. To ensure minimal qualitydegradation, we introduce a simple and effective heuristic approach thatutilizes only the model weights of a pre-trained model. This approach isapplicable to both Mixture-of-Experts (MoE) and dense models without requiringadditional fine-tuning. To demonstrate the effectiveness of our proposedmethod, we first analyze the challenges and issues associated with LLMquantization. Subsequently, we present our heuristic approach, which adaptivelyfinds the granularity of quantization, effectively addressing these problems.Furthermore, we implement highly efficient GPU GEMMs that perform on-the-flymatrix multiplication and dequantization, supporting the multiplication of fp16or bf16 activations with int8 or int4 weights. We evaluate our approach onlarge-scale open source models such as OPT-175B and internal MoE models,showcasing minimal accuracy loss while achieving up to 3.65 times higherthroughput on the same number of GPUs.

FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs