English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large Language Models (LLMs) have recently demonstrated a remarkable successacross various tasks. However, efficiently serving LLMs has been a challengedue to its large memory bottleneck, specifically in small batch inferencesettings (e.g. mobile devices). Weight-only quantization can be a promisingapproach, but sub-4 bit quantization remains a challenge due to large-magnitudeactivation outliers. To mitigate the undesirable outlier effect, we firstpropose per-IC quantization, a simple yet effective method that createsquantization groups within each input channel (IC) rather than the conventionalper-output channel (OC). Our method is motivated by the observation thatactivation outliers affect the input dimension of the weight matrix, sosimilarly grouping the weights in the IC direction can isolate outliers to bewithin a group. We also find that activation outliers do not dictatequantization difficulty, and inherent weight sensitivities also exist. Withper-IC quantization as a new outlier-friendly scheme, we then propose AdaptiveDimensions (AdaDim), a versatile quantization framework that can adapt tovarious weight sensitivity patterns. We demonstrate the effectiveness of AdaDimby augmenting prior methods such as Round-To-Nearest and GPTQ, showingsignificant improvements across various language modeling benchmarks for bothbase (up to +4.7% on MMLU) and instruction-tuned (up to +10% on HumanEval)LLMs.

Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models