English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Quantization scale and bit-width are the most important parameters whenconsidering how to quantize a neural network. Prior work focuses on optimizingquantization scales in a global manner through gradient methods (gradientdescent \& Hessian analysis). Yet, when applying perturbations to quantizationscales, we observe a very jagged, highly non-smooth test loss landscape. Infact, small perturbations in quantization scale can greatly affect accuracy,yielding a $0.5-0.8\%$ accuracy boost in 4-bit quantized vision transformers(ViTs). In this regime, gradient methods break down, since they cannot reliablyreach local minima. In our work, dubbed Evol-Q, we use evolutionary search toeffectively traverse the non-smooth landscape. Additionally, we propose usingan infoNCE loss, which not only helps combat overfitting on the smallcalibration dataset ($1,000$ images) but also makes traversing such a highlynon-smooth surface easier. Evol-Q improves the top-1 accuracy of a fullyquantized ViT-Base by $10.30\%$, $0.78\%$, and $0.15\%$ for $3$-bit, $4$-bit,and $8$-bit weight quantization levels. Extensive experiments on a variety ofCNN and ViT architectures further demonstrate its robustness in extremequantization scenarios. Our code is available athttps://github.com/enyac-group/evol-q

Jumping through Local Minima: Quantization in the Loss Landscape of Vision Transformers