English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The complicated architecture and high training cost of vision transformersurge the exploration of post-training quantization. However, the heavy-taileddistribution of vision transformer activations hinders the effectiveness ofprevious post-training quantization methods, even with advanced quantizerdesigns. Instead of tuning the quantizer to better fit the complicatedactivation distribution, this paper proposes NoisyQuant, a quantizer-agnosticenhancement for the post-training activation quantization performance of visiontransformers. We make a surprising theoretical discovery that for a givenquantizer, adding a fixed Uniform noisy bias to the values being quantized cansignificantly reduce the quantization error under provable conditions. Buildingon the theoretical insight, NoisyQuant achieves the first success on activelyaltering the heavy-tailed activation distribution with additive noisy bias tofit a given quantizer. Extensive experiments show NoisyQuant largely improvesthe post-training quantization performance of vision transformer with minimalcomputation overhead. For instance, on linear uniform 6-bit activationquantization, NoisyQuant improves SOTA top-1 accuracy on ImageNet by up to1.7%, 1.1% and 0.5% for ViT, DeiT, and Swin Transformer respectively, achievingon-par or even higher performance than previous nonlinear, mixed-precisionquantization.

NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers