English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While vision transformers (ViTs) have continuously achieved new milestones inthe field of computer vision, their sophisticated network architectures withhigh computation and memory costs have impeded their deployment onresource-limited edge devices. In this paper, we propose a hardware-efficientimage-adaptive token pruning framework called HeatViT for efficient yetaccurate ViT acceleration on embedded FPGAs. By analyzing the inherentcomputational patterns in ViTs, we first design an effective attention-basedmulti-head token selector, which can be progressively inserted beforetransformer blocks to dynamically identify and consolidate the non-informativetokens from input images. Moreover, we implement the token selector on hardwareby adding miniature control logic to heavily reuse existing hardware componentsbuilt for the backbone ViT. To improve the hardware efficiency, we furtheremploy 8-bit fixed-point quantization, and propose polynomial approximationswith regularization effect on quantization error for the frequently usednonlinear functions in ViTs. Finally, we propose a latency-aware multi-stagetraining strategy to determine the transformer blocks for inserting tokenselectors and optimize the desired (average) pruning rates for inserted tokenselectors, in order to improve both the model accuracy and inference latency onhardware. Compared to existing ViT pruning studies, under the similarcomputation cost, HeatViT can achieve 0.7%$\sim$8.9% higher accuracy; whileunder the similar model accuracy, HeatViT can achieve more than28.4%$\sim$65.3% computation reduction, for various widely used ViTs, includingDeiT-T, DeiT-S, DeiT-B, LV-ViT-S, and LV-ViT-M, on the ImageNet dataset.Compared to the baseline hardware accelerator, our implementations of HeatViTon the Xilinx ZCU102 FPGA achieve 3.46$\times$$\sim$4.89$\times$ speedup.

HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers