English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vision Transformer (ViT) has performed remarkably in various computer visiontasks. Nonetheless, affected by the massive amount of parameters, ViT usuallysuffers from serious overfitting problems with a relatively limited number oftraining samples. In addition, ViT generally demands heavy computing resources,which limit its deployment on resource-constrained devices. As a type ofmodel-compression method, model binarization is potentially a good choice tosolve the above problems. Compared with the full-precision one, the model withthe binarization method replaces complex tensor multiplication with simplebit-wise binary operations and represents full-precision model parameters andactivations with only 1-bit ones, which potentially solves the problem of modelsize and computational complexity, respectively. In this paper, we investigatea binarized ViT model. Empirically, we observe that the existing binarizationtechnology designed for Convolutional Neural Networks (CNN) cannot migrate wellto a ViT's binarization task. We also find that the decline of the accuracy ofthe binary ViT model is mainly due to the information loss of the Attentionmodule and the Value vector. Therefore, we propose a novel model binarizationtechnique, called Group Superposition Binarization (GSB), to deal with theseissues. Furthermore, in order to further improve the performance of thebinarization model, we have investigated the gradient calculation procedure inthe binarization process and derived more proper gradient calculation equationsfor GSB to reduce the influence of gradient mismatch. Then, the knowledgedistillation technique is introduced to alleviate the performance degradationcaused by model binarization. Analytically, model binarization can limit theparameters search space during parameter updates while training a model....

GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples