English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent work has explored the potential to adapt a pre-trained visiontransformer (ViT) by updating only a few parameters so as to improve storageefficiency, called parameter-efficient transfer learning (PETL). Current PETLmethods have shown that by tuning only 0.5% of the parameters, ViT can beadapted to downstream tasks with even better performance than full fine-tuning.In this paper, we aim to further promote the efficiency of PETL to meet theextreme storage constraint in real-world applications. To this end, we proposea tensorization-decomposition framework to store the weight increments, inwhich the weights of each ViT are tensorized into a single 3D tensor, and theirincrements are then decomposed into lightweight factors. In the fine-tuningprocess, only the factors need to be updated and stored, termed Factor-Tuning(FacT). On VTAB-1K benchmark, our method performs on par with NOAH, thestate-of-the-art PETL method, while being 5x more parameter-efficient. We alsopresent a tiny version that only uses 8K (0.01% of ViT's parameters) trainableparameters but outperforms full fine-tuning and many other PETL methods such asVPT and BitFit. In few-shot settings, FacT also beats all PETL baselines usingthe fewest parameters, demonstrating its strong capability in the low-dataregime.

FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer