English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Soft prompt learning has recently emerged as one of the methods of choice foradapting V&L models to a downstream task using a few training examples.However, current methods significantly overfit the training data, sufferingfrom large accuracy degradation when tested on unseen classes from the samedomain. To this end, in this paper, we make the following 4 contributions: (1)To alleviate base class overfitting, we propose a novel Language-Aware SoftPrompting (LASP) learning method by means of a text-to-text cross-entropy lossthat maximizes the probability of the learned prompts to be correctlyclassified with respect to pre-defined hand-crafted textual prompts. (2) Toincrease the representation capacity of the prompts, we propose grouped LASPwhere each group of prompts is optimized with respect to a separate subset oftextual prompts. (3) We identify a visual-language misalignment introduced byprompt learning and LASP, and more importantly, propose a re-calibrationmechanism to address it. (4) We show that LASP is inherently amenable toincluding, during training, virtual classes, i.e. class names for which novisual samples are available, further increasing the robustness of the learnedprompts. Through evaluations on 11 datasets, we show that our approach (a)significantly outperforms all prior works on soft prompting, and (b) matchesand surpasses, for the first time, the accuracy on novel classes obtained byhand-crafted prompts and CLIP for 8 out of 11 test datasets. Code will be madeavailable at https://www.adrianbulat.com/lasp

LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models