English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pre-trained language models (PLMs) have played an increasing role inmultimedia research. In terms of vision-language (VL) tasks, they often serveas a language encoder and still require an additional fusion network for VLreasoning, resulting in excessive memory overhead. In this paper, we focus onexploring PLMs as a stand-alone model for VL reasoning tasks. Inspired by therecently popular prompt tuning, we first prove that the processed visualfeatures can be also projected onto the semantic space of PLMs and act asprompt tokens to bridge the gap between single- and multi-modal learning.However, this solution exhibits obvious redundancy in visual information andmodel inference, and the placement of prompt tokens also greatly affects thefinal performance. Based on these observations, we further propose a noveltransfer learning approach for PLMs, termed Dynamic Visual Prompting (DVP).Concretely, DVP first deploys a cross-attention module to obtain text-relatedand compact visual prompt tokens, thereby greatly reducing the input length ofPLMs. To obtain the optimal placement, we also equip DVP with areinforcement-learning based search algorithm, which can automatically mergeDVP with PLMs for different VL tasks via a very short search process. Inaddition, we also experiment DVP with the recently popular adapter approach tokeep the most parameters of PLMs intact when adapting to VL tasks, helping PLMsachieve a quick shift between single- and multi-modal tasks. We apply DVP totwo representative PLMs, namely BERT and T5, and conduct extensive experimentson a set of VL reasoning benchmarks including VQA2.0, GQA and SNLIVE. Theexperimental results not only show the advantage of DVP on efficiency andperformance, but also confirm its superiority in adapting pre-trained languagemodels to VL tasks.

Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting