English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Current multimodal models, aimed at solving Vision and Language (V+L) tasks,predominantly repurpose Vision Encoders (VE) as feature extractors. While manyVEs -- of different architectures, trained on different data and objectives --are publicly available, they are not designed for the downstream V+L tasks.Nonetheless, most current work assumes that a \textit{single} pre-trained VEcan serve as a general-purpose encoder. In this work, we focus on analysis andaim to understand whether the information stored within different VEs iscomplementary, i.e. if providing the model with features from multiple VEs canimprove the performance on a target task, and how they are combined. Weexhaustively experiment with three popular VEs on six downstream V+L tasks andanalyze the attention and VE-dropout patterns. Our analyses suggest thatdiverse VEs complement each other, resulting in improved downstream V+L taskperformance, where the improvements are not due to simple ensemble effects(i.e. the performance does not always improve when increasing the number ofencoders). We demonstrate that future VEs, which are not \textit{repurposed},but explicitly \textit{designed} for V+L tasks, have the potential of improvingperformance on the target V+L tasks.

One does not fit all! On the Complementarity of Vision Encoders for Vision and Language Tasks