English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

While Multi-modal Language Models (MLMs) demonstrate impressive multimodalability, they still struggle on providing factual and precise responses fortasks like visual question answering (VQA). In this paper, we address thischallenge from the perspective of contextual information. We propose CausalContext Generation, Causal-CoG, which is a prompting strategy that engagescontextual information to enhance precise VQA during inference. Specifically,we prompt MLMs to generate contexts, i.e, text description of an image, andengage the generated contexts for question answering. Moreover, we investigatethe advantage of contexts on VQA from a causality perspective, introducingcausality filtering to select samples for which contextual information ishelpful. To show the effectiveness of Causal-CoG, we run extensive experimentson 10 multimodal benchmarks and show consistent improvements, e.g., +6.30% onPOPE, +13.69% on Vizwiz and +6.43% on VQAv2 compared to direct decoding,surpassing existing methods. We hope Casual-CoG inspires explorations ofcontext knowledge in multimodal models, and serves as a plug-and-play strategyfor MLM decoding.

Causal-CoG: A Causal-Effect Look at Context Generation for Boosting Multi-modal Language Models