English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Visual language tasks require AI models to comprehend and reason with bothvisual and textual content. Driven by the power of Large Language Models(LLMs), two prominent methods have emerged: (1) the hybrid integration betweenLLMs and Vision-Language Models (VLMs), where visual inputs are firstlyconverted into language descriptions by VLMs, serving as inputs for LLMs togenerate final answer(s); (2) visual feature alignment in language space, wherevisual inputs are encoded as embeddings and projected to LLMs' language spacevia further supervised fine-tuning. The first approach provides light trainingcosts and interpretability but is hard to be optimized in an end-to-endfashion. The second approach presents decent performance, but feature alignmentusually requires large amounts of training data and lacks interpretability. Totackle this dilemma, we propose a novel approach, Inner Monologue Multi-ModalOptimization (IMMO), to solve complex vision language problems by simulatinginner monologue processes, a cognitive process in which an individual engagesin silent verbal communication with themselves. We enable LLMs and VLMs tointeract through natural language conversation and propose to use a two-stagetraining process to learn how to do the inner monologue (self-asking questionsand answering questions). IMMO is evaluated on two popular tasks and theresults suggest by emulating the cognitive phenomenon of internal dialogue, ourapproach can enhance reasoning and explanation abilities, contributing to themore effective fusion of vision and language models. More importantly, insteadof using predefined human-crafted monologues, IMMO learns this process withinthe deep learning models, promising wider applicability to many different AIproblems beyond vision language tasks.

Tackling Vision Language Tasks Through Learning Inner Monologues