English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Although vision models such as Contrastive Language-Image Pre-Training (CLIP)show impressive generalization performance, their zero-shot robustness is stilllimited under Out-of-Distribution (OOD) scenarios without fine-tuning. Insteadof undesirably providing human supervision as commonly done, it is possible totake advantage of Multi-modal Large Language Models (MLLMs) that hold powerfulvisual understanding abilities. However, MLLMs are shown to struggle withvision problems due to the incompatibility of tasks, thus hindering theirutilization. In this paper, we propose to effectively leverage MLLMs to conductMachine Vision Therapy which aims to rectify the noisy predictions from visionmodels. By fine-tuning with the denoised labels, the learning model performancecan be boosted in an unsupervised manner. To solve the incompatibility issue,we propose a novel Denoising In-Context Learning (DICL) strategy to alignvision tasks with MLLMs. Concretely, by estimating a transition matrix thatcaptures the probability of one class being confused with another, aninstruction containing a correct exemplar and an erroneous one from the mostprobable noisy class can be constructed. Such an instruction can help any MLLMswith ICL ability to detect and rectify incorrect predictions of vision models.Through extensive experiments on ImageNet, WILDS, DomainBed, and other OODdatasets, we carefully validate the quantitative and qualitative effectivenessof our method. Our code is available athttps://github.com/tmllab/Machine_Vision_Therapy.

Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning