English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper investigates the challenges of applying vision-language models(VLMs) to zero-shot visual recognition tasks in an open-world setting, with afocus on contrastive vision-language models such as CLIP. We first examine theperformance of VLMs on concepts of different granularity levels. We propose away to fairly evaluate the performance discrepancy under two experimentalsetups and find that VLMs are better at recognizing fine-grained concepts.Furthermore, we find that the similarity scores from VLMs do not strictlyreflect the correctness of the textual inputs given visual input. We propose anevaluation protocol to test our hypothesis that the scores can be biasedtowards more informative descriptions, and the nature of the similarity scorebetween embedding makes it challenging for VLMs to recognize the correctnessbetween similar but wrong descriptions. Our study highlights the challenges ofusing VLMs in open-world settings and suggests directions for future researchto improve their zero-shot capabilities.

Challenges of Zero-Shot Recognition with Vision-Language Models: Granularity and Correctness