English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recent vision-language (VL) models are powerful, but can they reliablydistinguish "right" from "left"? We curate three new corpora to quantify modelcomprehension of such basic spatial relations. These tests isolate spatialreasoning more precisely than existing datasets like VQAv2, e.g., our What'sUpbenchmark contains sets of photographs varying only the spatial relations ofobjects, keeping their identity fixed (see Figure 1: models must comprehend notonly the usual case of a dog under a table, but also, the same dog on top ofthe same table). We evaluate 18 VL models, finding that all perform poorly,e.g., BLIP finetuned on VQAv2, which nears human parity on VQAv2, achieves 56%accuracy on our benchmarks vs. humans at 99%. We conclude by studying causes ofthis surprising behavior, finding: 1) that popular vision-language pretrainingcorpora like LAION-2B contain little reliable data for learning spatialrelationships; and 2) that basic modeling interventions like up-weightingpreposition-containing instances or fine-tuning on our corpora are notsufficient to address the challenges our benchmarks pose. We are hopeful thatthese corpora will facilitate further research, and we release our data andcode at https://github.com/amitakamath/whatsup_vlms.

What's "up" with vision-language models? Investigating their struggle with spatial reasoning