English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In the last year alone, a surge of new benchmarks to measure compositionalunderstanding of vision-language models have permeated the machine learningecosystem. Given an image, these benchmarks probe a model's ability to identifyits associated caption amongst a set of compositional distractors.Surprisingly, we find significant biases in all these benchmarks rendering themhackable. This hackability is so dire that blind models with no access to theimage outperform state-of-the-art vision-language models. To remedy thisrampant vulnerability, we introduce SugarCrepe, a new benchmark forvision-language compositionality evaluation. We employ large language models,instead of rule-based templates used in previous benchmarks, to generate fluentand sensical hard negatives, and utilize an adversarial refinement mechanism tomaximally reduce biases. We re-evaluate state-of-the-art models and recentlyproposed compositionality inducing strategies, and find that their improvementswere hugely overestimated, suggesting that more innovation is needed in thisimportant direction. We release SugarCrepe and the code for evaluation at:https://github.com/RAIVNLab/sugar-crepe.

SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality