English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Humans show language-biased image recognition for a word-embedded image,known as picture-word interference. Such interference depends on hierarchicalsemantic categories and reflects that human language processing highlyinteracts with visual processing. Similar to humans, recent artificial modelsjointly trained on texts and images, e.g., OpenAI CLIP, show language-biasedimage classification. Exploring whether the bias leads to interference similarto those observed in humans can contribute to understanding how much the modelacquires hierarchical semantic representations from joint learning of languageand vision. The present study introduces methodological tools from thecognitive science literature to assess the biases of artificial models.Specifically, we introduce a benchmark task to test whether words superimposedon images can distort the image classification across different category levelsand, if it can, whether the perturbation is due to the shared semanticrepresentation between language and vision. Our dataset is a set ofword-embedded images and consists of a mixture of natural image datasets andhierarchical word labels with superordinate/basic category levels. Using thisbenchmark test, we evaluate the CLIP model. We show that presenting wordsdistorts the image classification by the model across different categorylevels, but the effect does not depend on the semantic relationship betweenimages and embedded words. This suggests that the semantic word representationin the CLIP visual processing is not shared with the image representation,although the word representation strongly dominates for word-embedded images.

Language-biased image classification: evaluation based on semantic representations