English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large multi-modal models (LMMs) hold the potential to usher in a new era ofautomated visual assistance for people who are blind or low vision (BLV). Yet,these models have not been systematically evaluated on data captured by BLVusers. We address this by empirically assessing CLIP, a widely-used LMM likelyto underpin many assistive technologies. Testing 25 CLIP variants in azero-shot classification task, we find that their accuracy is 15 percentagepoints lower on average for images captured by BLV users than web-crawledimages. This disparity stems from CLIP's sensitivities to 1) image content(e.g. not recognizing disability objects as well as other objects); 2) imagequality (e.g. not being robust to lighting variation); and 3) text content(e.g. not recognizing objects described by tactile adjectives as well as visualones). We delve deeper with a textual analysis of three common pre-trainingdatasets: LAION-400M, LAION-2B and DataComp-1B, showing that disability contentis rarely mentioned. We then provide three examples that illustrate how theperformance disparities extend to three downstream models underpinned by CLIP:OWL-ViT, CLIPSeg and DALL-E2. We find that few-shot learning with as few as 5images can mitigate CLIP's quality-of-service disparities for BLV users in somescenarios, which we discuss alongside a set of other possible mitigations.

Explaining CLIP's performance disparities on data from blind/low vision users