English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The robustness of multimodal deep learning models to realistic changes in theinput text is critical for their applicability to important tasks such astext-to-image retrieval and cross-modal entailment. To measure robustness,several existing approaches edit the text data, but do so without leveragingthe cross-modal information present in multimodal data. Information from thevisual modality, such as color, size, and shape, provide additional attributesthat users can include in their inputs. Thus, we propose cross-modal attributeinsertions as a realistic perturbation strategy for vision-and-language datathat inserts visual attributes of the objects in the image into thecorresponding text (e.g., "girl on a chair" to "little girl on a woodenchair"). Our proposed approach for cross-modal attribute insertions is modular,controllable, and task-agnostic. We find that augmenting input text usingcross-modal insertions causes state-of-the-art approaches for text-to-imageretrieval and cross-modal entailment to perform poorly, resulting in relativedrops of 15% in MRR and 20% in $F_1$ score, respectively. Crowd-sourcedannotations demonstrate that cross-modal insertions lead to higher qualityaugmentations for multimodal data than augmentations using text-only data, andare equivalent in quality to original examples. We release the code toencourage robustness evaluations of deep vision-and-language models:https://github.com/claws-lab/multimodal-robustness-xmai.

Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning