English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Vision and language models (VL) are known to exploit unrobust indicators inindividual modalities (e.g., introduced by distributional biases) instead offocusing on relevant information in each modality. That a unimodal modelachieves similar accuracy on a VL task to a multimodal one, indicates thatso-called unimodal collapse occurred. However, accuracy-based tests fail todetect e.g., when the model prediction is wrong, while the model used relevantinformation from a modality. Instead, we propose MM-SHAP, aperformance-agnostic multimodality score based on Shapley values that reliablyquantifies in which proportions a multimodal model uses individual modalities.We apply MM-SHAP in two ways: (1) to compare models for their average degree ofmultimodality, and (2) to measure for individual models the contribution ofindividual modalities for different tasks and datasets. Experiments with six VLmodels -- LXMERT, CLIP and four ALBEF variants -- on four VL tasks highlightthat unimodal collapse can occur to different degrees and in differentdirections, contradicting the wide-spread assumption that unimodal collapse isone-sided. Based on our results, we recommend MM-SHAP for analysing multimodaltasks, to diagnose and guide progress towards multimodal integration. Codeavailable at \url{https://github.com/Heidelberg-NLP/MM-SHAP}.

MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks