English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Multimodal learning seeks to utilize data from multiple sources to improvethe overall performance of downstream tasks. It is desirable for redundanciesin the data to make multimodal systems robust to missing or corruptedobservations in some correlated modalities. However, we observe that theperformance of several existing multimodal networks significantly deterioratesif one or multiple modalities are absent at test time. To enable robustness tomissing modalities, we propose simple and parameter-efficient adaptationprocedures for pretrained multimodal networks. In particular, we exploitlow-rank adaptation and modulation of intermediate features to compensate forthe missing modalities. We demonstrate that such adaptation can partiallybridge performance drop due to missing modalities and outperform independent,dedicated networks trained for the available modality combinations in somecases. The proposed adaptation requires extremely small number of parameters(e.g., fewer than 0.7% of the total parameters in most experiments). We conducta series of experiments to highlight the robustness of our proposed methodusing diverse datasets for RGB-thermal and RGB-Depth semantic segmentation,multimodal material segmentation, and multimodal sentiment analysis tasks. Ourproposed method demonstrates versatility across various tasks and datasets, andoutperforms existing methods for robust multimodal learning with missingmodalities.

Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation