English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

With the growing success of multi-modal learning, research on the robustnessof multi-modal models, especially when facing situations with missingmodalities, is receiving increased attention. Nevertheless, previous studies inthis domain exhibit certain limitations, as they often lack theoreticalinsights or their methodologies are tied to specific network architectures ormodalities. We model the scenarios of multi-modal models encountering missingmodalities from an information-theoretic perspective and illustrate that theperformance ceiling in such scenarios can be approached by efficientlyutilizing the information inherent in non-missing modalities. In practice,there are two key aspects: (1) The encoder should be able to extractsufficiently good features from the non-missing modality; (2) The extractedfeatures should be robust enough not to be influenced by noise during thefusion process across modalities. To this end, we introduce Uni-Modal Ensemblewith Missing Modality Adaptation (UME-MMA). UME-MMA employs uni-modalpre-trained weights for the multi-modal model to enhance feature extraction andutilizes missing modality data augmentation techniques to better adapt tosituations with missing modalities. Apart from that, UME-MMA, built on alate-fusion learning framework, allows for the plug-and-play use of variousencoders, making it suitable for a wide range of modalities and enablingseamless integration of large-scale pre-trained encoders to further enhanceperformance. And we demonstrate UME-MMA's effectiveness in audio-visualdatasets~(e.g., AV-MNIST, Kinetics-Sound, AVE) and vision-languagedatasets~(e.g., MM-IMDB, UPMC Food101).

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?