English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

With the increasing amount of multimedia data on modern mobile systems andIoT infrastructures, harnessing these rich multimodal data without breachinguser privacy becomes a critical issue. Federated learning (FL) serves as aprivacy-conscious alternative to centralized machine learning. However,existing FL methods extended to multimodal data all rely on model aggregationon single modality level, which restrains the server and clients to haveidentical model architecture for each modality. This limits the global model interms of both model complexity and data capacity, not to mention taskdiversity. In this work, we propose Contrastive Representation Ensemble andAggregation for Multimodal FL (CreamFL), a multimodal federated learningframework that enables training larger server models from clients withheterogeneous model architectures and data modalities, while only communicatingknowledge on public dataset. To achieve better multimodal representationfusion, we design a global-local cross-modal ensemble strategy to aggregateclient representations. To mitigate local model drift caused by twounprecedented heterogeneous factors stemming from multimodal discrepancy(modality gap and task gap), we further propose two inter-modal and intra-modalcontrasts to regularize local training, which complements information of theabsent modality for uni-modal clients and regularizes local clients to headtowards global consensus. Thorough evaluations and ablation studies onimage-text retrieval and visual question answering tasks showcase thesuperiority of CreamFL over state-of-the-art FL methods and its practicalvalue.

Multimodal Federated Learning via Contrastive Representation Ensemble