English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, foundation models have exhibited remarkable advancements inmulti-modal learning. These models, equipped with millions (or billions) ofparameters, typically require a substantial amount of data for finetuning.However, collecting and centralizing training data from diverse sectors becomeschallenging due to distinct privacy regulations. Federated Learning (FL)emerges as a promising solution, enabling multiple clients to collaborativelytrain neural networks without centralizing their local data. To alleviateclient computation burdens and communication overheads, previous works haveadapted Parameter-efficient Finetuning (PEFT) methods for FL. Hereby, only asmall fraction of the model parameters are optimized and communicated duringfederated communications. Nevertheless, most previous works have focused on asingle modality and neglected one common phenomenon, i.e., the presence of dataheterogeneity across the clients. Therefore, in this work, we propose afinetuning framework tailored to heterogeneous multi-modal FL, called FederatedDual-Aadapter Teacher (FedDAT). Specifically, our approach leverages aDual-Adapter Teacher (DAT) to address data heterogeneity by regularizing theclient local updates and applying Mutual Knowledge Distillation (MKD) for anefficient knowledge transfer. FedDAT is the first approach that enables anefficient distributed finetuning of foundation models for a variety ofheterogeneous Vision-Language tasks. To demonstrate its effectiveness, weconduct extensive experiments on four multi-modality FL benchmarks withdifferent types of data heterogeneity, where FedDAT substantially outperformsthe existing centralized PEFT methods adapted for FL.

FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning