English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Foundation models (FMs) have shown prominent success in a wide range oftasks. Their applicability to specific domain-task pairings relies on theavailability of, both, high-quality data and significant computationalresources. These challenges are not new to the field and, indeed, FederatedLearning (FL) has been shown to be a promising solution in similar setups. Thispaper tackles the specific case of Domain-Adaptive Pre-Training (DAPT), a keystep in the application of FMs. We conduct the first comprehensive empiricalstudy to evaluate the performance of Federated Domain-Adaptive Pre-Training(FDAPT). We demonstrate that FDAPT can maintain competitive downstream taskperformance to the centralized baseline in both IID and non-IID situations.Finally, we propose a novel algorithm, Frozen Federated Domain-AdaptivePre-Training (FFDAPT). FFDAPT improves the computational efficiency by 12.1% onaverage and exhibits similar downstream task performance to vanilla FDAPT, withgeneral performance fluctuations remaining less than 1%.

FDAPT: Federated Domain-adaptive Pre-training for Language Models