English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Federated learning (FL) is a promising paradigm to enable collaborative modeltraining with decentralized data. However, the training process of LargeLanguage Models (LLMs) generally incurs the update of significant parameters,which limits the applicability of FL techniques to tackle the LLMs in realscenarios. Prompt tuning can significantly reduce the number of parameters toupdate, but it either incurs performance degradation or low trainingefficiency. The straightforward utilization of prompt tuning in the FL oftenraises non-trivial communication costs and dramatically degrades performance.In addition, the decentralized data is generally non-Independent andIdentically Distributed (non-IID), which brings client drift problems and thuspoor performance. This paper proposes a Parameter-efficient prompt Tuningapproach with Adaptive Optimization, i.e., FedPepTAO, to enable efficient andeffective FL of LLMs. First, an efficient partial prompt tuning approach isproposed to improve performance and efficiency simultaneously. Second, a noveladaptive optimization method is developed to address the client drift problemson both the device and server sides to enhance performance further. Extensiveexperiments based on 10 datasets demonstrate the superb performance (up to60.8\% in terms of accuracy) and efficiency (up to 97.59\% in terms of trainingtime) of FedPepTAO compared with 9 baseline approaches. Our code is availableat https://github.com/llm-eff/FedPepTAO.

Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization