English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In many applications of federated learning (FL), clients desire models thatare personalized using their local data, yet are also robust in the sense thatthey retain general global knowledge. However, the presence of dataheterogeneity across clients induces a fundamental trade-off betweenpersonalization (i.e., adaptation to a local distribution) and robustness(i.e., not forgetting previously learned general knowledge). It is critical tounderstand how to navigate this personalization vs robustness trade-off whendesigning federated systems, which are increasingly moving towards a paradigmof fine-tuning large foundation models. Due to limited computational andcommunication capabilities in most federated settings, this foundation modelfine-tuning must be done using parameter-efficient fine-tuning (PEFT)approaches. While some recent work has studied federated approaches to PEFT,the personalization vs robustness trade-off of federated PEFT has been largelyunexplored. In this work, we take a step towards bridging this gap bybenchmarking fundamental FL algorithms -- FedAvg and FedSGD pluspersonalization (via client local fine-tuning) -- applied to one of the mostubiquitous PEFT approaches to large language models (LLMs) -- prompt tuning --in a multitude of hyperparameter settings under varying levels of dataheterogeneity. Our results show that federated-trained prompts can besurprisingly robust when using a small learning rate with many local epochs forpersonalization, especially when using an adaptive optimizer as the clientoptimizer during federated training. We also demonstrate that simple approachessuch as adding regularization and interpolating two prompts are effective inimproving the personalization vs robustness trade-off in computation-limitedsettings with few local updates allowed for personalization.

Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuning