English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pre-trained large language models (LLMs) require fine-tuning to improve theirresponsiveness to natural language instructions. Federated learning (FL) offersa way to perform fine-tuning using the abundant data on end devices withoutcompromising data privacy. Most existing federated fine-tuning methods for LLMsrely on parameter-efficient fine-tuning techniques, which may not reach theperformance heights possible with full-parameter tuning. However, thecommunication overhead associated with full-parameter tuning is prohibitivelyhigh for both servers and clients. This work introduces FedKSeed, a novelapproach that employs zeroth-order optimization (ZOO) with a set of randomseeds. It enables federated full-parameter tuning of billion-sized LLMsdirectly on devices. Our method significantly reduces transmission requirementsbetween the server and clients to just a few scalar gradients and random seeds,amounting to only a few thousand bytes. Building on this, we develop a strategyto assess the significance of ZOO perturbations for FL, allowing forprobability-differentiated seed sampling. This prioritizes perturbations thathave a greater impact on model accuracy. Experiments across six scenarios withdifferent LLMs, datasets and data partitions demonstrate that our approachoutperforms existing federated LLM fine-tuning methods in terms of bothcommunication efficiency and new task generalization.

Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes