English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Language model training in distributed settings is limited by thecommunication cost of gradient exchanges. In this short note, we extend recentwork from Malladi et al. (2023), using shared randomness to perform distributedfine-tuning with low bandwidth. The method is a natural decentralized extensionof memory-efficient Simultaneous Perturbation Stochastic Approximation (SPSA).Each iteration, each machine seeds a Random Number Generator (RNG) to performlocal reproducible perturbations on model weights and calculate and exchangescalar projected gradients, which are then used to update each model. By usinga (machine, sample) identifier as the random seed, each model can regenerateone another's perturbations. As machines only exchange single-byte projectedgradients, this is highly communication efficient. There are also potentialprivacy benefits, as projected gradients may be calculated on differenttraining data, and models never access the other's data. Our approach not onlydrastically reduces communication bandwidth requirements but also accommodatesdynamic addition or removal of machines during the training process and retainsthe memory-efficient and inference-only advantages of recent work. We performproof-of-concept experiments to demonstrate the potential usefulness of thismethod, building off of rich literature on distributed optimization andmemory-efficient training.

Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomness