English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we question the rationale behind propagating large numbers ofparameters through a distributed system during federated learning. We start byexamining the rank characteristics of the subspace spanned by gradients acrossepochs (i.e., the gradient-space) in centralized model training, and observethat this gradient-space often consists of a few leading principal componentsaccounting for an overwhelming majority (95-99%) of the explained variance.Motivated by this, we propose the "Look-back Gradient Multiplier" (LBGM)algorithm, which exploits this low-rank property to enable gradient recyclingbetween model update rounds of federated learning, reducing transmissions oflarge parameters to single scalars for aggregation. We analyticallycharacterize the convergence behavior of LBGM, revealing the nature of thetrade-off between communication savings and model performance. Our subsequentexperimental results demonstrate the improvement LBGM obtains in communicationoverhead compared to conventional federated learning on several datasets anddeep learning models. Additionally, we show that LBGM is a generalplug-and-play algorithm that can be used standalone or stacked on top ofexisting sparsification techniques for distributed model training.

Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?