English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Weight decay is a standard technique to improve generalization performance inmodern deep neural network optimization, and is also widely adopted infederated learning (FL) to prevent overfitting in local clients. In this paper,we first explore the choices of weight decay and identify that weight decayvalue appreciably influences the convergence of existing FL algorithms. Whilepreventing overfitting is crucial, weight decay can introduce a differentoptimization goal towards the global objective, which is further amplified inFL due to multiple local updates and heterogeneous data distribution. Toaddress this challenge, we develop {\it Federated optimization with NormalizedAnnealing Regularization} (FedNAR), a simple yet effective and versatilealgorithmic plug-in that can be seamlessly integrated into any existing FLalgorithms. Essentially, we regulate the magnitude of each update by performingco-clipping of the gradient and weight decay. We provide a comprehensivetheoretical analysis of FedNAR's convergence rate and conduct extensiveexperiments on both vision and language datasets with different backbonefederated optimization algorithms. Our experimental results consistentlydemonstrate that incorporating FedNAR into existing FL algorithms leads toaccelerated convergence and heightened model accuracy. Moreover, FedNARexhibits resilience in the face of various hyperparameter configurations.Specifically, FedNAR has the ability to self-adjust the weight decay when theinitial specification is not optimal, while the accuracy of traditional FLalgorithms would markedly decline. Our codes are released at\href{https://github.com/ljb121002/fednar}{https://github.com/ljb121002/fednar}.

FedNAR: Federated Optimization with Normalized Annealing Regularization