English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

An oft-cited challenge of federated learning is the presence ofheterogeneity. \emph{Data heterogeneity} refers to the fact that data fromdifferent clients may follow very different distributions. \emph{Systemheterogeneity} refers to the fact that client devices have different systemcapabilities. A considerable number of federated optimization methods addressthis challenge. In the literature, empirical evaluations usually startfederated training from random initialization. However, in many practicalapplications of federated learning, the server has access to proxy data for thetraining task that can be used to pre-train a model before starting federatedtraining. We empirically study the impact of starting from a pre-trained modelin federated learning using four standard federated learning benchmarkdatasets. Unsurprisingly, starting from a pre-trained model reduces thetraining time required to reach a target error rate and enables the training ofmore accurate models (up to 40\%) than is possible when starting from randominitialization. Surprisingly, we also find that starting federated learningfrom a pre-trained initialization reduces the effect of both data and systemheterogeneity. We recommend that future work proposing and evaluating federatedoptimization methods evaluate the performance when starting from random andpre-trained initializations. We also believe this study raises severalquestions for further work on understanding the role of heterogeneity infederated optimization.

Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning