English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In data-parallel optimization of machine learning models, workers collaborateto improve their estimates of the model: more accurate gradients allow them touse larger learning rates and optimize faster. We consider the setting in whichall workers sample from the same dataset, and communicate over a sparse graph(decentralized). In this setting, current theory fails to capture importantaspects of real-world behavior. First, the 'spectral gap' of the communicationgraph is not predictive of its empirical performance in (deep) learning.Second, current theory does not explain that collaboration enables largerlearning rates than training alone. In fact, it prescribes smaller learningrates, which further decrease as graphs become larger, failing to explainconvergence in infinite graphs. This paper aims to paint an accurate picture ofsparsely-connected distributed optimization when workers share the same datadistribution. We quantify how the graph topology influences convergence in aquadratic toy problem and provide theoretical results for general smooth and(strongly) convex objectives. Our theory matches empirical observations in deeplearning, and accurately describes the relative merits of different graphtopologies.

Beyond spectral gap: The role of the topology in decentralized learning