English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Many popular distributed optimization methods for training machine learningmodels fit the following template: a local gradient estimate is computedindependently by each worker, then communicated to a master, which subsequentlyperforms averaging. The average is broadcast back to the workers, which use itto perform a gradient-type step to update the local version of the model. It isalso well known that many such methods, including SGD, SAGA, and acceleratedSGD for over-parameterized models, do not scale well with the number ofparallel workers. In this paper we observe that the above template isfundamentally inefficient in that too much data is unnecessarily communicatedby the workers, which slows down the overall system. We propose a fix based ona new update-sparsification method we develop in this work, which we suggest beused on top of existing methods. Namely, we develop a new variant of parallelblock coordinate descent based on independent sparsification of the localgradient estimates before communication. We demonstrate that with only $m/n$blocks sent by each of $n$ workers, where $m$ is the total number of parameterblocks, the theoretical iteration complexity of the underlying distributedmethods is essentially unaffected. As an illustration, this means that when$n=100$ parallel workers are used, the communication of $99\%$ blocks isredundant, and hence a waste of time. Our theoretical claims are supportedthrough extensive numerical experiments which demonstrate an almost perfectmatch with our theory on a number of synthetic and real datasets.

99% of Distributed Optimization is a Waste of Time: The Issue and How to Fix it