English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

A standard approach in large scale machine learning is distributed stochasticgradient training, which requires the computation of aggregated stochasticgradients over multiple nodes on a network. Communication is a major bottleneckin such applications, and in recent years, compressed stochastic gradientmethods such as QSGD (quantized SGD) and sparse SGD have been proposed toreduce communication. It was also shown that error compensation can be combinedwith compression to achieve better convergence in a scheme that each nodecompresses its local stochastic gradient and broadcast the result to all othernodes over the network in a single pass. However, such a single pass broadcastapproach is not realistic in many practical implementations. For example, underthe popular parameter server model for distributed learning, the worker nodesneed to send the compressed local gradients to the parameter server, whichperforms the aggregation. The parameter server has to compress the aggregatedstochastic gradient again before sending it back to the worker nodes. In thiswork, we provide a detailed analysis on this two-pass communication model andits asynchronous parallel variant, with error-compensated compression both onthe worker nodes and on the parameter server. We show that theerror-compensated stochastic gradient algorithm admits three very niceproperties: 1) it is compatible with an \emph{arbitrary} compression technique;2) it admits an improved convergence rate than the non error-compensatedstochastic gradient methods such as QSGD and sparse SGD; 3) it admits linearspeedup with respect to the number of workers. The empirical study is alsoconducted to validate our theoretical results.

DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression