English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Distributed synchronous stochastic gradient descent (S-SGD) has been widelyused in training large-scale deep neural networks (DNNs), but it typicallyrequires very high communication bandwidth between computational workers (e.g.,GPUs) to exchange gradients iteratively. Recently, Top-$k$ sparsificationtechniques have been proposed to reduce the volume of data to be exchangedamong workers. Top-$k$ sparsification can zero-out a significant portion ofgradients without impacting the model convergence. However, the sparsegradients should be transferred with their irregular indices, which makes thesparse gradients aggregation difficult. Current methods that use AllGather toaccumulate the sparse gradients have a communication complexity of $O(kP)$,where $P$ is the number of workers, which is inefficient on low bandwidthnetworks with a large number of workers. We observe that not all top-$k$gradients from $P$ workers are needed for the model update, and therefore wepropose a novel global Top-$k$ (gTop-$k$) sparsification mechanism to addressthe problem. Specifically, we choose global top-$k$ largest absolute values ofgradients from $P$ workers, instead of accumulating all local top-$k$ gradientsto update the model in each iteration. The gradient aggregation method based ongTop-$k$ sparsification reduces the communication complexity from $O(kP)$ to$O(k\log P)$. Through extensive experiments on different DNNs, we verify thatgTop-$k$ S-SGD has nearly consistent convergence performance with S-SGD, and ithas only slight degradations on generalization performance. In terms of scalingefficiency, we evaluate gTop-$k$ on a cluster with 32 GPU machines which areinterconnected with 1 Gbps Ethernet. The experimental results show that ourmethod achieves $2.7-12\times$ higher scaling efficiency than S-SGD and$1.1-1.7\times$ improvement than the existing Top-$k$ S-SGD.

A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks