English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Data parallelism has emerged as a necessary technique to accelerate thetraining of deep neural networks (DNN). In a typical data parallelism approach,the local workers push the latest updates of all the parameters to theparameter server and pull all merged parameters back periodically. However,with the increasing size of DNN models and the large number of workers inpractice, this typical data parallelism cannot achieve satisfactory trainingacceleration, since it usually suffers from the heavy communication cost due totransferring huge amount of information between workers and the parameterserver. In-depth understanding on DNN has revealed that it is usually highlyredundant, that deleting a considerable proportion of the parameters will notsignificantly decline the model performance. This redundancy property exposes agreat opportunity to reduce the communication cost by only transferring theinformation of those significant parameters during the parallel training.However, if we only transfer information of temporally significant parametersof the latest snapshot, we may miss the parameters that are insignificant nowbut have potential to become significant as the training process goes on. Tothis end, we design an Explore-Exploit framework to dynamically choose thesubset to be communicated, which is comprised of the significant parameters inthe latest snapshot together with a random explored set of other parameters. Wepropose to measure the significance of the parameter by the combination of itsmagnitude and gradient. Our experimental results demonstrate that our proposedSlim-DP can achieve better training acceleration than standard data parallelismand its communication-efficient version by saving communication time withoutloss of accuracy.

Slim-DP: A Light Communication Data Parallelism for DNN