English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

In distributed training of deep neural networks, parallel mini-batch SGD iswidely used to speed up the training process by using multiple workers. It usesmultiple workers to sample local stochastic gradient in parallel, aggregatesall gradients in a single server to obtain the average, and update eachworker's local model using a SGD update with the averaged gradient. Ideally,parallel mini-batch SGD can achieve a linear speed-up of the training time(with respect to the number of workers) compared with SGD over a single worker.However, such linear scalability in practice is significantly limited by thegrowing demand for gradient communication as more workers are involved. Modelaveraging, which periodically averages individual models trained over parallelworkers, is another common practice used for distributed training of deepneural networks since (Zinkevich et al. 2010) (McDonald, Hall, and Mann 2010).Compared with parallel mini-batch SGD, the communication overhead of modelaveraging is significantly reduced. Impressively, tremendous experimental workshave verified that model averaging can still achieve a good speed-up of thetraining time as long as the averaging interval is carefully controlled.However, it remains a mystery in theory why such a simple heuristic works sowell. This paper provides a thorough and rigorous theoretical study on whymodel averaging can work as well as parallel mini-batch SGD with significantlyless communication overhead.

Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning