English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Parallel training of neural networks at scale is challenging due tosignificant overheads arising from communication. Recently, deep learningresearchers have developed a variety of pruning algorithms that are capable ofpruning (i.e. setting to zero) 80-90% of the parameters in a neural network toyield sparse subnetworks that equal the accuracy of the unpruned parentnetwork. In this work, we propose a novel approach that exploits these sparsesubnetworks to optimize the memory utilization and communication in two popularalgorithms for parallel deep learning namely -- data and inter-layerparallelism. We integrate our approach into AxoNN, a highly scalable frameworkfor parallel deep learning that relies on data and inter-layer parallelism, anddemonstrate the reduction in communication time and memory utilization. On 512NVIDIA V100 GPUs, our optimizations reduce the memory consumption of a 2.7billion parameter model by 74%, and the total communication time by 40%, thusproviding an overall speedup of 34% over AxoNN, 32% over DeepSpeed-3D and 46%over Sputnik, a sparse matrix computation baseline.

Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Training