English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We study two factors in neural network training: data parallelism andsparsity; here, data parallelism means processing training data in parallelusing distributed systems (or equivalently increasing batch size), so thattraining can be accelerated; for sparsity, we refer to pruning parameters in aneural network model, so as to reduce computational and memory cost. Despitetheir promising benefits, however, understanding of their effects on neuralnetwork training remains elusive. In this work, we first measure these effectsrigorously by conducting extensive experiments while tuning all metaparametersinvolved in the optimization. As a result, we find across various workloads ofdata set, network model, and optimization algorithm that there exists a generalscaling trend between batch size and number of training steps to convergencefor the effect of data parallelism, and further, difficulty of training undersparsity. Then, we develop a theoretical analysis based on the convergenceproperties of stochastic gradient methods and smoothness of the optimizationlandscape, which illustrates the observed phenomena precisely and generally,establishing a better account of the effects of data parallelism and sparsityon neural network training.

Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training