English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Deep learning has permeated through many aspects of computing/processingsystems in recent years. While distributed training architectures/frameworksare adopted for training large deep learning models quickly, there has not beena systematic study of the communication bottlenecks of these architectures andtheir effects on the computation cycle time and scalability. In order toanalyze this problem for synchronous Stochastic Gradient Descent (SGD) trainingof deep learning models, we developed a performance model of computation timeand communication latency under three different system architectures: ParameterServer (PS), peer-to-peer (P2P), and Ring allreduce (RA). To complement andcorroborate our analytical models with quantitative results, we evaluated thecomputation and communication performance of these system architectures of thesystems via experiments performed with Tensorflow and Horovod frameworks. Wefound that the system architecture has a very significant effect on theperformance of training. RA-based systems achieve scalable performance as theysuccessfully decouple network usage from the number of workers in the system.In contrast, 1PS systems suffer from low performance due to network congestionat the parameter server side. While P2P systems fare better than 1PS systems,they still suffer from significant network bottleneck. Finally, RA systems alsoexcel by virtue of overlapping computation time and communication time, whichPS and P2P architectures fail to achieve.

Performance Analysis and Comparison of Distributed Machine Learning Systems