English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

We explore scaling of the standard distributed Tensorflow with GRPCprimitives on up to 512 Intel Xeon Phi (KNL) nodes of Cori supercomputer withsynchronous stochastic gradient descent (SGD), and identify causes of scalinginefficiency at higher node counts. To our knowledge, this is the firstexploration of distributed GRPC Tensorflow scalability on a HPC supercomputerat such large scale with synchronous SGD. We studied scaling of two convolutionneural networks - ResNet-50, a state-of-the-art deep network for classificationwith roughly 25.5 million parameters, and HEP-CNN, a shallow topology with lessthan 1 million parameters for common scientific usages. For ResNet-50, weachieve >80% scaling efficiency on up to 128 workers, using 32 parameterservers (PS tasks) with a steep decline down to 23% for 512 workers using 64 PStasks. Our analysis of the efficiency drop points to low network bandwidthutilization due to combined effect of three factors. (a) Heterogeneousdistributed parallelization algorithm which uses PS tasks as centralizedservers for gradient averaging is suboptimal for utilizing interconnectbandwidth. (b) Load imbalance among PS tasks hinders their efficient scaling.(c) Underlying communication primitive GRPC is currently inefficient on Corihigh-speed interconnect. The HEP-CNN demands less interconnect bandwidth, andshows >80% weak scaling efficiency for up to 256 nodes with only 1 PS task. Ourfindings are applicable to other deep learning networks. Big networks withmillions of parameters stumble upon the issues discussed here. Shallowernetworks like HEP-CNN with relatively lower number of parameters canefficiently enjoy weak scaling even with a single parameter server.

Scaling GRPC Tensorflow on 512 nodes of Cori Supercomputer