English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Deep Neural Network (DNN) frameworks use distributed training to enablefaster time to convergence and alleviate memory capacity limitations whentraining large models and/or using high dimension inputs. With the steadyincrease in datasets and model sizes, model/hybrid parallelism is deemed tohave an important role in the future of distributed training of DNNs. Weanalyze the compute, communication, and memory requirements of ConvolutionalNeural Networks (CNNs) to understand the trade-offs between differentparallelism approaches on performance and scalability. We leverage ourmodel-driven analysis to be the basis for an oracle utility which can help indetecting the limitations and bottlenecks of different parallelism approachesat scale. We evaluate the oracle on six parallelization strategies, with fourCNN models and multiple datasets (2D and 3D), on up to 1024 GPUs. The resultsdemonstrate that the oracle has an average accuracy of about 86.74% whencompared to empirical results, and as high as 97.57% for data parallelism.

An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks