English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

With the ever-increasing computational demand of DNN training workloads,distributed training has been widely adopted. A combination of data, model andpipeline parallelism strategy, called hybrid parallelism distributed training,is imported to tackle the problem of deploying large-scale models. However, howto evaluate the hybrid strategy and the utilization of each device remains achallenge since existing works either profile on a real large-scale clusterwith high time and money costs or only analyze a specific type of parallelismwithout considering the hybrid parallelism. In this work, we proposed DistSim,an event-based performance model to accurately analyze each device'scomputation and communication activities with low profiling costs. DistDimbreaks down the model into events according to the given distributed strategy,which can be profiled on two nodes. Then DistSim leverages the hierarchy ofdifferent parallel strategies to generate the computation and communicationevent-flow from layer level to model level and finally the activity timeline ofeach device participating in training. Experiment shows that DistSim can reach\revise{<4\%} errors when predicting distributing training batch time and\revise{<5\%} errors when predicting a single device's activity time in varioushybrid strategy settings. We also provide a use-case of DistSim, automaticallyevaluate and search the best distributed training strategy, and find a hybridstrategy with at most $7.37\times$ throughput improvement.

DistSim: A performance model of large-scale hybrid distributed DNN training