English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

DNN models are becoming increasingly larger to achieve unprecedentedaccuracy, and the accompanying increased computation and memory requirementsnecessitate the employment of massive clusters and elaborate parallelizationstrategies to accelerate DNN training. In order to better optimize theperformance and analyze the cost, it is indispensable to model the trainingthroughput of distributed DNN training. However, complex parallelizationstrategies and the resulting complex runtime behaviors make it challenging toconstruct an accurate performance model. In this paper, we present Proteus, thefirst standalone simulator to model the performance of complex parallelizationstrategies through simulation execution. Proteus first models complexparallelization strategies with a unified representation named Strategy Tree.Then, it compiles the strategy tree into a distributed execution graph andsimulates the complex runtime behaviors, comp-comm overlap and bandwidthsharing, with a Hierarchical Topo-Aware Executor (HTAE). We finally evaluateProteus across a wide variety of DNNs on three hardware configurations.Experimental results show that Proteus achieves $3.0\%$ average predictionerror and preserves order for training throughput of various parallelizationstrategies. Compared to state-of-the-art approaches, Proteus reduces predictionerror by up to $133.8\%$.

Proteus: Simulating the Performance of Distributed DNN Training