English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The rapidly growing size of deep neural network (DNN) models and datasets hasgiven rise to a variety of distribution strategies such as data, tensor-model,pipeline parallelism, and hybrid combinations thereof. Each of these strategiesoffers its own trade-offs and exhibits optimal performance across differentmodels and hardware topologies. Selecting the best set of strategies for agiven setup is challenging because the search space grows combinatorially, anddebugging and testing on clusters is expensive. In this work we propose DistIR,an expressive intermediate representation for distributed DNN computation thatis tailored for efficient analyses, such as simulation. This enablesautomatically identifying the top-performing strategies without having toexecute on physical hardware. Unlike prior work, DistIR can naturally expressmany distribution strategies including pipeline parallelism with arbitraryschedules. Our evaluation on MLP training and GPT-2 inference modelsdemonstrates how DistIR and its simulator enable fast grid searches overcomplex distribution spaces spanning up to 1000+ configurations, reducingoptimization time by an order of magnitude for certain regimes.

DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distribution