English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Deep Neural Networks have gained significant attraction due to their wideapplicability in different domains. DNN sizes and training samples areconstantly growing, making training of such workloads more challenging.Distributed training is a solution to reduce the training time.High-performance distributed training platforms should leveragemulti-dimensional hierarchical networks, which interconnect acceleratorsthrough different levels of the network, to dramatically reduce expensive NICsrequired for the scale-out network. However, it comes at the expense ofcommunication overhead between distributed accelerators to exchange gradientsor input/output activation. In order to allow for further scaling of theworkloads, communication overhead needs to be minimized. In this paper, wemotivate the fact that in training platforms, adding more intermediate networkdimensions is beneficial for efficiently mitigating the excessive use ofexpensive NIC resources. Further, we address different challenges of the DNNtraining on hierarchical networks. We discuss when designing the interconnect,how to distribute network bandwidth resources across different dimensions inorder to (i) maximize BW utilization of all dimensions, and (ii) minimizing theoverall training time for the target workload. We then implement a frameworkthat, for a given workload, determines the best network configuration thatmaximizes performance, or performance-per-cost.

Exploring Multi-dimensional Hierarchical Network Topologies for Efficient Distributed Training of Trillion Parameter DL Models