English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Deep Neural Network (DNN) models have continuously been growing in size inorder to improve the accuracy and quality of the models. Moreover, for trainingof large DNN models, the use of heterogeneous GPUs is inevitable due to theshort release cycle of new GPU architectures. In this paper, we investigate howto enable training of large DNN models on a heterogeneous GPU cluster thatpossibly includes whimpy GPUs that, as a standalone, could not be used fortraining. We present a DNN training system, HetPipe (Heterogeneous Pipeline),that integrates pipelined model parallelism (PMP) with data parallelism (DP).In HetPipe, a group of multiple GPUs, called a virtual worker, processesminibatches in a pipelined manner, and multiple such virtual workers employdata parallelism for higher performance. We also propose a novel parametersynchronization model, which we refer to as Wave Synchronous Parallel (WSP) toaccommodate both PMP and DP for virtual workers, and provide convergence proofof WSP. Our experimental results on a given heterogeneous setting show thatwith HetPipe, DNN models converge up to 49% faster compared to thestate-of-the-art DP technique.

HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism