English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Driven by the tremendous effort in researching novel deep learning (DL)algorithms, the training cost of developing new models increases staggeringlyin recent years. We analyze GPU cluster usage statistics from a top researchinstitute for more insights into the hardware efficiency achieved by typical DLtraining jobs. Our study reveals that single-accelerator training jobs candominate the cluster-wide resource consumption when launched repetitively(e.g., for hyper-parameter tuning) while severely under-utilizing the hardware.Fortunately, we observe that such workloads have the following uniquecharacteristics: (i) the models among jobs often have the same types ofoperators with the same shapes, and (ii) the inter-model horizontal fusion ofsuch operators is mathematically equivalent to other already well-optimizedoperators. Thus, to help DL researchers and practitioners effectively improvethe hardware utilization of their novel DL training workloads, we proposeHorizontally Fused Training Array (HFTA). HFTA is a new DL framework extensionlibrary that horizontally fuses the models from different repetitive jobsdeeply down to operators and then trains them simultaneously on a sharedaccelerator. To show the generality of our solution, we apply HFTA to six DLmodels training on state-of-the-art accelerators (GPUs and TPUs). Our resultsindicate that HFTA is highly effective in improving hardware utilization andachieves up to $15.1 \times$ higher training throughput vs. the standardpractice of running each job on a separate accelerator.

Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models