English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

As emerging deep neural network (DNN) models continue to grow in size, usinglarge GPU clusters to train DNNs is becoming an essential requirement toachieving acceptable training times. In this paper, we consider the case wherefuture increases in cluster size will cause the global batch size that can beused to train models to reach a fundamental limit: beyond a certain point,larger global batch sizes cause sample efficiency to degrade, increasingoverall time to accuracy. As a result, to achieve further improvements intraining performance, we must instead consider "strong scaling" strategies thathold the global batch size constant and allocate smaller batches to each GPU.Unfortunately, this makes it significantly more difficult to use clusterresources efficiently. We present DeepPool, a system that addresses thisefficiency challenge through two key ideas. First, burst parallelism allocateslarge numbers of GPUs to foreground jobs in bursts to exploit the unevenness inparallelism across layers. Second, GPU multiplexing prioritizes throughput forforeground training jobs, while packing in background training jobs to reclaimunderutilized GPU resources, thereby improving cluster-wide utilization.Together, these two ideas enable DeepPool to deliver a 1.2 - 2.3x improvementin total cluster throughput over standard data parallelism with a single taskwhen the cluster scale is large.

Efficient Strong Scaling Through Burst Parallel Training