English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Efficient distributed training is a principal driver of recent advances indeep learning. However, communication often proves costly and becomes theprimary bottleneck in these systems. As a result, there is a demand for thedesign of efficient communication mechanisms that can empirically boostthroughput while providing theoretical guarantees. In this work, we introduceGlobal-QSGD, a novel family of quantization operators, engineered to acceleratedistributed training based on global scaling. We demonstrate that Global-QSGDis the first theoretically rigorous Allreduce-compatible compression mechanismthat achieves a provable speed-up by striking a balance between compressionerror and communication savings. Importantly, Global-QSGD does not rely oncostly error feedback due to its inherent unbiasedness and offers up to$O(\sqrt{n})$ additional compression ratio compared to the popular QSGDquantization ($n$ represents the number of workers). To obtain theoreticalguarantees, we generalize the notion of standard unbiased compression operatorsto incorporate Global-QSGD. We show that this wider class permits standardanalysis for unbiased compressors and thus ensures convergence for popularoptimization algorithms (e.g., distributed SGD) under typical settings. For theempirical component of our work, we carry out a performance modeling analysisto determine if Global-QSGD can enhance training throughput under specifichardware configurations. We also conduct extensive empirical evaluations onvarious tasks, testing our theory on both NVLink and PCIe connections as wellas a large-scale cloud system.

Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees