English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Distributed model training suffers from communication bottlenecks due tofrequent model updates transmitted across compute nodes. To alleviate thesebottlenecks, practitioners use gradient compression techniques likesparsification, quantization, or low-rank updates. The techniques usuallyrequire choosing a static compression ratio, often requiring users to balancethe trade-off between model accuracy and per-iteration speedup. In this work,we show that such performance degradation due to choosing a high compressionratio is not fundamental. An adaptive compression strategy can reducecommunication while maintaining final test accuracy. Inspired by recentfindings on critical learning regimes, in which small gradient errors can haveirrecoverable impact on model performance, we propose Accordion a simple yeteffective adaptive compression algorithm. While Accordion maintains a highenough compression rate on average, it avoids over-compressing gradientswhenever in critical learning regimes, detected by a simple gradient-norm basedcriterion. Our extensive experimental study over a number of machine learningtasks in distributed environments indicates that Accordion, maintains similarmodel accuracy to uncompressed training, yet achieves up to 5.5x bettercompression and up to 4.1x end-to-end speedup over static approaches. We showthat Accordion also works for adjusting the batch size, another popularstrategy for alleviating communication bottlenecks.

Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification