English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Gradient compression is a widely-established remedy to tackle thecommunication bottleneck in distributed training of large deep neural networks(DNNs). Under the error-feedback framework, Top-$k$ sparsification, sometimeswith $k$ as little as $0.1\%$ of the gradient size, enables training to thesame model quality as the uncompressed case for a similar iteration count. Fromthe optimization perspective, we find that Top-$k$ is the communication-optimalsparsifier given a per-iteration $k$ element budget. We argue that to furtherthe benefits of gradient sparsification, especially for DNNs, a differentperspective is necessary -- one that moves from per-iteration optimality toconsider optimality for the entire training.  We identify that the total error -- the sum of the compression errors for alliterations -- encapsulates sparsification throughout training. Then, we proposea communication complexity model that minimizes the total error under acommunication budget for the entire training. We find that the hard-thresholdsparsifier, a variant of the Top-$k$ sparsifier with $k$ determined by aconstant hard-threshold, is the optimal sparsifier for this model. Motivated bythis, we provide convex and non-convex convergence analyses for thehard-threshold sparsifier with error-feedback. Unlike with Top-$k$ sparsifier,we show that hard-threshold has the same asymptotic convergence and linearspeedup property as SGD in the convex case and has no impact on thedata-heterogeneity in the non-convex case. Our diverse experiments on variousDNNs and a logistic regression model demonstrated that the hard-thresholdsparsifier is more communication-efficient than Top-$k$.

Rethinking gradient sparsification as total error minimization