English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider distributed convex optimization problems in the regime when thecommunication between the server and the workers is expensive in both uplinkand downlink directions. We develop a new and provably accelerated method,which we call 2Direction, based on fast bidirectional compressed communicationand a new bespoke error-feedback mechanism which may be of independentinterest. Indeed, we find that the EF and EF21-P mechanisms (Seide et al.,2014; Gruntkowska et al., 2023) that have considerable success in the design ofefficient non-accelerated methods are not appropriate for accelerated methods.In particular, we prove that 2Direction improves the previous state-of-the-artcommunication complexity $\widetilde{\Theta}\left(K \times\left(\frac{L}{\alpha \mu} + \frac{L_{\max} \omega}{n \mu} +\omega\right)\right)$ (Gruntkowska et al., 2023) to $\widetilde{\Theta}(K\times (\sqrt{\frac{L (\omega + 1)}{\alpha \mu}} + \sqrt{\frac{L_{\max}\omega^2}{n \mu}} + \frac{1}{\alpha} + \omega))$ in the $\mu$-strongly-convexsetting, where $L$ and $L_{\max}$ are smoothness constants, $n$ is # ofworkers, $\omega$ and $\alpha$ are compression errors of the Rand$K$ and Top$K$sparsifiers (as examples), $K$ is # of coordinates/bits that the server andworkers send to each other. Moreover, our method is the first that improvesupon the communication complexity of the vanilla accelerated gradient descent(AGD) method (Nesterov, 2018). We obtain similar improvements in the generalconvex regime as well. Finally, our theoretical findings are corroborated byexperimental evidence.

2Direction: Theoretically Faster Distributed Training with Bidirectional Communication Compression