English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

In this paper, we propose a method of distributed stochastic gradient descent(SGD), with low communication load and computational complexity, and still fastconvergence. To reduce the communication load, at each iteration of thealgorithm, the worker nodes calculate and communicate some scalers, that arethe directional derivatives of the sample functions in some \emph{pre-shareddirections}. However, to maintain accuracy, after every specific number ofiterations, they communicate the vectors of stochastic gradients. To reduce thecomputational complexity in each iteration, the worker nodes approximate thedirectional derivatives with zeroth-order stochastic gradient estimation, byperforming just two function evaluations rather than computing a first-ordergradient vector. The proposed method highly improves the convergence rate ofthe zeroth-order methods, guaranteeing order-wise faster convergence. Moreover,compared to the famous communication-efficient methods of model averaging (thatperform local model updates and periodic communication of the gradients tosynchronize the local models), we prove that for the general class ofnon-convex stochastic problems and with reasonable choice of parameters, theproposed method guarantees the same orders of communication load andconvergence rate, while having order-wise less computational complexity.Experimental results on various learning problems in neural networksapplications demonstrate the effectiveness of the proposed approach compared tovarious state-of-the-art distributed SGD methods.

A Hybrid-Order Distributed SGD Method for Non-Convex Optimization to Balance Communication Overhead, Computational Complexity, and Convergence Rate