Distributed SignSGD With Improved Accuracy and Network-Fault Tolerance | Zendy

Trieu Le Phong | Zendy; Tran Thi Phuong | Zendy

AI Assistant Blog Pricing

Home ZAIA Blog

Open Access

Distributed SignSGD With Improved Accuracy and Network-Fault Tolerance

Author(s) -

Trieu Le Phong,

Tran Thi Phuong

Publication year - 2020

Publication title -

ieee access

Language(s) - English

Resource type - Journals

SCImago Journal Rank - 0.587

H-Index - 127

ISSN - 2169-3536

DOI - 10.1109/access.2020.3032637

Subject(s) - aerospace , bioengineering , communication, networking and broadcast technologies , components, circuits, devices and systems , computing and processing , engineered materials, dielectrics and plasmas , engineering profession , fields, waves and electromagnetics , general topics for engineers , geoscience , nuclear engineering , photonics and electrooptics , power, energy and industry applications , robotics and control systems , signal processing and analysis , transportation

This paper proposes DropSignSGD, a communication-efficient and network-fault tolerant algorithm for training deep neural networks in a distributed and synchronous fashion. In DropSignSGD, all numerical elements communicated between machines are either 1 or -1, represented by only one bit. More importantly, DropSignSGD does not decline the benchmark accuracy on the ImageNet dataset when compared with the traditional distributed stochastic gradient descent algorithm, owing to a little trick in memorizing unused gradients. Experimental results are supported by a mathematical proof showing that DropSignSGD converges under standard assumptions.

The content you want is available to Zendy users.

Already have an account? Click here to sign in.

Having issues? You can contact us here

Accelerating Research