English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Distributed deep learning is an effective way to reduce the training time ofdeep learning for large datasets as well as complex models. However, thelimited scalability caused by network overheads makes it difficult tosynchronize the parameters of all workers. To resolve this problem,gossip-based methods that demonstrates stable scalability regardless of thenumber of workers have been proposed. However, to use gossip-based methods ingeneral cases, the validation accuracy for a large mini-batch needs to beverified. To verify this, we first empirically study the characteristics ofgossip methods in a large mini-batch problem and observe that the gossipmethods preserve higher validation accuracy than AllReduce-SGD(StochasticGradient Descent) when the number of batch sizes is increased and the number ofworkers is fixed. However, the delayed parameter propagation of thegossip-based models decreases validation accuracy in large node scales. To copewith this problem, we propose Crossover-SGD that alleviates the delaypropagation of weight parameters via segment-wise communication and loadbalancing random network topology. We also adapt hierarchical communication tolimit the number of workers in gossip-based communication methods. To validatethe effectiveness of our proposed method, we conduct empirical experiments andobserve that our Crossover-SGD shows higher node scalability thanSGP(Stochastic Gradient Push).

Crossover-SGD: A gossip-based communication in distributed deep learning for alleviating large mini-batch problem and enhancing scalability