English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The parameter server architecture is prevalently used for distributed deeplearning. Each worker machine in a parameter server system trains the completemodel, which leads to a hefty amount of network data transfer between workersand servers. We empirically observe that the data transfer has a non-negligibleimpact on training time.  To tackle the problem, we design a new distributed training system calledStanza. Stanza exploits the fact that in many models such as convolution neuralnetworks, most data exchange is attributed to the fully connected layers, whilemost computation is carried out in convolutional layers. Thus, we propose layerseparation in distributed training: the majority of the nodes just train theconvolutional layers, and the rest train the fully connected layers only.Gradients and parameters of the fully connected layers no longer need to beexchanged across the cluster, thereby substantially reducing the data transfervolume. We implement Stanza on PyTorch and evaluate its performance on Azureand EC2. Results show that Stanza accelerates training significantly overcurrent parameter server systems: on EC2 instances with Tesla V100 GPU and 10Gbbandwidth for example, Stanza is 1.34x--13.9x faster for common deep learningmodels.

Stanza: Layer Separation for Distributed Training in Deep Learning