English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Gradient-based methods for the distributed training of residual networks(ResNets) typically require a forward pass of the input data, followed byback-propagating the error gradient to update model parameters, which becomestime-consuming as the network goes deeper. To break the algorithmic locking andexploit synchronous module parallelism in both the forward and backward modes,auxiliary-variable methods have attracted much interest lately but suffer fromsignificant communication overhead and lack of data augmentation. In this work,a novel joint learning framework for training realistic ResNets across multiplecompute devices is established by trading off the storage and recomputation ofexternal auxiliary variables. More specifically, the input data of eachindependent processor is generated from its low-capacity auxiliary network(AuxNet), which permits the use of data augmentation and realizes forwardunlocking. The backward passes are then executed in parallel, each with a localloss function that originates from the penalty or augmented Lagrangian (AL)methods. Finally, the proposed AuxNet is employed to reproduce the updatedauxiliary variables through an end-to-end training process. We demonstrate theeffectiveness of our methods on ResNets and WideResNets across CIFAR-10,CIFAR-100, and ImageNet datasets, achieving speedup over the traditionallayer-serial training method while maintaining comparable testing accuracy.

Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networks