English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We consider distributed optimization under communication constraints fortraining deep learning models. We propose a new algorithm, whose parameterupdates rely on two forces: a regular gradient step, and a corrective directiondictated by the currently best-performing worker (leader). Our method differsfrom the parameter-averaging scheme EASGD in a number of ways: (i) ourobjective formulation does not change the location of stationary pointscompared to the original optimization problem; (ii) we avoid convergencedecelerations caused by pulling local workers descending to different localminima to each other (i.e. to the average of their parameters); (iii) ourupdate by design breaks the curse of symmetry (the phenomenon of being trappedin poorly generalizing sub-optimal solutions in symmetric non-convexlandscapes); and (iv) our approach is more communication efficient since itbroadcasts only parameters of the leader rather than all workers. We providetheoretical analysis of the batch version of the proposed algorithm, which wecall Leader Gradient Descent (LGD), and its stochastic variant (LSGD). Finally,we implement an asynchronous version of our algorithm and extend it to themulti-leader setting, where we form groups of workers, each represented by itsown local leader (the best performer in a group), and update each worker with acorrective direction comprised of two attractive forces: one to the local, andone to the global leader (the best performer among all workers). Themulti-leader setting is well-aligned with current hardware architecture, wherelocal workers forming a group lie within a single computational node anddifferent groups correspond to different nodes. For training convolutionalneural networks, we empirically demonstrate that our approach comparesfavorably to state-of-the-art baselines. This work is a gentle extension of[2].

Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: Extension