English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Large models are prevalent in modern machine learning scenarios, includingdeep learning, recommender systems, etc., which can have millions or evenbillions of parameters. Parallel algorithms have become an essential solutiontechnique to many large-scale machine learning jobs. In this paper, we proposea model parallel proximal stochastic gradient algorithm, AsyB-ProxSGD, to dealwith large models using model parallel blockwise updates while in the meantimehandling a large amount of training data using proximal stochastic gradientdescent (ProxSGD). In our algorithm, worker nodes communicate with theparameter servers asynchronously, and each worker performs proximal stochasticgradient for only one block of model parameters during each iteration. Ourproposed algorithm generalizes ProxSGD to the asynchronous and model parallelsetting. We prove that AsyB-ProxSGD achieves a convergence rate of$O(1/\sqrt{K})$ to stationary points for nonconvex problems under\emph{constant} minibatch sizes, where $K$ is the total number of blockupdates. This rate matches the best-known rates of convergence for a wide rangeof gradient-like algorithms. Furthermore, we show that when the number ofworkers is bounded by $O(K^{1/4})$, we can expect AsyB-ProxSGD to achievelinear speedup as the number of workers increases. We implement the proposedalgorithm on MXNet and demonstrate its convergence behavior and near-linearspeedup on a real-world dataset involving both a large model size and largeamounts of data.

A Model Parallel Proximal Stochastic Gradient Algorithm for Partially Asynchronous Systems