English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Asynchronous pipeline model parallelism with a "1F1B" (one forward, onebackward) schedule generates little bubble overhead and always provides quite ahigh throughput. However, the "1F1B" schedule inevitably leads to weightinconsistency and weight staleness issues due to the cross-training ofdifferent mini-batches across GPUs. To simultaneously address these twoproblems, in this paper, we propose an optimizer-dependent weight predictionstrategy (a.k.a PipeOptim) for asynchronous pipeline training. The key insightof our proposal is that we employ a weight prediction strategy in the forwardpass to ensure that each mini-batch uses consistent and staleness-free weightsto compute the forward pass. To be concrete, we first construct the weightprediction scheme based on the update rule of the used optimizer when trainingthe deep neural network models. Then throughout the "1F1B" pipelined training,each mini-batch is mandated to execute weight prediction ahead of the forwardpass, subsequently employing the predicted weights to perform the forward pass.As a result, PipeOptim 1) inherits the advantage of the "1F1B" schedule andgenerates pretty high throughput, and 2) can ensure effective parameterlearning regardless of the type of the used optimizer. To verify theeffectiveness of our proposal, we conducted extensive experimental evaluationsusing eight different deep-learning models spanning three machine-learningtasks including image classification, sentiment analysis, and machinetranslation. The experiment results demonstrate that PipeOptim outperforms thepopular pipelined approaches including GPipe, PipeDream, PipeDream-2BW, andSpecTrain. The code of PipeOptim can be accessible athttps://github.com/guanleics/PipeOptim.

PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Prediction