English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Pipeline parallelism enables efficient training of Large Language Models(LLMs) on large-scale distributed accelerator clusters. Yet, pipeline bubblesduring startup and tear-down reduce the utilization of accelerators. Althoughefficient pipeline schemes with micro-batching and bidirectional pipelines havebeen proposed to maximize utilization, a significant number of bubbles cannotbe filled using synchronous forward and backward passes. To address thisproblem, we suggest that extra work be assigned to the bubbles to gainauxiliary benefits in LLM training. As an example in this direction, we proposePipeFisher, which assigns the work of K-FAC, a second-order optimization methodbased on the Fisher information matrix, to the bubbles to accelerateconvergence. In Phase 1 pretraining of BERT-Base and -Large models, PipeFisherreduces the (simulated) training time to 50-75% compared to training with afirst-order optimizer by greatly improving the accelerator utilization andbenefiting from the improved convergence by K-FAC.

PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices