English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Scaling model parameters improves model quality at the price of highcomputation overhead. Sparsely activated models, usually in the form of Mixtureof Experts (MoE) architecture, have sub-linear scaling of computation cost withmodel size, thus providing opportunities to train and serve a larger model atlower cost than their dense counterparts. However, distributed MoE training andinference is inefficient, mainly due to the interleaved all-to-allcommunication during model computation. This paper makes two maincontributions. First, we systematically analyze all-to-all overhead indistributed MoE and present the main causes for it to be the bottleneck intraining and inference, respectively. Second, we design and build Lina toaddress the all-to-all bottleneck head-on. Lina opportunistically prioritizesall-to-all over the concurrent allreduce whenever feasible using tensorpartitioning, so all-to-all and training step time is improved. Lina furtherexploits the inherent pattern of expert selection to dynamically scheduleresources during inference, so that the transfer size and bandwidth ofall-to-all across devices are balanced amid the highly skewed expert popularityin practice. Experiments on an A100 GPU testbed show that Lina reduces thetraining step time by up to 1.73x and reduces the 95%ile inference time by anaverage of 1.63x over the state-of-the-art systems.

Accelerating Distributed MoE Training and Inference with Lina