English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Mixture-of-Experts (MoE) models have gained popularity in achievingstate-of-the-art performance in a wide range of tasks in computer vision andnatural language processing. They effectively expand the model capacity whileincurring a minimal increase in computation cost during training. However,deploying such models for inference is difficult due to their large size andcomplex communication pattern. In this work, we provide a characterization oftwo MoE workloads, namely Language Modeling (LM) and Machine Translation (MT)and identify their sources of inefficiencies at deployment. We propose threeoptimization techniques to mitigate sources of inefficiencies, namely (1)Dynamic gating, (2) Expert Buffering, and (3) Expert load balancing. We showthat dynamic gating improves maximum throughput by 6.21-11.23$\times$ for LM,5.75-10.98$\times$ for MT Encoder and 2.58-5.71$\times$ for MT Decoder. It alsoreduces memory usage by up to 1.36$\times$ for LM and up to 1.1$\times$ for MT.We further propose Expert Buffering, a new caching mechanism that only keepshot, active experts in GPU memory while buffering the rest in CPU memory. Thisreduces static memory allocation by up to 1.47$\times$. We finally propose aload balancing methodology that provides additional scalability to theworkload.

Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference