English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Mixture of experts (MoE) is a popular technique to improve capacity of largemodels with conditionally-activated parallel neural network modules (experts).Due to its remarkable scaling performance with sparse computation, it is widelyused in modern Large Language Models (LLMs) and Large Vision Models (LVMs).However, serving such large models on edge devices is challenging due to memoryconstraints. Typical solutions like memory swapping or weight pruning may leadto significantly higher latency or severe accuracy loss.  In this paper, we introduce SwapMoE, a framework for efficient continuousMoE-based large models serving with tunable memory budgets. The main idea ofSwapMoE is to keep a small dynamic set of important experts, namely VirtualExperts, in the main memory for inference, while seamlessly maintaining how theVirtual Experts map to the actual experts. We use a profiling-guided planner toallocate the resources for SwapMoE that can fully utilize the memory budgetsand bandwidth, and an importance-aware scheduler to efficiently identify,update, and use the Virtual Experts for accurate inference.  To evaluate SwapMoE, we conduct experiments on multiple edge devices withstate-of-the-art MoE-based Large Language Models and Large Vision Models. Theresults demonstrate remarkable performance of SwapMoE under various memoryconstraints. Specifically, SwapMoE can enable running large MoE models undertight memory budgets with similar latency to pruned compact models, while withsignificantly higher accuracy.

SwapMoE: Efficient Memory-Constrained Serving of Large Sparse MoE Models via Dynamic Expert Pruning and Swapping