English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Mixture of experts (MoE) is a popular technique to improve capacity of LargeLanguage Models (LLMs) with conditionally-activated parallel experts. However,serving MoE models on memory-constrained devices is challenging due to thelarge parameter size. Typical solutions such as memory swapping or expertpruning may lead to significantly higher latency or severe accuracy loss. Inthis paper, we introduce SwapMoE, a framework for efficient serving ofMoE-based large language models with tunable memory budgets. The main idea ofSwapMoE is to keep a small dynamic set of important experts, namely VirtualExperts, in the main memory for inference, while seamlessly maintaining how theVirtual Experts map to the actual experts. Experiments have shown that SwapMoEcan reduce the memory footprint while maintaining reasonable accuracy. Forexample, on text summarization tasks with Switch Transformer, SwapMoE canreduce the memory consumption from 14.2 GiB to 4.7 GiB, together with 50\%latency reduction and a slight Rouge-2 score drop of 0.041.

SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget