English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Recently, Mixture-of-Experts (short as MoE) architecture has achievedremarkable success in increasing the model capacity of large-scale languagemodels. However, MoE requires incorporating significantly more parameters thanthe base model being extended. In this paper, we propose building aparameter-efficient MoE architecture by sharing information among experts. Weadopt the matrix product operator (MPO, a tensor decomposition from quantummany-body physics) to reconstruct the parameter matrix in the expert layer andincrease model capacity for pre-trained language models by sharing parametersof the central tensor (containing the core information) among different expertswhile enabling the specificity through the auxiliary tensors (complementing thecentral tensor) of different experts. To address the unbalanced optimizationissue, we further design the gradient mask strategy for the MPO-based MoEarchitecture. Extensive experiments based on T5 and GPT-2 show improvedperformance and efficiency of the pre-trained language model (27.2x reductionin total parameters for the superior model performance, compared with theSwitch Transformers). Our code is publicly available athttps://github.com/RUCAIBox/MPOE.

Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models