English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The Mixture of Experts (MoE) has emerged as a highly successful technique indeep learning, based on the principle of divide-and-conquer to maximize modelcapacity without significant additional computational cost. Even in the era oflarge-scale language models (LLMs), MoE continues to play a crucial role, assome researchers have indicated that GPT-4 adopts the MoE structure to ensurediverse inference results. However, MoE is susceptible to performancedegeneracy, particularly evident in the issues of imbalance and homogeneousrepresentation among experts. While previous studies have extensively addressedthe problem of imbalance, the challenge of homogeneous representation remainsunresolved. In this study, we shed light on the homogeneous representationproblem, wherein experts in the MoE fail to specialize and lack diversity,leading to frustratingly high similarities in their representations (up to 99%in a well-performed MoE model). This problem restricts the expressive power ofthe MoE and, we argue, contradicts its original intention. To tackle thisissue, we propose a straightforward yet highly effective solution: OMoE, anorthogonal expert optimizer. Additionally, we introduce an alternating trainingstrategy that encourages each expert to update in a direction orthogonal to thesubspace spanned by other experts. Our algorithm facilitates MoE training intwo key ways: firstly, it explicitly enhances representation diversity, andsecondly, it implicitly fosters interaction between experts during orthogonalweights computation. Through extensive experiments, we demonstrate that ourproposed optimization algorithm significantly improves the performance offine-tuning the MoE model on the GLUE benchmark, SuperGLUE benchmark,question-answering task, and name entity recognition tasks.

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer