English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Despite their remarkable achievement, gigantic transformers encountersignificant drawbacks, including exorbitant computational and memory footprintsduring training, as well as severe collapse evidenced by a high degree ofparameter redundancy. Sparsely-activated Mixture-of-Experts (SMoEs) have shownpromise to mitigate the issue of training efficiency, yet they are prone to (1)redundant experts due to representational collapse; and (2) poor expertscalability for inference and downstream fine-tuning, primarily due tooverfitting of the learned routing policy to the number of activated expertsduring training. As recent research efforts are predominantly focused onimproving routing policies to encourage expert specializations, this workfocuses on exploring the overlooked scalability bottleneck of SMoEs andleveraging it to effectively scale dense transformers. To this end, we proposea new plug-and-play training framework, SMoE-Dropout, to enable scalingtransformers to better accuracy in their full capacity without collapse.Specifically, SMoE-Dropout consists of a randomly initialized and fixed routernetwork to activate experts and gradually increases the activated expert numberas training progresses over time. Transformers trained by SMoE-Dropoutnaturally exhibit a self-slimmable property subject to resource availability,offering smooth and consistent performance boosts with an increase in activatedexperts during inference or fine-tuning. Our extensive experiments demonstratethe superior performance and substantial computation savings of SMoE-Dropout,compared to dense training baselines with equivalent parameter counts. Inparticular, our trained BERT outperforms its densely trained counterpart withconsistent improvements of {1.03%, 0.78%, 1.09%} on challenging reasoning tasks{ASDiv-A, MAWPS, SVAMP}, respectively.

Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers