English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In deep learning, mixture-of-experts (MoE) activates one or few experts(sub-networks) on a per-sample or per-token basis, resulting in significantcomputation reduction. The recently proposed \underline{p}atch-level routing in\underline{MoE} (pMoE) divides each input into $n$ patches (or tokens) andsends $l$ patches ($l\ll n$) to each expert through prioritized routing. pMoEhas demonstrated great empirical success in reducing training and inferencecosts while maintaining test accuracy. However, the theoretical explanation ofpMoE and the general MoE remains elusive. Focusing on a supervisedclassification task using a mixture of two-layer convolutional neural networks(CNNs), we show for the first time that pMoE provably reduces the requirednumber of training samples to achieve desirable generalization (referred to asthe sample complexity) by a factor in the polynomial order of $n/l$, andoutperforms its single-expert counterpart of the same or even larger capacity.The advantage results from the discriminative routing property, which isjustified in both theory and practice that pMoE routers can filterlabel-irrelevant patches and route similar class-discriminative patches to thesame expert. Our experimental results on MNIST, CIFAR-10, and CelebA supportour theoretical findings on pMoE's generalization and show that pMoE can avoidlearning spurious correlations.

Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks