English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Sparse Mixture-of-Experts models (MoEs) have recently gained popularity dueto their ability to decouple model size from inference efficiency by onlyactivating a small subset of the model parameters for any given input token. Assuch, sparse MoEs have enabled unprecedented scalability, resulting intremendous successes across domains such as natural language processing andcomputer vision. In this work, we instead explore the use of sparse MoEs toscale-down Vision Transformers (ViTs) to make them more attractive forresource-constrained vision applications. To this end, we propose a simplifiedand mobile-friendly MoE design where entire images rather than individualpatches are routed to the experts. We also propose a stable MoE trainingprocedure that uses super-class information to guide the router. We empiricallyshow that our sparse Mobile Vision MoEs (V-MoEs) can achieve a better trade-offbetween performance and efficiency than the corresponding dense ViTs. Forexample, for the ViT-Tiny model, our Mobile V-MoE outperforms its densecounterpart by 3.39% on ImageNet-1k. For an even smaller ViT variant with only54M FLOPs inference cost, our MoE achieves an improvement of 4.66%.

Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts