English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Multi-task learning (MTL) encapsulates multiple learned tasks in a singlemodel and often lets those tasks learn better jointly. However, when deployingMTL onto those real-world systems that are often resource-constrained orlatency-sensitive, two prominent challenges arise: (i) during training,simultaneously optimizing all tasks is often difficult due to gradientconflicts across tasks; (ii) at inference, current MTL regimes have to activatenearly the entire model even to just execute a single task. Yet most realsystems demand only one or two tasks at each moment, and switch between tasksas needed: therefore such all tasks activated inference is also highlyinefficient and non-scalable. In this paper, we present a model-acceleratorco-design framework to enable efficient on-device MTL. Our framework, dubbedM$^3$ViT, customizes mixture-of-experts (MoE) layers into a vision transformer(ViT) backbone for MTL, and sparsely activates task-specific experts duringtraining. Then at inference with any task of interest, the same design allowsfor activating only the task-corresponding sparse expert pathway, instead ofthe full model. Our new model design is further enhanced by hardware-levelinnovations, in particular, a novel computation reordering scheme tailored formemory-constrained MTL that achieves zero-overhead switching between tasks andcan scale to any number of experts. When executing single-task inference,M$^{3}$ViT achieves higher accuracies than encoder-focused MTL methods, whilesignificantly reducing 88% inference FLOPs. When implemented on a hardwareplatform of one Xilinx ZCU104 FPGA, our co-design framework reduces the memoryrequirement by 2.4 times, while achieving energy efficiency up to 9.23 timeshigher than a comparable FPGA baseline. Code is available at:https://github.com/VITA-Group/M3ViT.

M$^3$ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design