English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

With the rapid growth in the scale of pre-trained foundation models,parameter-efficient fine-tuning techniques have gained significant attention,among which Adapter Tuning is the most widely used. Despite achievingefficiency, it still underperforms full fine-tuning, and the performanceimproves at the cost of an increase in parameters. Recent efforts have eitherfocused on training multiple adapter experts to increase model capacity or onpruning adapters to achieve parameter efficiency. However, both approachesintroduce more parameters compared to the original adapter, hence are notcomputationally efficient. Motivated by this, we propose Mixture of SparseAdapters, or MoSA, as a novel Adapter Tuning method to fully unleash thepotential of each parameter in the adapter. We first split the standard adapterinto multiple non-overlapping modules, then stochastically activate them forsparse training, and finally merge them to form a complete adapter aftertuning. In this way, MoSA can achieve significantly better performance thanstandard adapters without any additional computational or storage overhead.Furthermore, we propose a hierarchical sparse strategy to better leveragelimited training data. Extensive experiments on a series of 27 visual tasksdemonstrate that MoSA consistently outperforms other Adapter Tuning methods aswell as other baselines by a large margin. Furthermore, MoSA brings consistentimprovements across various model scales, architectures, and different PEFTmethods. Code will be released.

MoSA: Mixture of Sparse Adapters for Visual Efficient Tuning