English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

With the rapid growth in the scale of pre-trained foundation models,parameter-efficient fine-tuning techniques have gained significant attention,among which Adapter Tuning is the most widely used. Despite achievingefficiency, Adapter Tuning still underperforms full fine-tuning, and theperformance improves at the cost of an increase in parameters. Recent effortsaddress this issue by pruning the original adapters, but it also introducestraining instability and suboptimal performance on certain datasets. Motivatedby this, we propose Mixture of Sparse Adapters, or MoSA, as a novel AdapterTuning method to fully unleash the potential of each parameter in the adapter.We first split the standard adapter into multiple non-overlapping modules, thenstochastically activate modules for sparse training, and finally merge them toform a complete adapter after tuning. In this way, MoSA can achievesignificantly better performance than standard adapters without any additionalcomputational or storage overhead. Furthermore, we propose a hierarchicalsparse strategy to better leverage limited training data. Extensive experimentson a series of 27 visual tasks demonstrate that MoSA consistently outperformsother Adapter Tuning methods as well as other baselines by a significantmargin. Furthermore, in two challenging scenarios with low-resource andmulti-task settings, MoSA achieves satisfactory results, further demonstratingthe effectiveness of our design. Our code will be released.

Split & Merge: Unlocking the Potential of Visual Adapters via Sparse Training