English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Multi-task learning (MTL) has shown considerable practical benefits,particularly when using pre-trained language models (PLMs). While this iscommonly achieved by simultaneously learning $n$ tasks under a jointoptimization procedure, recent methods such as AdapterFusion structure theproblem into two distinct stages: (i) task learning, where knowledge specificto a task is encapsulated within sets of parameters (e.g., adapters), and (ii)transfer, where this already learned knowledge is leveraged for a target task.This separation of concerns provides numerous benefits, such as promotingreusability, and addressing cases involving data privacy and societal concerns;on the flip side, current two-stage MTL methods come with the cost ofintroducing a substantial number of additional parameters. In this work, weaddress this issue by leveraging the usefulness of linearly scaling the outputrepresentations of source adapters for transfer learning. We introduceScaLearn, a simple and highly parameter-efficient two-stage MTL method thatcapitalizes on the knowledge of the source tasks by learning a minimal set ofscaling parameters that enable effective knowledge transfer to a target task.Our experiments on three benchmarks (GLUE, SuperGLUE, and HumSet) show that ourScaLearn, in addition to facilitating the benefits of two-stage MTL,consistently outperforms strong baselines with only a small number of transferparameters - roughly 0.35% of those of AdapterFusion. Remarkably, we observethat ScaLearn maintains its strong abilities even when further reducingparameters through uniform scaling and layer-sharing, achieving similarlycompetitive results with only $8$ transfer parameters for each target task. Ourproposed approach thus demonstrates the power of simple scaling as a promisefor more efficient task transfer.

ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale