English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Why can pre-trained language models (PLMs) learn universal representationsand effectively adapt to broad NLP tasks differing a lot superficially? In thiswork, we empirically find evidence indicating that the adaptations of PLMs tovarious few-shot tasks can be reparameterized as optimizing only a few freeparameters in a unified low-dimensional intrinsic task subspace, which may helpus understand why PLMs could easily adapt to various NLP tasks with small-scaledata. To find such a subspace and examine its universality, we propose ananalysis pipeline called intrinsic prompt tuning (IPT). Specifically, we resortto the recent success of prompt tuning and decompose the soft prompts ofmultiple NLP tasks into the same low-dimensional nonlinear subspace, then welearn to adapt the PLM to unseen data or tasks by only tuning parameters inthis subspace. In the experiments, we study diverse few-shot NLP tasks andsurprisingly find that in a 250-dimensional subspace found with 100 tasks, byonly tuning 250 free parameters, we can recover 97% and 83% of the full prompttuning performance for 100 seen tasks (using different training data) and 20unseen tasks, respectively, showing great generalization ability of the foundintrinsic task subspace. Besides being an analysis tool, IPT could further helpus improve the prompt tuning stability.

Exploring Universal Intrinsic Task Subspace via Prompt Tuning