English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The Parameter-Efficient Fine-Tuning (PEFT) method, which adjusts orintroduces fewer trainable parameters to calibrate pre-trained models ondownstream tasks, has become a recent research interest. However, existing PEFTmethods within the traditional fine-tiuning framework have two mainshortcomings: 1) They overlook the explicit association between trainableparameters and downstream task knowledge. 2) They neglect the interactionbetween the intrinsic task-agnostic knowledge of pre-trained models and thetask-specific knowledge in downstream tasks. To address this gap, we propose anovel fine-tuning framework, named GIST, in a plug-and-play manner.Specifically, our framework first introduces a trainable token, called the Gisttoken, when applying PEFT methods on downstream tasks. This token serves as anaggregator of the task-specific knowledge learned by the PEFT methods and formsan explicit association with downstream knowledge. Furthermore, to facilitateexplicit interaction between task-agnostic and task-specific knowledge, weintroduce the concept of Knowledge Interaction via a BidirectionalKullback-Leibler Divergence objective. As a result, PEFT methods within ourframework can make the pre-trained model understand downstream tasks morecomprehensively by leveraging the knowledge interaction. Extensive experimentsdemonstrate the universality and scalability of our framework. Notably, on theVTAB-1K benchmark, we employ the Adapter (a prevalent PEFT method) within ourGIST framework and achieve a performance boost of 2.25%, with an increase ofonly 0.8K parameters. The Code will be released.

GIST: Improving Parameter Efficient Fine Tuning via Knowledge Interaction