English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

With the increasing size of pre-trained language models (PLMs), fine-tuningall the parameters in the model is not efficient, especially when there are alarge number of downstream tasks, which incur significant training and storagecosts. Many parameter-efficient fine-tuning (PEFT) approaches have beenproposed, among which, Low-Rank Adaptation (LoRA) is a representative approachthat injects trainable rank decomposition matrices into every target module.Yet LoRA ignores the importance of parameters in different modules. To addressthis problem, many works have been proposed to prune the parameters of LoRA.However, under limited training conditions, the upper bound of the rank of thepruned parameter matrix is still affected by the preset values. We, therefore,propose IncreLoRA, an incremental parameter allocation method that adaptivelyadds trainable parameters during training based on the importance scores ofeach module. This approach is different from the pruning method as it is notlimited by the initial number of training parameters, and each parameter matrixhas a higher rank upper bound for the same training overhead. We conductextensive experiments on GLUE to demonstrate the effectiveness of IncreLoRA.The results show that our method owns higher parameter efficiency, especiallywhen under the low-resource settings where our method significantly outperformsthe baselines. Our code is publicly available.

IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning