English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we present Delta-LoRA, which is a novel parameter-efficientapproach to fine-tune large language models (LLMs). In contrast to LoRA andother low-rank adaptation methods such as AdaLoRA, Delta-LoRA not only updatesthe low-rank matrices $\bA$ and $\bB$, but also propagate the learning to thepre-trained weights $\bW$ via updates utilizing the delta of the product of twolow-rank matrices ($\bA^{(t+1)}\bB^{(t+1)} - \bA^{(t)}\bB^{(t)}$). Such astrategy effectively addresses the limitation that the incremental update oflow-rank matrices is inadequate for learning representations capable fordownstream tasks. Moreover, as the update of $\bW$ does not need to compute thegradients of $\bW$ and store their momentums, Delta-LoRA shares comparablememory requirements and computational costs with LoRA. Extensive experimentsshow that Delta-LoRA significantly outperforms existing low-rank adaptationmethods. We further support these results with comprehensive analyses thatunderscore the effectiveness of Delta-LoRA.

Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices