English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The low-rank adaptation (LoRA) method can largely reduce the amount oftrainable parameters for fine-tuning large language models (LLMs), however, itstill requires expensive activation memory to update low-rank weights. Reducingthe number of LoRA layers or using activation recomputation could harm thefine-tuning performance or increase the computational overhead. In this work,we present LoRA-FA, a memory-efficient fine-tuning method that reduces theactivation memory without performance degradation and expensive recomputation.LoRA-FA chooses to freeze the projection-down weight of $A$ and update theprojection-up weight of $B$ in each LoRA layer. It ensures the change of modelweight reside in a low-rank space during LLMs fine-tuning, while eliminatingthe requirement to store full-rank input activations. We conduct extensiveexperiments across multiple model types (RoBERTa, T5, LLaMA) and model scales.Our results show that LoRA-FA can always achieve close fine-tuning accuracyacross different tasks compared to full parameter fine-tuning and LoRA.Furthermore, LoRA-FA can reduce the overall memory cost by up to 1.4$\times$compared to LoRA.

LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning