English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We introduce a method that dramatically reduces fine-tuning VRAM requirementsand rectifies quantization errors in quantized Large Language Models. First, wedevelop an extremely memory-efficient fine-tuning (EMEF) method for quantizedmodels using Low-Rank Adaptation (LoRA), and drawing upon it, we construct anerror-correcting algorithm designed to minimize errors induced by thequantization process. Our method reduces the memory requirements by up to 5.6times, which enables fine-tuning a 7 billion parameter Large Language Model(LLM) on consumer laptops. At the same time, we propose a Low-Rank ErrorCorrection (LREC) method that exploits the added LoRA layers to ameliorate thegap between the quantized model and its float point counterpart. Our errorcorrection framework leads to a fully functional INT2 quantized LLM with thecapacity to generate coherent English text. To the best of our knowledge, thisis the first INT2 Large Language Model that has been able to reach such aperformance. The overhead of our method is merely a 1.05 times increase inmodel size, which translates to an effective precision of INT2.1. Also, ourmethod readily generalizes to other quantization standards, such as INT3, INT4,and INT8, restoring their lost performance, which marks a significant milestonein the field of model quantization. The strategies delineated in this paperhold promising implications for the future development and optimization ofquantized models, marking a pivotal shift in the landscape of low-resourcemachine learning computations.

INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation