English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Supervised fine-tuning (SFT) is a crucial step for large language models(LLMs), enabling them to align with human instructions and enhance theircapabilities in downstream tasks. When the models are required to align with abroader range of downstream tasks, or there is a desire to notably improve theperformance on a specific task, a substantial increase in fine-tuning dataoften emerges as the solution. However, we find that large-scale increases ininstruction data can disrupt the world knowledge previously stored in the LLMs,i.e., world knowledge forgetting. In this paper, we introduce LoRAMoE toaddress the above challenge. The LoRAMoE is a plugin version of Mixture ofExperts (MoE). The plugin form ensures the integrity of world knowledge byfreezing the backbone model during the training phase. We then propose the useof localized balancing constraints to coordinate parts of experts for taskutilization, meanwhile enabling other experts to fully leverage the worldknowledge stored in the models. Experimental results demonstrate that LoRAMoEcan reasonably coordinate experts based on data type during inference, and evendramatically increasing instruction data does not result in knowledgeforgetting. Moreover, LoRAMoE provides additional benefits for the performanceof downstream tasks, indicating the potential of our approach for multi-tasklearning.

LoRAMoE: Revolutionizing Mixture of Experts for Maintaining World Knowledge in Language Model Alignment