English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Supervised fine-tuning (SFT) is a crucial step for large language models(LLMs), enabling them to align with human instructions and enhance theircapabilities in downstream tasks. Increasing instruction data substantially isa direct solution to align the model with a broader range of downstream tasksor notably improve its performance on a specific task. However, we find thatlarge-scale increases in instruction data can damage the world knowledgepreviously stored in LLMs. To address this challenge, we propose LoRAMoE, anovelty framework that introduces several low-rank adapters (LoRA) andintegrates them by using a router network, like a plugin version of Mixture ofExperts (MoE). It freezes the backbone model and forces a portion of LoRAs tofocus on leveraging world knowledge to solve downstream tasks, to alleviateworld knowledge-edge forgetting. Experimental results show that, as theinstruction data increases, LoRAMoE can significantly improve the ability toprocess downstream tasks, while maintaining the world knowledge stored in theLLM.

LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin