English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

The recent surge in Large Language Models (LLMs) has garnered significantattention across numerous fields. Fine-tuning is often required to fit generalLLMs for a specific domain, like the web-based healthcare system. However, twoproblems arise during fine-tuning LLMs for medical applications. One is thetask variety problem, which involves distinct tasks in real-world medicalscenarios. The variety often leads to sub-optimal fine-tuning for dataimbalance and seesaw problems. Besides, the large amount of parameters in LLMsleads to huge time and computation consumption by fine-tuning. To address thesetwo problems, we propose a novel parameter efficient fine-tuning framework formulti-task medical applications, dubbed as MOELoRA. The designed framework aimsto absorb both the benefits of mixture-of-expert (MOE) for multi-task learningand low-rank adaptation (LoRA) for parameter efficient fine-tuning. Forunifying MOE and LoRA, we devise multiple experts as the trainable parameters,where each expert consists of a pair of low-rank matrices to retain the smallsize of trainable parameters. Then, a task-motivated gate function for allMOELoRA layers is proposed, which can control the contributions of each expertand produce distinct parameters for various tasks. We conduct experiments on amulti-task medical dataset, indicating MOELoRA outperforms the existingparameter efficient fine-tuning methods. The code is available online.

When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications