English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we uncover that Language Models (LMs), either encoder- ordecoder-based, can obtain new capabilities by assimilating the parameters ofhomologous models without retraining or GPUs. Typically, new abilities of LMscan be imparted by Supervised Fine-Tuning (SFT), reflected in the disparitybetween fine-tuned and pre-trained parameters (i.e., delta parameters). Weinitially observe that by introducing a novel operation called DARE (Drop AndREscale), most delta parameters can be directly set to zeros without affectingthe capabilities of SFT LMs and larger models can tolerate a higher proportionof discarded parameters. Based on this observation, we further sparsify deltaparameters of multiple SFT homologous models with DARE and subsequently mergethem into a single model by parameter averaging. We conduct experiments oneight datasets from the GLUE benchmark with BERT and RoBERTa. We also mergeWizardLM, WizardMath, and Code Alpaca based on Llama 2. Experimental resultsshow that: (1) The delta parameter value ranges for SFT models are typicallysmall, often within 0.005, and DARE can eliminate 99% of them effortlessly.However, once the models are continuously pre-trained, the value ranges cangrow to around 0.03, making DARE impractical. We have also tried to removefine-tuned instead of delta parameters and find that a 10% reduction can leadto drastically decreased performance (even to 0). This highlights that SFTmerely stimulates the abilities via delta parameters rather than injecting newabilities into LMs; (2) DARE can merge multiple task-specific LMs into one LMwith diverse abilities. For instance, the merger of WizardLM and WizardMathimproves the GSM8K zero-shot accuracy of WizardLM from 2.2 to 66.3, retainingits instruction-following ability while surpassing WizardMath's original 64.2performance. Codes are available at https://github.com/yule-BUAA/MergeLM.

Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch