English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

We present Junk DNA Hypothesis by adopting a novel task-centric angle for thepre-trained weights of large language models (LLMs). It has been believed thatweights in LLMs contain significant redundancy, leading to the conception thata considerable chunk of the parameters can be removed by pruning withoutcompromising performance. Contrary to this belief, this paper presents acounter-argument: small-magnitude weights of pre-trained model weights encodevital knowledge essential for tackling difficult downstream tasks - manifestedas the monotonic relationship between the performance drop of downstream tasksacross the difficulty spectrum, as we prune more pre-trained weights bymagnitude. Moreover, we reveal that these seemingly inconsequential weights canresult in irreparable loss of knowledge and performance degradation indifficult tasks, even when downstream continual training is allowed.Interestingly, our evaluations show that the other popular compression, namelyquantization, fails to exhibit similar monotonic effect and does not asconvincingly disentangle this task-difficulty information. To study formally,we introduce several quantifiable metrics to gauge the downstream taskdifficulty: (1) within the same task category, and (2) across different taskcategories. Our extensive experiments substantiate the Junk DNA Hypothesisacross a diverse range of model sizes, tasks, datasets, and even pruningmethods. Codes are available at:https://github.com/VITA-Group/Junk_DNA_Hypothesis.git.

Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs