English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The traditional notion of "Junk DNA" has long been linked to non-codingsegments within the human genome, constituting roughly 98% of its composition.However, recent research has unveiled the critical roles some of theseseemingly non-functional DNA sequences play in cellular processes.Intriguingly, the weights within deep neural networks exhibit a remarkablesimilarity to the redundancy observed in human genes. It was believed thatweights in gigantic models contained excessive redundancy, and could be removedwithout compromising performance. This paper challenges this conventionalwisdom by presenting a compelling counter-argument. We employ sparsity as atool to isolate and quantify the nuanced significance of low-magnitude weightsin pre-trained large language models (LLMs). Our study demonstrates a strongcorrelation between these weight magnitudes and the knowledge they encapsulate,from a downstream task-centric angle. we raise the "Junk DNA Hypothesis" backedby our in-depth investigation: while small-magnitude weights may appear"useless" for simple tasks and suitable for pruning, they actually encodecrucial knowledge necessary for solving more difficult downstream tasks.Removing these seemingly insignificant weights can lead to irreversibleknowledge forgetting and performance damage in difficult tasks. These findingsoffer fresh insights into how LLMs encode knowledge in a task-sensitive manner,pave future research direction in model pruning, and open avenues fortask-aware conditional computation during inference.

Junk DNA Hypothesis: A Task-Centric Angle of LLM Pre-trained Weights through Sparsity