English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

We analyze a family of large language models in such a lightweight mannerthat can be done on a single GPU. Specifically, we focus on the OPT family ofmodels ranging from 125m to 66b parameters and rely only on whether an FFNneuron is activated or not. First, we find that the early part of the networkis sparse and represents many discrete features. Here, many neurons (more than70% in some layers of the 66b model) are "dead", i.e. they never activate on alarge collection of diverse data. At the same time, many of the alive neuronsare reserved for discrete features and act as token and n-gram detectors.Interestingly, their corresponding FFN updates not only promote next tokencandidates as could be expected, but also explicitly focus on removing theinformation about triggering them tokens, i.e., current input. To the best ofour knowledge, this is the first example of mechanisms specialized at removing(rather than adding) information from the residual stream. With scale, modelsbecome more sparse in a sense that they have more dead neurons and tokendetectors. Finally, some neurons are positional: them being activated or notdepends largely (or solely) on position and less so (or not at all) on textualdata. We find that smaller models have sets of neurons acting as position rangeindicators while larger models operate in a less explicit manner.

Neurons in Large Language Models: Dead, N-gram, Positional