English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

This paper studies the curious phenomenon for machine learning models withTransformer architectures that their activation maps are sparse. By activationmap we refer to the intermediate output of the multi-layer perceptrons (MLPs)after a ReLU activation function, and by sparse we mean that on average veryfew entries (e.g., 3.0% for T5-Base and 6.3% for ViT-B16) are nonzero for eachinput to MLP. Moreover, larger Transformers with more layers and wider MLPhidden dimensions are sparser as measured by the percentage of nonzero entries.Through extensive experiments we demonstrate that the emergence of sparsity isa prevalent phenomenon that occurs for both natural language processing andvision tasks, on both training and evaluation data, for Transformers of variousconfigurations, at layers of all depth levels, as well as for otherarchitectures including MLP-mixers and 2-layer MLPs. We show that sparsity alsoemerges using training datasets with random labels, or with random inputs, orwith infinite amount of data, demonstrating that sparsity is not a result of aspecific family of datasets. We discuss how sparsity immediately implies a wayto significantly reduce the FLOP count and improve efficiency for Transformers.Moreover, we demonstrate perhaps surprisingly that enforcing an even sparseractivation via Top-k thresholding with a small value of k brings a collectionof desired but missing properties for Transformers, namely less sensitivity tonoisy training data, more robustness to input corruptions, and bettercalibration for their prediction confidence.

The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers