English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we contend that a natural objective of representation learningis to compress and transform the distribution of the data, say sets of tokens,towards a low-dimensional Gaussian mixture supported on incoherent subspaces.The goodness of such a representation can be evaluated by a principled measure,called sparse rate reduction, that simultaneously maximizes the intrinsicinformation gain and extrinsic sparsity of the learned representation. Fromthis perspective, popular deep network architectures, including transformers,can be viewed as realizing iterative schemes to optimize this measure.Particularly, we derive a transformer block from alternating optimization onparts of this objective: the multi-head self-attention operator compresses therepresentation by implementing an approximate gradient descent step on thecoding rate of the features, and the subsequent multi-layer perceptronsparsifies the features. This leads to a family of white-box transformer-likedeep network architectures, named CRATE, which are mathematically fullyinterpretable. We show, by way of a novel connection between denoising andcompression, that the inverse to the aforementioned compressive encoding can berealized by the same class of CRATE architectures. Thus, the so-derivedwhite-box architectures are universal to both encoders and decoders.Experiments show that these networks, despite their simplicity, indeed learn tocompress and sparsify representations of large-scale real-world image and textdatasets, and achieve performance very close to highly engineeredtransformer-based models: ViT, MAE, DINO, BERT, and GPT2. We believe theproposed computational framework demonstrates great potential in bridging thegap between theory and practice of deep learning, from a unified perspective ofdata compression. Code is available at: https://ma-lab-berkeley.github.io/CRATE .

White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?