English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The quadratic complexity of attention in transformer architectures remains abig bottleneck in scaling up large foundation models for long context. In fact,recent theoretical results show the hardness of approximating the output ofsoftmax attention mechanism in sub-quadratic time assuming Strong ExponentialTime Hypothesis. In this paper, we show how to break this theoretical barrierby replacing softmax with a polynomial function and polynomial sketching. Inparticular we show that sketches for Polynomial Kernel from the randomizednumerical linear algebra literature can be used to approximate the polynomialattention which leads to a significantly faster attention mechanism withoutassuming any sparse structure for the attention matrix that has been done inmany previous works.  In addition, we propose an efficient block-based algorithm that lets us applythe causal mask to the attention matrix without explicitly realizing the $n\times n$ attention matrix and compute the output of the polynomial attentionmechanism in time linear in the context length. The block-based algorithm givessignificant speedups over the \emph{cumulative sum} algorithm used by Performerto apply the causal mask to the attention matrix. These observations help usdesign \emph{PolySketchFormer}, a practical linear-time transformerarchitecture for language modeling with provable guarantees.  We validate our design empirically by training language models with longcontext lengths. We first show that the eval perplexities of our models arecomparable to that of models trained with softmax attention. We then show thatfor large context lengths our training times are significantly faster thanFlashAttention.

PolySketchFormer: Fast Transformers via Sketches for Polynomial Kernels