English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Efficient Transformers have been developed for long sequence modeling, due totheir subquadratic memory and time complexity. Sparse Transformer is a popularapproach to improving the efficiency of Transformers by restrictingself-attention to locations specified by the predefined sparse patterns.However, leveraging sparsity may sacrifice expressiveness compared tofull-attention, when important token correlations are multiple hops away. Tocombine advantages of both the efficiency of sparse transformer and theexpressiveness of full-attention Transformer, we propose \textit{Diffuser}, anew state-of-the-art efficient Transformer. Diffuser incorporates all tokeninteractions within one attention layer while maintaining low computation andmemory costs. The key idea is to expand the receptive field of sparse attentionusing Attention Diffusion, which computes multi-hop token correlations based onall paths between corresponding disconnected tokens, besides attention amongneighboring tokens. Theoretically, we show the expressiveness of Diffuser as auniversal sequence approximator for sequence-to-sequence modeling, andinvestigate its ability to approximate full-attention by analyzing the graphexpander property from the spectral perspective. Experimentally, we investigatethe effectiveness of Diffuser with extensive evaluations, including languagemodeling, image modeling, and Long Range Arena (LRA). Evaluation results showthat Diffuser achieves improvements by an average of 0.94% on textclassification tasks and 2.30% on LRA, with 1.67$\times$ memory savingscompared to state-of-the-art benchmarks, which demonstrates superiorperformance of Diffuser in both expressiveness and efficiency aspects.

Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequences