English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformer-based models have achieved great success in various NLP, vision,and speech tasks. However, the core of Transformer, the self-attentionmechanism, has a quadratic time and memory complexity with respect to thesequence length, which hinders applications of Transformer-based models to longsequences. Many approaches have been proposed to mitigate this problem, such assparse attention mechanisms, low-rank matrix approximations and scalablekernels, and token mixing alternatives to self-attention. We propose a novelPooling Network (PoNet) for token mixing in long sequences with linearcomplexity. We design multi-granularity pooling and pooling fusion to capturedifferent levels of contextual information and combine their interactions withtokens. On the Long Range Arena benchmark, PoNet significantly outperformsTransformer and achieves competitive accuracy, while being only slightly slowerthan the fastest model, FNet, across all sequence lengths measured on GPUs. Wealso conduct systematic studies on the transfer learning capability of PoNetand observe that PoNet achieves 95.7% of the accuracy of BERT on the GLUEbenchmark, outperforming FNet by 4.5% relative. Comprehensive ablation analysisdemonstrates effectiveness of the designed multi-granularity pooling andpooling fusion for token mixing in long sequences and efficacy of the designedpre-training tasks for PoNet to learn transferable contextualized languagerepresentations.

PoNet: Pooling Network for Efficient Token Mixing in Long Sequences