English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformer and its variants are fundamental neural architectures in deeplearning. Recent works show that learning attention in the Fourier space canimprove the long sequence learning capability of Transformers. We argue thatwavelet transform shall be a better choice because it captures both positionand frequency information with linear time complexity. Therefore, in thispaper, we systematically study the synergy between wavelet transform andTransformers. We propose Wavelet Space Attention (WavSpA) that facilitatesattention learning in a learnable wavelet coefficient space which replaces theattention in Transformers by (1) applying forward wavelet transform to projectthe input sequences to multi-resolution bases, (2) conducting attentionlearning in the wavelet coefficient space, and (3) reconstructing therepresentation in input space via backward wavelet transform. Extensiveexperiments on the Long Range Arena demonstrate that learning attention in thewavelet space using either fixed or adaptive wavelets can consistently improveTransformer's performance and also significantly outperform learning in Fourierspace. We further show our method can enhance Transformer's reasoningextrapolation capability over distance on the LEGO chain-of-reasoning task.

WavSpA: Wavelet Space Attention for Boosting Transformers' Long Sequence Learning Ability