English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Convolutional models have been widely used in multiple domains. However, mostexisting models only use local convolution, making the model unable to handlelong-range dependency efficiently. Attention overcomes this problem byaggregating global information but also makes the computational complexityquadratic to the sequence length. Recently, Gu et al. [2021] proposed a modelcalled S4 inspired by the state space model. S4 can be efficiently implementedas a global convolutional model whose kernel size equals the input sequencelength. S4 can model much longer sequences than Transformers and achievesignificant gains over SoTA on several long-range tasks. Despite its empiricalsuccess, S4 is involved. It requires sophisticated parameterization andinitialization schemes. As a result, S4 is less intuitive and hard to use. Herewe aim to demystify S4 and extract basic principles that contribute to thesuccess of S4 as a global convolutional model. We focus on the structure of theconvolution kernel and identify two critical but intuitive principles enjoyedby S4 that are sufficient to make up an effective global convolutional model:1) The parameterization of the convolutional kernel needs to be efficient inthe sense that the number of parameters should scale sub-linearly with sequencelength. 2) The kernel needs to satisfy a decaying structure that the weightsfor convolving with closer neighbors are larger than the more distant ones.Based on the two principles, we propose a simple yet effective convolutionalmodel called Structured Global Convolution (SGConv). SGConv exhibits strongempirical performance over several tasks: 1) With faster speed, SGConvsurpasses S4 on Long Range Arena and Speech Command datasets. 2) When pluggingSGConv into standard language and vision models, it shows the potential toimprove both efficiency and performance.

What Makes Convolutional Models Great on Long Sequence Modeling?