English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Linear Recurrence has proven to be a powerful tool for modeling longsequences efficiently. In this work, we show that existing models fail to takefull advantage of its potential. Motivated by this finding, we developGateLoop, a foundational sequence model that generalizes linear recurrentmodels such as S4, S5, LRU and RetNet, by employing data-controlled statetransitions. Utilizing this theoretical advance, GateLoop empiricallyoutperforms existing models for auto-regressive language modeling. Our methodcomes with a low-cost $O(l)$ recurrent mode and an efficient $O(l \log_{2} l)$parallel mode making use of highly optimized associative scan implementations.Furthermore, we derive an $O(l^2)$ surrogate attention mode, revealingremarkable implications for Transformer and recently proposed architectures.Specifically, we prove that our approach can be interpreted as providingdata-controlled relative-positional information to Attention. While manyexisting models solely rely on data-controlled cumulative sums for contextaggregation, our findings suggest that incorporating data-controlled complexcumulative products may be a crucial step towards more powerful sequencemodels.

GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling