English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Deep neural networks based on linear RNNs interleaved with position-wise MLPsare gaining traction as competitive approaches for sequence modeling. Examplesof such architectures include state-space models (SSMs) like S4, LRU, andMamba: recently proposed models that achieve promising performance on text,genetics, and other data that require long-range reasoning. Despiteexperimental evidence highlighting these architectures' effectiveness andcomputational efficiency, their expressive power remains relatively unexplored,especially in connection to specific choices crucial in practice - e.g.,carefully designed initialization distribution and potential use of complexnumbers. In this paper, we show that combining MLPs with both real or complexlinear diagonal recurrences leads to arbitrarily precise approximation ofregular causal sequence-to-sequence maps. At the heart of our proof, we rely ona separation of concerns: the linear RNN provides a lossless encoding of theinput sequence, and the MLP performs non-linear processing on this encoding.While we show that real diagonal linear recurrences are enough to achieveuniversality in this architecture, we prove that employing complex eigenvaluesnear unit disk - i.e., empirically the most successful strategy in S4 - greatlyhelps the RNN in storing information. We connect this finding with thevanishing gradient issue and provide experiments supporting our claims.

Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues