English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

State-space models (SSMs) have recently emerged as a framework for learninglong-range sequence tasks. An example is the structured state-space sequence(S4) layer, which uses the diagonal-plus-low-rank structure of the HiPPOinitialization framework. However, the complicated structure of the S4 layerposes challenges; and, in an effort to address these challenges, models such asS4D and S5 have considered a purely diagonal structure. This choice simplifiesthe implementation, improves computational efficiency, and allows channelcommunication. However, diagonalizing the HiPPO framework is itself anill-posed problem. In this paper, we propose a general solution for this andrelated ill-posed diagonalization problems in machine learning. We introduce ageneric, backward-stable "perturb-then-diagonalize" (PTD) methodology, which isbased on the pseudospectral theory of non-normal operators, and which may beinterpreted as the approximate diagonalization of the non-normal matricesdefining SSMs. Based on this, we introduce the S4-PTD and S5-PTD models.Through theoretical analysis of the transfer functions of differentinitialization schemes, we demonstrate that the S4-PTD/S5-PTD initializationstrongly converges to the HiPPO framework, while the S4D/S5 initialization onlyachieves weak convergences. As a result, our new models show resilience toFourier-mode noise-perturbed inputs, a crucial property not achieved by theS4D/S5 models. In addition to improved robustness, our S5-PTD model averages87.6% accuracy on the Long-Range Arena benchmark, demonstrating that the PTDmethodology helps to improve the accuracy of deep learning models.

Robustifying State-space Models for Long Sequences via Approximate Diagonalization