English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

To improve the robustness of transformer neural networks used fortemporal-dynamics prediction of chaotic systems, we propose a novel attentionmechanism called easy attention which we demonstrate in time-seriesreconstruction and prediction. As a consequence of the fact that self attentiononly makes useof the inner product of queries and keys, it is demonstrated thatthe keys, queries and softmax are not necessary for obtaining the attentionscore required to capture long-term dependencies in temporal sequences. Throughimplementing singular-value decomposition (SVD) on the softmax attention score,we further observe that the self attention compresses contribution from bothqueries and keys in the spanned space of the attention score. Therefore, ourproposed easy-attention method directly treats the attention scores aslearnable parameters. This approach produces excellent results whenreconstructing and predicting the temporal dynamics of chaotic systemsexhibiting more robustness and less complexity than the self attention or thewidely-used long short-term memory (LSTM) network. Our results show greatpotential for applications in more complex high-dimensional dynamical systems.Keywords: Machine Learning, Transformer, Self Attention, Koopman Operator,Chaotic System.

Easy attention: A simple self-attention mechanism for transformer-based time-series reconstruction and prediction