English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Transformers with linearised attention (''linear Transformers'') havedemonstrated the practical scalability and effectiveness of outer product-basedFast Weight Programmers (FWPs) from the '90s. However, the original FWPformulation is more general than the one of linear Transformers: a slow neuralnetwork (NN) continually reprograms the weights of a fast NN with arbitraryarchitecture. In existing linear Transformers, both NNs are feedforward andconsist of a single layer. Here we explore new variations by adding recurrenceto the slow and fast nets. We evaluate our novel recurrent FWPs (RFWPs) on twosynthetic algorithmic tasks (code execution and sequential ListOps),Wikitext-103 language models, and on the Atari 2600 2D game environment. Ourmodels exhibit properties of Transformers and RNNs. In the reinforcementlearning setting, we report large improvements over LSTM in several Atarigames. Our code is public.

Going Beyond Linear Transformers with Recurrent Fast Weight Programmers