English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Short-term memory in standard, general-purpose, sequence-processing recurrentneural networks (RNNs) is stored as activations of nodes or "neurons."Generalising feedforward NNs to such RNNs is mathematically straightforward andnatural, and even historical: already in 1943, McCulloch and Pitts proposedthis as a surrogate to "synaptic modifications" (in effect, generalising theLenz-Ising model, the first non-sequence processing RNN architecture of the1920s). A lesser known alternative approach to storing short-term memory in"synaptic connections" -- by parameterising and controlling the dynamics of acontext-sensitive time-varying weight matrix through another NN -- yieldsanother "natural" type of short-term memory in sequence processing NNs: theFast Weight Programmers (FWPs) of the early 1990s. FWPs have seen a recentrevival as generic sequence processors, achieving competitive performanceacross various tasks. They are formally closely related to the now popularTransformers. Here we present them in the context of artificial NNs as anabstraction of biological NNs -- a perspective that has not been stressedenough in previous FWP work. We first review aspects of FWPs for pedagogicalpurposes, then discuss connections to related works motivated by insights fromneuroscience.

Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks