English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Biological cortical neurons are remarkably sophisticated computationaldevices, temporally integrating their vast synaptic input over an intricatedendritic tree, subject to complex, nonlinearly interacting internal biologicalprocesses. A recent study proposed to characterize this complexity by fittingaccurate surrogate models to replicate the input-output relationship of adetailed biophysical cortical pyramidal neuron model and discovered it neededtemporal convolutional networks (TCN) with millions of parameters. Requiringthese many parameters, however, could be the result of a misalignment betweenthe inductive biases of the TCN and cortical neuron's computations. In light ofthis, and with the aim to explore the computational implications of leakymemory units and nonlinear dendritic processing, we introduce the ExpressiveLeaky Memory (ELM) neuron model, a biologically inspired phenomenological modelof a cortical neuron. Remarkably, by exploiting a few such slowly decayingmemory-like hidden states and two-layered nonlinear integration of synapticinput, our ELM neuron can accurately match the aforementioned input-outputrelationship with under ten-thousand trainable parameters. To further assessthe computational ramifications of our neuron design, we evaluate on varioustasks with demanding temporal structures, including the Long Range Arena (LRA)datasets, as well as a novel neuromorphic dataset based on the SpikingHeidelberg Digits dataset (SHD-Adding). Leveraging a larger number of memoryunits with sufficiently long timescales, and correspondingly sophisticatedsynaptic integration, the ELM neuron proves to be competitive on both datasets,reliably outperforming the classic Transformer or Chrono-LSTM architectures onlatter, even solving the Pathfinder-X task with over $70\%$ accuracy (16kcontext length).

The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks