English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

The weight matrix (WM) of a neural network (NN) is its program. The programsof many traditional NNs are learned through gradient descent in some errorfunction, then remain fixed. The WM of a self-referential NN, however, can keeprapidly modifying all of itself during runtime. In principle, such NNs canmeta-learn to learn, and meta-meta-learn to meta-learn to learn, and so on, inthe sense of recursive self-improvement. While NN architectures potentiallycapable of implementing such behaviour have been proposed since the '90s, therehave been few if any practical studies. Here we revisit such NNs, building uponrecent successes of fast weight programmers and closely related linearTransformers. We propose a scalable self-referential WM (SRWM) that learns touse outer products and the delta update rule to modify itself. We evaluate ourSRWM in supervised few-shot learning and in multi-task reinforcement learningwith procedurally generated game environments. Our experiments demonstrate bothpractical applicability and competitive performance of the proposed SRWM. Ourcode is public.

A Modern Self-Referential Weight Matrix That Learns to Modify Itself