English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Grokking is a phenomenon where a model trained on an algorithmic task firstoverfits but, then, after a large amount of additional training, undergoes aphase transition to generalize perfectly. We empirically study the internalstructure of networks undergoing grokking on the sparse parity task, and findthat the grokking phase transition corresponds to the emergence of a sparsesubnetwork that dominates model predictions. On an optimization level, we findthat this subnetwork arises when a small subset of neurons undergoes rapid normgrowth, whereas the other neurons in the network decay slowly in norm. Thus, wesuggest that the grokking phase transition can be understood to emerge fromcompetition of two largely distinct subnetworks: a dense one that dominatesbefore the transition and generalizes poorly, and a sparse one that dominatesafterwards.

A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks