English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Representational drift refers to over-time changes in neural activationaccompanied by a stable task performance. Despite being observed in the brainand in artificial networks, the mechanisms of drift and its implications arenot fully understood. Motivated by recent experimental findings ofstimulus-dependent drift in the piriform cortex, we use theory and simulationsto study this phenomenon in a two-layer linear feedforward network.Specifically, in a continual online learning scenario, we study the driftinduced by the noise inherent in the Stochastic Gradient Descent (SGD). Bydecomposing the learning dynamics into the normal and tangent spaces of theminimum-loss manifold, we show the former corresponds to a finite variancefluctuation, while the latter could be considered as an effective diffusionprocess on the manifold. We analytically compute the fluctuation and thediffusion coefficients for the stimuli representations in the hidden layer asfunctions of network parameters and input distribution. Further, consistentwith experiments, we show that the drift rate is slower for a more frequentlypresented stimulus. Overall, our analysis yields a theoretical framework forbetter understanding of the drift phenomenon in biological and artificialneural networks.

Stochastic Gradient Descent-Induced Drift of Representation in a Two-Layer Neural Network