English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

One technique to visualize the training of neural networks is to perform PCAon the parameters over the course of training and to project to the subspacespanned by the first few PCA components. In this paper we compare thistechnique to the PCA of a high dimensional random walk. We compute theeigenvalues and eigenvectors of the covariance of the trajectory and prove thatin the long trajectory and high dimensional limit most of the variance is inthe first few PCA components, and that the projection of the trajectory ontoany subspace spanned by PCA components is a Lissajous curve. We generalizethese results to a random walk with momentum and to an Ornstein-Uhlenbeckprocesses (i.e., a random walk in a quadratic potential) and show that in highdimensions the walk is not mean reverting, but will instead be trapped at afixed distance from the minimum. We finally compare the distribution of PCAvariances and the PCA projected training trajectories of a linear model trainedon CIFAR-10 and ResNet-50-v2 trained on Imagenet and find that the distributionof PCA variances resembles a random walk with drift.

PCA of high dimensional random walks with comparison to neural network training