English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Recent works show that overparameterized networks contain small subnetworksthat exhibit comparable accuracy to the full model when trained in isolation.These results highlight the potential to reduce training costs of deep neuralnetworks without sacrificing generalization performance. However, existingapproaches for finding these small networks rely on expensive multi-roundtrain-and-prune procedures and are non-practical for large data sets andmodels. In this paper, we show how to find small networks that exhibit the sameperformance as their overparameterized counterparts after only a few trainingepochs. We find that hidden layer activations in overparameterized networksexist primarily in subspaces smaller than the actual model width. Building onthis observation, we use PCA to find a basis of high variance for layer inputsand represent layer weights using these directions. We eliminate all weightsnot relevant to the found PCA basis and term these network architecturesPrincipal Component Networks. On CIFAR-10 and ImageNet, we show that PCNs trainfaster and use less energy than overparameterized models, without accuracyloss. We find that our transformation leads to networks with up to 23.8x fewerparameters, with equal or higher end-model accuracy---in some cases we observeimprovements up to 3%. We also show that ResNet-20 PCNs outperform deepResNet-110 networks while training faster.

Principal Component Networks: Parameter Reduction Early in Training