English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we introduce a new perspective on training deep neuralnetworks capable of state-of-the-art performance without the need for theexpensive over-parameterization by proposing the concept of In-TimeOver-Parameterization (ITOP) in sparse training. By starting from a randomsparse network and continuously exploring sparse connectivities duringtraining, we can perform an Over-Parameterization in the space-time manifold,closing the gap in the expressibility between sparse training and densetraining. We further use ITOP to understand the underlying mechanism of DynamicSparse Training (DST) and indicate that the benefits of DST come from itsability to consider across time all possible parameters when searching for theoptimal sparse connectivity. As long as there are sufficient parameters thathave been reliably explored during training, DST can outperform the denseneural network by a large margin. We present a series of experiments to supportour conjecture and achieve the state-of-the-art sparse training performancewith ResNet-50 on ImageNet. More impressively, our method achieves dominantperformance over the overparameterization-based sparse methods at extremesparsity levels. When trained on CIFAR-100, our method can match theperformance of the dense model even at an extreme sparsity (98%). Code can befound https://github.com/Shiweiliu/In-Time-Over-Parameterization.

Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training