English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Random masks define surprisingly effective sparse neural network models, ashas been shown empirically. The resulting sparse networks can often competewith dense architectures and state-of-the-art lottery ticket pruningalgorithms, even though they do not rely on computationally expensiveprune-train iterations and can be drawn initially without significantcomputational overhead. We offer a theoretical explanation of how random maskscan approximate arbitrary target networks if they are wider by a logarithmicfactor in the inverse sparsity $1 / \log(1/\text{sparsity})$. Thisoverparameterization factor is necessary at least for 3-layer random networks,which elucidates the observed degrading performance of random networks athigher sparsity. At moderate to high sparsity levels, however, our resultsimply that sparser networks are contained within random source networks so thatany dense-to-sparse training scheme can be turned into a computationally moreefficient sparse-to-sparse one by constraining the search to a fixed randommask. We demonstrate the feasibility of this approach in experiments fordifferent pruning methods and propose particularly effective choices of initiallayer-wise sparsity ratios of the random source network. As a special case, weshow theoretically and experimentally that random source networks also containstrong lottery tickets.

Why Random Pruning Is All We Need to Start Sparse