English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Tools annual

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools monthly

Global: Zendy Plus monthly

Presents the access of premium content as premium feature

Premium Content

Global: Zendy Plus annual

Global: Zendy Free Plan

Neural network pruning is a fruitful area of research with surging interestin high sparsity regimes. Benchmarking in this domain heavily relies onfaithful representation of the sparsity of subnetworks, which has beentraditionally computed as the fraction of removed connections (directsparsity). This definition, however, fails to recognize unpruned parametersthat detached from input or output layers of underlying subnetworks,potentially underestimating actual effective sparsity: the fraction ofinactivated connections. While this effect might be negligible for moderatelypruned networks (up to 10-100 compression rates), we find that it plays anincreasing role for thinner subnetworks, greatly distorting comparison betweendifferent pruning algorithms. For example, we show that effective compressionof a randomly pruned LeNet-300-100 can be orders of magnitude larger than itsdirect counterpart, while no discrepancy is ever observed when using SynFlowfor pruning [Tanaka et al., 2020]. In this work, we adopt the lens of effectivesparsity to reevaluate several recent pruning algorithms on common benchmarkarchitectures (e.g., LeNet-300-100, VGG-19, ResNet-18) and discover that theirabsolute and relative performance changes dramatically in this new and moreappropriate framework. To aim for effective, rather than direct, sparsity, wedevelop a low-cost extension to most pruning algorithms. Further, equipped witheffective sparsity as a reference frame, we partially reconfirm that randompruning with appropriate sparsity allocation across layers performs as well orbetter than more sophisticated algorithms for pruning at initialization [Su etal., 2020]. In response to this observation, using a simple analogy of pressuredistribution in coupled cylinders from physics, we design novel layerwisesparsity quotas that outperform all existing baselines in the context of randompruning.

Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity