English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The pressing need to reduce the capacity of deep neural networks hasstimulated the development of network dilution methods and their analysis.While the ability of $L_1$ and $L_0$ regularization to encourage sparsity isoften mentioned, $L_2$ regularization is seldom discussed in this context. Wepresent a novel framework for weight pruning by sampling from a probabilityfunction that favors the zeroing of smaller weights. In addition, we examinethe contribution of $L_1$ and $L_2$ regularization to the dynamics of nodepruning while optimizing for weight pruning. We then demonstrate theeffectiveness of the proposed stochastic framework when used together with aweight decay regularizer on popular classification models in removing 50% ofthe nodes in an MLP for MNIST classification, 60% of the filters in VGG-16 forCIFAR10 classification, and on medical image models in removing 60% of thechannels in a U-Net for instance segmentation and 50% of the channels in CNNmodel for COVID-19 detection. For these node-pruned networks, we also presentcompetitive weight pruning results that are only slightly less accurate thanthe original, dense networks.

The Role of Regularization in Shaping Weight and Node Pruning Dependency and Dynamics