English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Training deep neural networks with an $L_0$ regularization is one of theprominent approaches for network pruning or sparsification. The method prunesthe network during training by encouraging weights to become exactly zero.However, recent work of Gale et al. reveals that although this method yieldshigh compression rates on smaller datasets, it performs inconsistently onlarge-scale learning tasks, such as ResNet50 on ImageNet. We analyze thisphenomenon through the lens of variational inference and find that it is likelydue to the independent modeling of binary gates, the mean-field approximation,which is known in Bayesian statistics for its poor performance due to the crudeapproximation. To mitigate this deficiency, we propose a dependency modeling ofbinary gates, which can be modeled effectively as a multi-layer perceptron(MLP). We term our algorithm Dep-$L_0$ as it prunes networks via adependency-enabled $L_0$ regularization. Extensive experiments on CIFAR10,CIFAR100 and ImageNet with VGG16, ResNet50, ResNet56 show that our Dep-$L_0$outperforms the original $L_0$-HC algorithm of Louizos et al. by a significantmargin, especially on ImageNet. Compared with the state-of-the-arts networksparsification algorithms, our dependency modeling makes the $L_0$-basedsparsification once again very competitive on large-scale learning tasks. Oursource code is available at https://github.com/leo-yangli/dep-l0.

Dep-$L_0$: Improving $L_0$-based Network Sparsification via Dependency Modeling