English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Model pruning is an essential procedure for building compact andcomputationally-efficient machine learning models. A key feature of a goodpruning algorithm is that it accurately quantifies the relative importance ofthe model weights. While model pruning has a rich history, we still don't havea full grasp of the pruning mechanics even for relatively simple problemsinvolving linear models or shallow neural nets. In this work, we provide aprincipled exploration of pruning by building on a natural notion ofimportance. For linear models, we show that this notion of importance iscaptured by covariance scaling which connects to the well-known Hessian-basedpruning. We then derive asymptotic formulas that allow us to precisely comparethe performance of different pruning methods. For neural networks, wedemonstrate that the importance can be at odds with larger magnitudes andproper initialization is critical for magnitude-based pruning. Specifically, weidentify settings in which weights become more important despite becomingsmaller, which in turn leads to a catastrophic failure of magnitude-basedpruning. Our results also elucidate that implicit regularization in the form ofHessian structure has a catalytic role in identifying the important weights,which dictate the pruning performance.

Exploring Weight Importance and Hessian Bias in Model Pruning