English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

How much can pruning algorithms teach us about the fundamentals of learningrepresentations in neural networks? And how much can these fundamentals helpwhile devising new pruning techniques? A lot, it turns out. Neural networkpruning has become a topic of great interest in recent years, and manydifferent techniques have been proposed to address this problem. The decisionof what to prune and when to prune necessarily forces us to confront ourassumptions about how neural networks actually learn to represent patterns indata. In this work, we set out to test several long-held hypotheses aboutneural network learning representations, approaches to pruning and therelevance of one in the context of the other. To accomplish this, we argue infavor of pruning whole neurons as opposed to the traditional method of pruningweights from optimally trained networks. We first review the historicalliterature, point out some common assumptions it makes, and propose methods todemonstrate the inherent flaws in these assumptions. We then propose our novelapproach to pruning and set about analyzing the quality of the decisions itmakes. Our analysis led us to question the validity of many widely-heldassumptions behind pruning algorithms and the trade-offs we often make in theinterest of reducing computational complexity. We discovered that there is astraightforward way, however expensive, to serially prune 40-70% of the neuronsin a trained network with minimal effect on the learning representation andwithout any re-training. It is to be noted here that the motivation behind thiswork is not to propose an algorithm that would outperform all existing methods,but to shed light on what some inherent flaws in these methods can teach usabout learning representations and how this can lead us to superior pruningtechniques.

The Incredible Shrinking Neural Network: New Perspectives on Learning Representations Through The Lens of Pruning