English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

The growing energy and performance costs of deep learning have driven thecommunity to reduce the size of neural networks by selectively pruningcomponents. Similarly to their biological counterparts, sparse networksgeneralize just as well, if not better than, the original dense networks.Sparsity can reduce the memory footprint of regular networks to fit mobiledevices, as well as shorten training time for ever growing networks. In thispaper, we survey prior work on sparsity in deep learning and provide anextensive tutorial of sparsification for both inference and training. Wedescribe approaches to remove and add elements of neural networks, differenttraining strategies to achieve model sparsity, and mechanisms to exploitsparsity in practice. Our work distills ideas from more than 300 researchpapers and provides guidance to practitioners who wish to utilize sparsitytoday, as well as to researchers whose goal is to push the frontier forward. Weinclude the necessary background on mathematical methods in sparsification,describe phenomena such as early structure adaptation, the intricate relationsbetween sparsity and the training process, and show techniques for achievingacceleration on real hardware. We also define a metric of pruned parameterefficiency that could serve as a baseline for comparison of different sparsenetworks. We close by speculating on how sparsity can improve future workloadsand outline major open problems in the field.

Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks