English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Artificial neural networks open up unprecedented machine learningcapabilities at the cost of ever growing computational requirements.Sparsifying the parameters, often achieved through weight pruning, has beenidentified as a powerful technique to compress the number of model parametersand reduce the computational operations of neural networks. Yet, sparseactivations, while omnipresent in both biological neural networks and deeplearning systems, have not been fully utilized as a compression technique indeep learning. Moreover, the interaction between sparse activations and weightpruning is not fully understood. In this work, we demonstrate that activitysparsity can compose multiplicatively with parameter sparsity in a recurrentneural network model based on the GRU that is designed to be activity sparse.We achieve up to $20\times$ reduction of computation while maintainingperplexities below $60$ on the Penn Treebank language modeling task. Thismagnitude of reduction has not been achieved previously with solely sparselyconnected LSTMs, and the language modeling performance of our model has notbeen achieved previously with any sparsely activated recurrent neural networksor spiking neural networks. Neuromorphic computing devices are especially goodat taking advantage of the dynamic activity sparsity, and our results providestrong evidence that making deep learning models activity sparse and portingthem to neuromorphic devices can be a viable strategy that does not compromiseon task performance. Our results also drive further convergence of methods fromdeep learning and neuromorphic computing for efficient machine learning.

Activity Sparsity Complements Weight Sparsity for Efficient RNN Inference