English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Sparse neural networks attract increasing interest as they exhibit comparableperformance to their dense counterparts while being computationally efficient.Pruning the dense neural networks is among the most widely used methods toobtain a sparse neural network. Driven by the high training cost of suchmethods that can be unaffordable for a low-resource device, training sparseneural networks sparsely from scratch has recently gained attention. However,existing sparse training algorithms suffer from various issues, including poorperformance in high sparsity scenarios, computing dense gradient informationduring training, or pure random topology search. In this paper, inspired by theevolution of the biological brain and the Hebbian learning theory, we present anew sparse training approach that evolves sparse neural networks according tothe behavior of neurons in the network. Concretely, by exploiting the cosinesimilarity metric to measure the importance of the connections, our proposedmethod, Cosine similarity-based and Random Topology Exploration (CTRE), evolvesthe topology of sparse neural networks by adding the most important connectionsto the network without calculating dense gradient in the backward. We carriedout different experiments on eight datasets, including tabular, image, and textdatasets, and demonstrate that our proposed method outperforms severalstate-of-the-art sparse training algorithms in extremely sparse neural networksby a large gap. The implementation code is available onhttps://github.com/zahraatashgahi/CTRE

A Brain-inspired Algorithm for Training Highly Sparse Neural Networks