English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

From the point of view of the human brain, continual learning can performvarious tasks without mutual interference. An effective way to reduce mutualinterference can be found in sparsity and selectivity of neurons. According toAljundi et al. and Hadsell et al., imposing sparsity at the representationallevel is advantageous for continual learning because sparse neuronalactivations encourage less overlap between parameters, resulting in lessinterference. Similarly, highly selective neural networks are likely to induceless interference since particular response in neurons will reduce the chanceof overlap with other parameters. Considering that the human brain performscontinual learning over the lifespan, finding conditions where sparsity andselectivity naturally arises may provide insight for understanding how thebrain functions. This paper investigates various conditions that naturallyincrease sparsity and selectivity in a neural network. This paper testeddifferent optimizers with Hoyer's sparsity metric and CCMAS selectivity metricin MNIST classification task. It is essential to note that investigations onthe natural occurrence of sparsity and selectivity concerning variousconditions have not been acknowledged in any sector of neuroscience nor machinelearning until this day. This paper found that particular conditions increasesparsity and selectivity such as applying a large learning rate and lowering abatch size. In addition to the relationship between the condition, sparsity,and selectivity, the following will be discussed based on empirical analysis:1. The relationship between sparsity and selectivity and 2. The relationshipbetween test accuracy, sparsity, and selectivity.

Representations learnt by SGD and Adaptive learning rules: Conditions that vary sparsity and selectivity in neural networks