English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

It is commonly observed that deep networks trained for classification exhibitclass-selective neurons in their early and intermediate layers. Intriguingly,recent studies have shown that these class-selective neurons can be ablatedwithout deteriorating network function. But if class-selective neurons are notnecessary, why do they exist? We attempt to answer this question in a series ofexperiments on ResNet-50s trained on ImageNet. We first show thatclass-selective neurons emerge during the first few epochs of training, beforereceding rapidly but not completely; this suggests that class-selective neuronsfound in trained networks are in fact vestigial remains of early training. Withsingle-neuron ablation experiments, we then show that class-selective neuronsare important for network function in this early phase of training. We alsoobserve that the network is close to a linear regime in this early phase; wethus speculate that class-selective neurons appear early in training asquasi-linear shortcut solutions to the classification task. Finally, in causalexperiments where we regularize against class selectivity at different pointsin training, we show that the presence of class-selective neurons early intraining is critical to the successful training of the network; in contrast,class-selective neurons can be suppressed later in training with little effecton final accuracy. It remains to be understood by which mechanism the presenceof class-selective neurons in the early phase of training contributes to thesuccessful training of networks.

On the special role of class-selective neurons in early training