English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Despite their ability to memorize large datasets, deep neural networks oftenachieve good generalization performance. However, the differences between thelearned solutions of networks which generalize and those which do not remainunclear. Additionally, the tuning properties of single directions (defined asthe activation of a single unit or some linear combination of units in responseto some input) have been highlighted, but their importance has not beenevaluated. Here, we connect these lines of inquiry to demonstrate that anetwork's reliance on single directions is a good predictor of itsgeneralization performance, across networks trained on datasets with differentfractions of corrupted labels, across ensembles of networks trained on datasetswith unmodified labels, across different hyperparameters, and over the courseof training. While dropout only regularizes this quantity up to a point, batchnormalization implicitly discourages single direction reliance, in part bydecreasing the class selectivity of individual units. Finally, we find thatclass selectivity is a poor predictor of task importance, suggesting not onlythat networks which generalize well minimize their dependence on individualunits by reducing their selectivity, but also that individually selective unitsmay not be necessary for strong network performance.

On the importance of single directions for generalization