English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In neural networks, task-relevant information is represented jointly bygroups of neurons. However, the specific way in which this mutual informationabout the classification label is distributed among the individual neurons isnot well understood: While parts of it may only be obtainable from specificsingle neurons, other parts are carried redundantly or synergistically bymultiple neurons. We show how Partial Information Decomposition (PID), a recentextension of information theory, can disentangle these different contributions.From this, we introduce the measure of "Representational Complexity", whichquantifies the difficulty of accessing information spread across multipleneurons. We show how this complexity is directly computable for smaller layers.For larger layers, we propose subsampling and coarse-graining procedures andprove corresponding bounds on the latter. Empirically, for quantized deepneural networks solving the MNIST and CIFAR10 tasks, we observe thatrepresentational complexity decreases both through successive hidden layers andover training, and compare the results to related measures. Overall, we proposerepresentational complexity as a principled and interpretable summary statisticfor analyzing the structure and evolution of neural representations and complexsystems in general.

A Measure of the Complexity of Neural Representations based on Partial Information Decomposition