English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Analyzing deep neural networks (DNNs) via information plane (IP) theory hasgained tremendous attention recently as a tool to gain insight into, amongothers, their generalization ability. However, it is by no means obvious how toestimate mutual information (MI) between each hidden layer and theinput/desired output, to construct the IP. For instance, hidden layers withmany neurons require MI estimators with robustness towards the highdimensionality associated with such layers. MI estimators should also be ableto naturally handle convolutional layers, while at the same time beingcomputationally tractable to scale to large networks. None of the existing IPmethods to date have been able to study truly deep Convolutional NeuralNetworks (CNNs), such as the e.g.\ VGG-16. In this paper, we propose an IPanalysis using the new matrix--based R\'enyi's entropy coupled with tensorkernels over convolutional layers, leveraging the power of kernel methods torepresent properties of the probability distribution independently of thedimensionality of the data. The obtained results shed new light on the previousliterature concerning small-scale DNNs, however using a completely newapproach. Importantly, the new framework enables us to provide the firstcomprehensive IP analysis of contemporary large-scale DNNs and CNNs,investigating the different training phases and providing new insights into thetraining dynamics of large-scale neural networks.

Information Plane Analysis of Deep Neural Networks via Matrix-Based Renyi's Entropy and Tensor Kernels