English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In this paper, we propose a probabilistic representation of MultiLayerPerceptrons (MLPs) to improve the information-theoretic interpretability. Aboveall, we demonstrate that the activations being i.i.d. is not valid for all thehidden layers of MLPs, thus the existing mutual information estimators based onnon-parametric inference methods, e.g., empirical distributions and KernelDensity Estimate (KDE), are invalid for measuring the information flow in MLPs.Moreover, we introduce explicit probabilistic explanations for MLPs: (i) wedefine the probability space (Omega_F, t, P_F) for a fully connected layer fand demonstrate the great effect of an activation function on the probabilitymeasure P_F ; (ii) we prove the entire architecture of MLPs as a Gibbsdistribution P; and (iii) the back-propagation aims to optimize the samplespace Omega_F of all the fully connected layers of MLPs for learning an optimalGibbs distribution P* to express the statistical connection between the inputand the label. Based on the probabilistic explanations for MLPs, we improve theinformation-theoretic interpretability of MLPs in three aspects: (i) the randomvariable of f is discrete and the corresponding entropy is finite; (ii) theinformation bottleneck theory cannot correctly explain the information flow inMLPs if we take into account the back-propagation; and (iii) we propose novelinformation-theoretic explanations for the generalization of MLPs. Finally, wedemonstrate the proposed probabilistic representation and information-theoreticexplanations for MLPs in a synthetic dataset and benchmark datasets.

A Probabilistic Representation of Deep Learning for Improving The Information Theoretic Interpretability