English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In the Information Bottleneck (IB), when tuning the relative strength betweencompression and prediction terms, how do the two terms behave, and what's theirrelationship with the dataset and the learned representation? In this paper, weset out to answer these questions by studying multiple phase transitions in theIB objective: $\text{IB}_\beta[p(z|x)] = I(X; Z) - \beta I(Y; Z)$ defined onthe encoding distribution p(z|x) for input $X$, target $Y$ and representation$Z$, where sudden jumps of $dI(Y; Z)/d \beta$ and prediction accuracy areobserved with increasing $\beta$. We introduce a definition for IB phasetransitions as a qualitative change of the IB loss landscape, and show that thetransitions correspond to the onset of learning new classes. Using second-ordercalculus of variations, we derive a formula that provides a practical conditionfor IB phase transitions, and draw its connection with the Fisher informationmatrix for parameterized models. We provide two perspectives to understand theformula, revealing that each IB phase transition is finding a component ofmaximum (nonlinear) correlation between $X$ and $Y$ orthogonal to the learnedrepresentation, in close analogy with canonical-correlation analysis (CCA) inlinear settings. Based on the theory, we present an algorithm for discoveringphase transition points. Finally, we verify that our theory and algorithmaccurately predict phase transitions in categorical datasets, predict the onsetof learning new classes and class difficulty in MNIST, and predict prominentphase transitions in CIFAR10.

Phase Transitions for the Information Bottleneck in Representation Learning