English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Deep neural networks excel in supervised learning tasks but are constrainedby the need for extensive labeled data. Self-supervised learning emerges as apromising alternative, allowing models to learn without explicit labels.Information theory, and notably the information bottleneck principle, has beenpivotal in shaping deep neural networks. This principle focuses on optimizingthe trade-off between compression and preserving relevant information,providing a foundation for efficient network design in supervised contexts.However, its precise role and adaptation in self-supervised learning remainunclear. In this work, we scrutinize various self-supervised learningapproaches from an information-theoretic perspective, introducing a unifiedframework that encapsulates the \textit{self-supervised information-theoreticlearning problem}. We weave together existing research into a cohesivenarrative, delve into contemporary self-supervised methodologies, and spotlightpotential research avenues and inherent challenges. Additionally, we discussthe empirical evaluation of information-theoretic quantities and theirestimation methods. Overall, this paper furnishes an exhaustive review of theintersection of information theory, self-supervised learning, and deep neuralnetworks.

To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review