Premium
Entropy concentration and the empirical coding game
Author(s) -
Grünwald Peter
Publication year - 2008
Publication title -
statistica neerlandica
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.52
H-Index - 39
eISSN - 1467-9574
pISSN - 0039-0402
DOI - 10.1111/j.1467-9574.2008.00401.x
Subject(s) - mathematics , maximum entropy probability distribution , entropy (arrow of time) , conditional entropy , principle of maximum entropy , min entropy , maximum entropy thermodynamics , inference , discrete mathematics , combinatorics , binary entropy function , statistical physics , statistics , computer science , artificial intelligence , physics , thermodynamics
We give a characterization of maximum entropy/minimum relative entropy inference by providing two ‘strong entropy concentration’ theorems. These theorems unify and generalize Jaynes’‘concentration phenomenon’ and Van Campenhout and Cover's ‘conditional limit theorem’. The theorems characterize exactly in what sense a prior distribution Q conditioned on a given constraint and the distribution minimizing D(P ‖ Q ) over all P satisfying the constraint are ‘close’ to each other. We then apply our theorems to establish the relationship between entropy concentration and a game‐theoretic characterization of maximum entropy inference of Topsøe and others.