Premium
Information Measures in Perspective
Author(s) -
Ebrahimi Nader,
Soofi Ehsan S.,
Soyer Refik
Publication year - 2010
Publication title -
international statistical review
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.051
H-Index - 54
eISSN - 1751-5823
pISSN - 0306-7734
DOI - 10.1111/j.1751-5823.2010.00105.x
Subject(s) - kullback–leibler divergence , perspective (graphical) , information theory , computer science , probability distribution , entropy (arrow of time) , bayesian probability , range (aeronautics) , principle of maximum entropy , prior probability , set (abstract data type) , mathematics , axiom , econometrics , statistics , artificial intelligence , physics , quantum mechanics , materials science , geometry , composite material , programming language
Summary Information‐theoretic methodologies are increasingly being used in various disciplines. Frequently an information measure is adapted for a problem, yet the perspective of information as the unifying notion is overlooked. We set forth this perspective through presenting information‐theoretic methodologies for a set of problems in probability and statistics. Our focal measures are Shannon entropy and Kullback–Leibler information. The background topics for these measures include notions of uncertainty and information, their axiomatic foundation, interpretations, properties, and generalizations. Topics with broad methodological applications include discrepancy between distributions, derivation of probability models, dependence between variables, and Bayesian analysis. More specific methodological topics include model selection, limiting distributions, optimal prior distribution and design of experiment, modeling duration variables, order statistics, data disclosure, and relative importance of predictors. Illustrations range from very basic to highly technical ones that draw attention to subtle points.