Information and complexity in statistical modeling / Jorma Rissanen.

Format:

Book

Author/Creator:

Rissanen, Jorma.

Series:

Information science and statistics

Language:

English

Subjects (All):

Statistics.

Physical Description:

xii., 142 pages : illustrations ; 17 cm.

Place of Publication:

New York : Springer, [2007]

Summary:

No statistical model is "true" or "false," "right" or "wrong"; the models just have varying performance, which can be assessed. The main theme in this book is to teach modeling based on the principle that the objective is to extract the information from data that can be learned with suggested classes of probability models. The intuitive and fundamental concepts of complexity, learnable information, and noise are formalized, which provides a firm information theoretic foundation for statistical modeling. Inspired by Kolmogorov's structure function in the algorithmic theory of complexity, this is accomplished by finding the shortest code length, called the stochastic complexity, with which the data can be encoded when advantage is taken of the models in a suggested class, which amounts to the MDL (Minimum Description Length) principle. The complexity, in turn, breaks up into the shortest code length for the optimal model in a set of models that can be optimally distinguished from the given data and the rest, which defines "noise" as the incompressible part in the data without useful information.

Contents:

Part I Information and Coding

2 Shannon-Wiener Information 9

2.1 Coding of Random Variables 9

2.1.1 Universal prior for integers and their coding 14

2.2 Basic Properties of Entropy and Related Quantities 17

2.3 Channel Capacity 19

2.4 Chain Rules 21

2.5 Jensen's Inequality 22

2.6 Theory of Types 23

2.7 Equipartition Property 25

3 Coding of Random Processes 27

3.1 Random Processes 27

3.2 Entropy of Stationary Processes 28

3.3 Markov Processes 29

3.4 Tree Machines 30

3.5 Tunstall's Algorithm 32

3.6 Arithmetic Codes 34

3.7 Universal Coding 38

3.7.1 Lempel-Ziv and Ma algorithms 39

Part II Statistical Modeling

4 Kolmogorov Complexity 47

4.1 Elements of Recursive Function Theory 48

4.2 Complexities 51

4.3 Kolmogorov's Structure Function 52

5 Stochastic Complexity 57

5.1 Model Classes 57

5.1.1 Exponential family 58

5.1.2 Maximum entropy family with simple loss functions 59

5.2 Universal Models 60

5.2.1 Mixture models 61

5.2.2 Normalized maximum-likelihood model 63

5.2.3 A predictive universal model 67

5.2.4 Conditional NML model 71

5.3 Strong Optimality 73

5.3.1 Prediction bound for [alpha]-loss functions 75

6 Structure Function 79

6.1 Partition of Parameter Space 79

6.2 Model Complexity 81

6.3 Sets of Typical Sequences 83

7 Optimally Distinguishable Models 89

7.1 Special Cases 92

7.1.1 Bernoulli class 92

7.1.2 Normal distributions 93

8 The MDL Principle 97

9 Applications 103

9.1 Hypothesis Testing 103

9.2 Universal Tree Machine and Variable-Order Markov Chain 106

9.2.1 Extension to time series 111

9.3 Linear Regression 113

9.3.1 Orthonormal regression matrix 118

9.4 MDL Denoising 120

9.4.1 Linear-quadratic denoising 122

9.4.2 Histogram denoising 123

9.5 AR and ARMA Models 126

9.5.1 AR models 127

9.5.2 ARMA models 128

9.6 Logit Regression 129.

Notes:

Includes bibliographical references (pages [135]-139) and index.

ISBN:

0387366105

9780387366104

9780387688121

0387688129

OCLC:

122308853

Publisher Number:

99945215479

Online:

Publisher description