Neural networks for conditional probability estimation : forecasting beyond point predictions / Dirk Husmeier.

Format:

Book

Author/Creator:

Husmeier, Dirk, 1964-

Series:

Perspectives in neural computing

Language:

English

Subjects (All):

Neural networks (Computer science).

Distribution (Probability theory)--Data processing.

Distribution (Probability theory).

Physical Description:

xxiii, 275 pages : illustrations ; 24 cm.

Place of Publication:

London ; New York : Springer, [1999]

Contents:

1.1 Conventional forecasting and Takens' embedding theorem 1

1.2 Implications of observational noise 5

1.3 Implications of dynamic noise 9

2. A Universal Approximator Network for Predicting Conditional Probability Densities 21

2.2 A single-hidden-layer network 22

2.3 An additional hidden layer 23

2.4 Regaining the conditional probability density 25

2.5 Moments of the conditional probability density 26

2.6 Interpretation of the network parameters 28

2.7 Gaussian mixture model 29

2.8 Derivative-of-sigmoid versus Gaussian mixture model 30

2.9 Comparison with other approaches 31

2.9.1 Predicting local error bars 31

2.9.2 Indirect method 31

2.9.3 Complete kernel expansion: Conditional Density Estimation Network (CDEN) and Mixture Density Network (MDN) 32

2.9.4 Distorted Probability Mixture Network (DPMN) 32

2.9.5 Mixture of Experts (ME) and Hierarchical Mixture of Experts (HME) 33

2.9.6 Soft histogram 33

2.11 Appendix: The moment generating function for the DSM network 35

3. A Maximum Likelihood Training Scheme 39

3.1 The cost function 39

3.2 A gradient-descent training scheme 43

3.2.1 Output weights 45

3.2.2 Kernel widths 47

3.2.3 Remaining weights 48

3.2.4 Interpretation of the parameter adaptation rules 49

3.2.5 Deficiencies of gradient descent and their remedy 51

4. Benchmark Problems 57

4.1 Logistic map with intrinsic noise 57

4.2 Stochastic combination of two stochastic dynamical systems 60

4.3 Brownian motion in a double-well potential 63

5. Demonstration of the Model Performance on the Benchmark Problems 69

5.2 Logistic map with intrinsic noise 71

5.2.1 Method 71

5.2.2 Results 73

5.3 Stochastic coupling between two stochastic dynamical systems 75

5.3.1 Method 75

5.3.2 Results 77

5.3.3 Auto-pruning 78

5.4 Brownian motion in a double-well potential 80

5.4.1 Method 80

5.4.2 Results 82

5.4.3 Comparison with other approaches 82

5.6 Discussion 84

6. Random Vector Functional Link (RVFL) Networks 87

6.1 The RVFL theorem 87

6.2 Proof of the RVFL theorem 89

6.3 Comparison with the multilayer perceptron 93

6.4 A simple illustration 95

7. Improved Training Scheme Combining the Expectation Maximisation (EM) Algorithm with the RVFL Approach 99

7.1 Review of the Expectation Maximisation (EM) algorithm 99

7.2 Simulation: Application of the GM network trained with the EM algorithm 104

7.2.1 Method 104

7.2.2 Results 105

7.2.3 Discussion 108

7.3 Combining EM and RVFL 109

7.4 Preventing numerical instability 112

7.5 Regularisation 117

8. Empirical Demonstration: Combining EM and RVFL 121

8.1 Method 121

8.2 Application of the GM-RVFL network to predicting the stochastic logistic-kappa map 122

8.2.1 Training a single model 122

8.2.2 Training an ensemble of models 126

8.3 Application of the GM-RVFL network to the double-well problem 129

8.3.1 Committee selection 130

8.3.2 Prediction 131

8.3.3 Comparison with other approaches 132

9. A simple Bayesian regularisation scheme 137

9.1 A Bayesian approach to regularisation 137

9.2 A simple example: repeated coin flips 139

9.3 A conjugate prior 140

9.4 EM algorithm with regularisation 142

9.5 The posterior mode 143

10. The Bayesian Evidence Scheme for Regularisation 147

10.2 A simple illustration of the evidence idea 150

10.3 Overview of the evidence scheme 152

10.3.1 First step: Gaussian approximation to the probability in parameter space 152

10.3.2 Second step: Optimising the hyperparameters 153

10.3.3 A self-consistent iteration scheme 154

10.4 Implementation of the evidence scheme 155

10.4.1 First step: Gaussian approximation to the probability in parameter space 156

10.4.2 Second step: Optimising the hyperparameters 157

10.4.3 Algorithm 159

10.5.1 Improvement over the maximum likelihood estimate 160

10.5.2 Justification of the approximations 161

10.5.3 Final remark 162

11. The Bayesian Evidence Scheme for Model Selection 165

11.1 The evidence for the model 165

11.2 An uninformative prior 168

11.3 Comparison with MacKay's work 171

11.4 Interpretation of the model evidence 172

11.4.1 Ockham factors for the weight groups 173

11.4.2 Ockham factors for the kernel widths 174

11.4.3 Ockham factor for the priors 175

12. Demonstration of the Bayesian Evidence Scheme for Regularisation 179

12.1 Method and objective 179

12.1.1 Initialisation 179

12.1.2 Different training and regularisation schemes 180

12.1.3 Pruning 181

12.2 Large Data Set 181

12.3 Small Data Set 183

12.4 Number of well-determined parameters and pruning 185

12.4.1 Automatic self-pruning 185

12.4.2 Mathematical elucidation of the pruning scheme 189

13. Network Committees and Weighting Schemes 193

13.1 Network committees for interpolation 193

13.2 Network committees for modelling conditional probability densities 196

13.3 Weighting Schemes for Predictors 198

13.3.2 A Bayesian approach 199

13.3.3 Numerical problems with the model evidence 199

13.3.4 A weighting scheme based on the cross-validation performance 201

14. Demonstration: Committees of Networks Trained with Different Regularisation Schemes 203

14.1 Method and objective 203

14.2 Single-model prediction 204

14.3 Committee prediction 207

14.3.1 Best and average single-model performance 207

14.3.2 Improvement over the average single-model performance 209

14.3.3 Improvement over the best single-model performance 210

14.3.4 Robustness of the committee performance 210

14.3.5 Dependence on the temperature 211

14.3.6 Dependence on the temperature when including biased models 212

14.3.7 Optimal temperature 213

14.3.8 Model selection and evidence 213

14.3.9 Advantage of under-regularisation and over-fitting 215

15. Automatic Relevance Determination (ARD) 221

15.2 Two alternative ARD schemes 223

15.3 Mathematical implementation 224

15.4 Empirical demonstration 227

16. A Real-World Application: The Boston Housing Data 229

16.1 A real-world regression problem: The Boston house-price data 230

16.2 Prediction with a single model 231

16.2.1 Methodology 231

16.2.2 Results 232

16.3 Test of the ARD scheme 234

16.3.1 Methodology 234

16.3.2 Results 234

16.4 Prediction with network committees 236

16.4.1 Objective 236

16.4.2 Methodology 237

16.4.3 Weighting scheme and temperature 238

16.4.4 ARD parameters 239

16.4.5 Comparison between the two ARD schemes 240

16.4.6 Number of kernels 240

16.4.7 Bayesian regularisation 241

16.4.8 Network complexity 241

16.4.9 Cross-validation 242

16.5 Discussion: How overfitting can be useful 242

16.6 Increasing diversity 244

16.6.1 Bagging 245

16.6.2 Nonlinear Preprocessing 246

16.7 Comparison with Neal's results 248

18. Appendix: Derivation of the Hessian for the Bayesian Evidence Scheme 255

18.2 A decomposition of the Hessian using EM 256

18.3 Explicit calculation of the Hessian 258.

Notes:

Includes bibliographical references (pages [267]-272) and index.

ISBN:

1852330953

OCLC:

40251635