Neural smithing : supervised learning in feedforward artificial neural networks / Russell D. Reed and Robert J. Marks II.

Format:

Book

Author/Creator:

Reed, Russell D.

Contributor:

Marks, Robert J., II (Robert Jackson), 1950-

Language:

English

Subjects (All):

Neural networks (Computer science).

Physical Description:

viii, 346 pages : illustrations ; 24 cm

Place of Publication:

Cambridge, Mass. : The MIT Press, [1998]

Summary:

Artificial neural networks are nonlinear mapping systems whose structure is loosely based on principles observed in the nervous systems of humans and animals. The basic idea is that massive systems of simple units linked together in appropriate ways can generate many complex and interesting behaviors. This book focuses on the subset of feedforward artificial neural networks called multilayer perceptrons (MLP). These are the mostly widely used neural networks, with applications as diverse as finance (forecasting), manufacturing (process control), and science (speech and image recognition).

This book presents an extensive and practical overview of almost every aspect of MLP methodology, progressing from an initial discussion of what MLPs are and how they might be used to an in-depth examination of technical factors affecting performance. The book can be used as a tool kit by readers interested in applying networks to specific problems, yet it also presents theory and references outlining the last ten years of MLP research.

Contents:

2 Supervised Learning 7

2.1 Objective Functions 9

2.2 Alternatives and Extensions 11

3 Single-Layer Networks 15

3.1 Hyperplane Geometry 15

3.2 Linear Separability 18

3.3 Hyperplane Capacity 20

3.4 Learning Rules for Single-Layer Networks 23

3.5 Adalines and the Widrow-Hoff Learning Rule 29

4 MLP Representational Capabilities 31

4.1 Representational Capability 31

4.2 Universal Approximation Capabilities 35

4.3 Size versus Depth 38

4.4 Capacity versus Size 41

5 Back-Propagation 49

5.1 Preliminaries 50

5.2 Back-Propagation: The Derivative Calculation 53

5.3 Back-Propagation: The Weight Update Algorithm 57

5.4 Common Modifications 62

5.5 Pseudocode Examples 63

5.6 Remarks 66

5.7 Training Time 67

6 Learning Rate and Momentum 71

6.1 Learning Rate 71

6.2 Momentum 85

6.3 Remarks 95

7 Weight-Initialization Techniques 97

7.1 Random Initialization 97

7.2 Nonrandom Initialization 105

8 The Error Surface 113

8.1 Characteristic Features 113

8.2 The Gradient is the Sum of Single-Pattern Gradients 117

8.3 Weight-Space Symmetries 118

8.4 Remarks 120

8.5 Local Minima 121

8.6 Properties of the Hessian Matrix 127

8.7 Gain Scaling 132

9 Faster Variations of Back-Propagation 135

9.1 Adaptive Learning Rate Methods 135

9.2 Vogl's Method (Bold Driver) 136

9.3 Delta-Bar-Delta 137

9.4 Silva and Almeida 140

9.5 SuperSAB 142

9.6 Rprop 142

9.7 Quickprop 145

9.8 Search Then Converge 147

9.9 Fuzzy Control of Back-Propagation 148

9.10 Other Heuristics 150

9.11 Remarks 151

9.12 Other Notes 153

10 Classical Optimization Techniques 155

10.1 The Objective Function 155

10.2 Factors Affecting the Choice of a Method 156

10.3 Line Search 158

10.4 Evaluation-Only Methods 159

10.5 First-Order Gradient Methods 163

10.6 Second-Order Gradient Methods 169

10.7 Stochastic Evaluation-Only Methods 175

10.8 Discussion 179

11 Genetic Algorithms and Neural Networks 185

11.1 The Basic Algorithm 186

11.2 Example 189

11.3 Application to Neural Network Design 191

11.4 Remarks 194

12 Constructive Methods 197

12.1 Dynamic Node Creation 199

12.2 Cascade-Correlation 201

12.3 The Upstart Algorithm 204

12.4 The Tiling Algorithm 206

12.5 Marchand's Algorithm 209

12.6 Meiosis Networks 212

12.7 Principal Components Node Splitting 213

12.8 Construction from a Voronoi Diagram 215

12.9 Other Algorithms 217

13 Pruning Algorithms 219

13.1 Pruning Algorithms 220

13.2 Sensitivity Calculation Methods 221

13.3 Penalty-Term Methods 226

13.4 Other Methods 232

13.5 Discussion 235

14 Factors Influencing Generalization 239

14.1 Definitions 239

14.2 The Need for Additional Information 240

14.3 Network Complexity versus Target Complexity 241

14.4 The Training Data 242

14.5 The Learning Algorithm 249

14.6 Other Factors 253

15 Generalization Prediction and Assessment 257

15.1 Cross-Validation 257

15.2 The Bayesian Approach 258

15.3 Akaike's Final Prediction Error 260

15.4 PAC Learning and the VC Dimension 261

16 Heuristics for Improving Generalization 265

16.1 Early Stopping 265

16.2 Regularization 266

16.3 Pruning Methods 268

16.4 Constructive Methods 268

16.5 Weight Decay 269

16.6 Information Minimization 271

16.7 Replicated Networks 272

16.8 Training with Noisy Data 273

16.9 Use of Domain-Dependent Prior Information 274

16.10 Hint Functions 275

16.11 Knowledge-Based Neural Nets 275

16.12 Physical Models to Generate Additional Data 276

17 Effects of Training with Noisy Inputs 277

17.1 Convolution Property of Training with Jitter 277

17.2 Error Regularization and Training with Jitter 281

17.3 Training with Jitter and Sigmoid Scaling 283

17.4 Extension to General Layered Neural Networks 288

17.5 Remarks 289

17.6 Further Examples 290

A Linear Regression 293

A.1 Newton's Method 294

A.2 Gradient Descent 295

A.3 The LMS Algorithm 298

B Principal Components Analysis 299

B.1 Autoencoder Networks and Principal Components 303

B.2 Discriminant Analysis Projections 306

C Jitter Calculations 311

C.1 Jitter: Small-Perturbation Approximation 311

C.2 Jitter: CDF-PDF Convolution in n Dimensions 311

C.3 Jitter: CDF-PDF Convolution in One Dimension 314

D Sigmoid-like Nonlinear Functions 315.

Notes:

"A Bradford book."

Includes bibliographical references (pages [319]-338) and index.

ISBN:

0262181908

OCLC:

38468402