Adaptive blind signal and image processing : learning algorithms and applications / Andrzej Cichocki, Shun-ichi Amari.

Format:

Book

Author/Creator:

Cichocki, Andrzej.

Contributor:

Amari, Shun'ichi.

Alumni and Friends Memorial Book Fund.

Language:

English

Subjects (All):

Adaptive signal processing.

Machine learning.

Algorithms.

Physical Description:

xxxi, 554 pages : illustrations ; 25 cm + 1 CD-ROM (color ; 4 3/4 in.)

Place of Publication:

Chichester ; New York : J. Wiley, [2002]

System Details:

text file

Summary:

Accompanying CD-ROM ... "includes an electronic, interactive version of the book with hyperlinks, full-colour figures, and text." -- p. [4] of cover.

Contents:

1 Introduction to Blind Signal Processing: Problems and Applications 1

1.1 Problem Formulations

An Overview 2

1.1.1 Generalized Blind Signal Processing Problem 2

1.1.2 Instantaneous Blind Source Separation and Independent Component Analysis 5

1.1.3 Independent Component Analysis for Noisy Data 11

1.1.4 Multichannel Blind Deconvolution and Separation 14

1.1.5 Blind Extraction of Signals 18

1.1.6 Generalized Multichannel Blind Deconvolution

State Space Models 19

1.1.7 Nonlinear State Space Models

Semi-Blind Signal Processing 21

1.1.8 Why State Space Demixing Models? 22

1.2 Potential Applications of Blind and Semi-Blind Signal Processing 23

1.2.1 Biomedical Signal Processing 24

1.2.2 Blind Separation of Electrocardiographic Signals of Fetus and Mother 25

1.2.3 Enhancement and Decomposition of EMG Signals 27

1.2.4 EEG and Data MEG Processing 27

1.2.5 Application of ICA/BSS for Noise and Interference Cancellation in Multi-sensory Biomedical Signals 29

1.2.6 Cocktail Party Problem 34

1.2.7 Digital Communication Systems 35

1.2.7.1 Why Blind? 37

1.2.8 Image Restoration and Understanding 37

2 Solving a System of Algebraic Equations and Related Problems 43

2.1 Formulation of the Problem for Systems of Linear Equations 44

2.2 Least-Squares Problems 45

2.2.1 Basic Features of the Least-Squares Solution 45

2.2.2 Weighted Least-Squares and Best Linear Unbiased Estimation 47

2.2.3 Basic Network Structure-Least-Squares Criteria 49

2.2.4 Iterative Parallel Algorithms for Large and Sparse Systems 49

2.2.5 Iterative Algorithms with Non-negativity Constraints 51

2.2.6 Robust Circuit Structure by Using the Interactively Reweighted Least-Squares Criteria 54

2.2.7 Tikhonov Regularization and SVD 57

2.3 Least Absolute Deviation (1-norm) Solution of Systems of Linear Equations 61

2.3.1 Neural Network Architectures Using a Smooth Approximation and Regularization 62

2.3.2 Neural Network Model for LAD Problem Exploiting Inhibition Principles 64

2.4 Total Least-Squares and Data Least-Squares Problems 67

2.4.1 Problems Formulation 67

2.4.1.1 A Historical Overview of the TLS Problem 67

2.4.2 Total Least-Squares Estimation 69

2.4.3 Adaptive Generalized Total Least-Squares 73

2.4.4 Extended TLS for Correlated Noise Statistics 75

2.4.4.1 Choice of R[subscript NN] in Some Practical Situations 77

2.4.5 Adaptive Extended Total Least-Squares 77

2.4.6 An Illustrative Example - Fitting a Straight Line to a Set of Points 78

2.5 Sparse Signal Representation and Minimum Fuel Consumption Problem 79

2.5.1 Approximate Solution of Minimum Fuel Problem Using Iterative LS Approach 81

2.5.2 FOCUSS Algorithms 83

3 Principal/Minor Component Analysis and Related Problems 87

3.2 Basic Properties of PCA 88

3.2.1 Eigenvalue Decomposition 88

3.2.2 Estimation of Sample Covariance Matrices 90

3.2.3 Signal and Noise Subspaces - AIC and MDL Criteria for their Estimation 91

3.2.4 Basic Properties of PCA 93

3.3 Extraction of Principal Components 94

3.4 Basic Cost Functions and Adaptive Algorithms for PCA 98

3.4.1 The Rayleigh Quotient

Basic Properties 98

3.4.2 Basic Cost Functions for Computing Principal and Minor Components 99

3.4.3 Fast PCA Algorithm Based on the Power Method 101

3.4.4 Inverse Power Iteration Method 104

3.5 Robust PCA 104

3.6 Adaptive Learning Algorithms for MCA 107

3.7 Unified Parallel Algorithms for PCA/MCA and PSA/MSA 110

3.7.1 Cost Function for Parallel Processing 111

3.7.2 Gradient of J(W) 112

3.7.3 Stability Analysis 113

3.7.4 Unified Stable Algorithms 116

3.8 SVD in Relation to PCA and Matrix Subspaces 118

3.9 Multistage PCA for BSS 119

Appendix A. Basic Neural Networks Algorithms for Real and Complex-Valued PCA 122

Appendix B. Hierarchical Neural Network for Complex-valued PCA 125

4 Blind Decorrelation and SOS for Robust Blind Identification 129

4.1 Spatial Decorrelation - Whitening Transforms 130

4.1.1 Batch Approach 130

4.1.2 Optimization Criteria for Adaptive Blind Spatial Decorrelation 132

4.1.3 Derivation of Equivariant Adaptive Algorithms for Blind Spatial Decorrelation 133

4.1.4 Simple Local Learning Rule 136

4.1.5 Gram-Schmidt Orthogonalization 138

4.1.6 Blind Separation of Decorrelated Sources Versus Spatial Decorrelation 139

4.1.7 Bias Removal for Noisy Data 139

4.1.8 Robust Prewhitening - Batch Algorithm 140

4.2 SOS Blind Identification Based on EVD 141

4.2.1 Mixing Model 141

4.2.2 Basic Principles: SD and EVD 143

4.3 Improved Blind Identification Algorithms Based on EVD/SVD 148

4.3.1 Robust Orthogonalization of Mixing Matrices for Colored Sources 148

4.3.2 Improved Algorithm Based on GEVD 153

4.3.3 Improved Two-stage Symmetric EVD/SVD Algorithm 155

4.3.4 BSS and Identification Using Bandpass Filters 156

4.4 Joint Diagonalization - Robust SOBI Algorithms 157

4.4.1 Modified SOBI Algorithm for Nonstationary Sources: SONS Algorithm 160

4.4.2 Computer Simulation Experiments 161

4.4.3 Extensions of Joint Approximate Diagonalization Technique 162

4.4.4 Comparison of the JAD and Symmetric EVD 163

4.5 Cancellation of Correlation 164

4.5.1 Standard Estimation of Mixing Matrix and Noise Covariance Matrix 164

4.5.2 Blind Identification of Mixing Matrix Using the Concept of Cancellation of Correlation 165

Appendix A. Stability of the Amari's Natural Gradient and the Atick-Redlich Formula 168

Appendix B. Gradient Descent Learning Algorithms with Invariant Frobenius Norm of the Separating Matrix 171

Appendix C. JADE Algorithm 173

5 Sequential Blind Signal Extraction 177

5.1 Introduction and Problem Formulation 178

5.2 Learning Algorithms Based on Kurtosis as Cost Function 180

5.2.1 A Cascade Neural Network for Blind Extraction of Non-Gaussian Sources with Learning Rule Based on Normalized Kurtosis 181

5.2.2 Algorithms Based on Optimization of Generalized Kurtosis 184

5.2.3 KuicNet Learning Algorithm 186

5.2.4 Fixed-point Algorithms 187

5.2.5 Sequential Extraction and Deflation Procedure 191

5.3 On Line Algorithms for Blind Signal Extraction of Temporally Correlated Sources 193

5.3.1 On Line Algorithms for Blind Extraction Using Linear Predictor 195

5.3.2 Neural Network for Multi-unit Blind Extraction 197

5.4 Batch Algorithms for Blind Extraction of Temporally Correlated Sources 199

5.4.1 Blind Extraction Using a First Order Linear Predictor 201

5.4.2 Blind Extraction of Sources Using Bank of Adaptive Bandpass Filters 202

5.4.3 Blind Extraction of Desired Sources Correlated with Reference Signals 205

5.5 Statistical Approach to Sequential Extraction of Independent Sources 206

5.5.1 Log Likelihood and Cost Function 206

5.5.2 Learning Dynamics 208

5.5.3 Equilibrium of Dynamics 209

5.5.4 Stability of Learning Dynamics and Newton's Method 210

5.6 Statistical Approach to Temporally Correlated Sources 212

5.7 On-line Sequential Extraction of Convolved and Mixed Sources 214

5.7.1 Formulation of the Problem 214

5.7.2 Extraction of Single i.i.d. Source Signal 215

5.7.3 Extraction of Multiple i.i.d.

Sources 217

5.7.4 Extraction of Colored Sources from Convolutive Mixture 218

5.8 Computer Simulations: Illustrative Examples 219

5.8.1 Extraction of Colored Gaussian Signals 219

5.8.2 Extraction of Natural Speech Signals from Colored Gaussian Signals 221

5.8.3 Extraction of Colored and White Sources 222

5.8.4 Extraction of Natural Image Signal from Interferences 223

Appendix A. Global Convergence of Algorithms for Blind Source Extraction Based on Kurtosis 225

Appendix B. Analysis of Extraction and Deflation Procedure 227

Appendix C. Conditions for Extraction of Sources Using Linear Predictor Approach 228

6 Natural Gradient Approach to Independent Component Analysis 231

6.1 Basic Natural Gradient Algorithms 232

6.1.1 Kullback-Leibler Divergence - Relative Entropy as Measure of Stochastic Independence 232

6.1.2 Derivation of Natural Gradient Basic Learning Rules 235

6.2 Generalizations of Basic Natural Gradient Algorithm 237

6.2.1 Nonholonomic Learning Rules 237

6.2.2 Natural Riemannian Gradient in Orthogonality Constraint 239

6.2.2.1 Local Stability Analysis 240

6.3 NG Algorithms for Blind Extraction 242

6.3.1 Stiefel Manifolds Approach 242

6.4 Generalized Gaussian Distribution Model 243

6.4.1 The Moments of the Generalized Gaussian Distribution 248

6.4.2 Kurtosis and Gaussian Exponent 249

6.4.3 The Flexible ICA Algorithm 250

6.4.4 Pearson Model 253

6.5 Natural Gradient Algorithms for Non-stationary Sources 254

6.5.1 Model Assumptions 254

6.5.2 Second Order Statistics Cost Function 255

6.5.3 Derivation of NG Learning Algorithms 255

Appendix A. Derivation of Local Stability Conditions for NG ICA Algorithm (6.19) 258

Appendix B. Derivation of the Learning Rule (6.32) and Stability Conditions for ICA 260

Appendix C. Stability of Generalized Adaptive Learning Algorithm 262

Appendix D. Dynamic Properties and Stability of Nonholonomic NG Algorithms 264

Appendix E. Summary of Stability Conditions 267

Appendix F. Natural Gradient for Non-square Separating Matrix 268

Appendix G. Lie Groups and Natural Gradient for General Case 269

G.0.1 Lie Group Gl(n, m) 270

G.0.2 Derivation of Natural Learning Algorithm for m > n 271

7 Locally Adaptive Algorithms for ICA and their Implementations 273

7.1 Modified Jutten-Herault Algorithms for Blind Separation of Sources 274

7.1.1 Recurrent Neural Network 274

7.1.2 Statistical Independence 274

7.1.3 Self-normalization 277

7.1.4 Feed-forward Neural Network and Associated Learning Algorithms 278

7.1.5 Multilayer Neural Networks 282

7.2 Iterative Matrix Inversion Approach to Derivation of Family of Robust ICA Algorithms 285

7.2.1 Derivation of Robust ICA Algorithm Using Generalized Natural Gradient Approach 288

7.2.2 Practical Implementation of the Algorithms 289

7.2.3 Special Forms of the Flexible Robust Algorithm 291

7.2.4 Decorrelation Algorithm 291

7.2.5 Natural Gradient Algorithms 291

7.2.6 Generalized EASI Algorithm 291

7.2.7 Non-linear PCA Algorithm 292

7.2.8 Flexible ICA Algorithm for Unknown Number of Sources and their Statistics 293

7.3 Computer Simulations 294

Appendix A. Stability Conditions for the Robust ICA Algorithm (7.50) [332] 300

8 Robust Techniques for BSS and ICA with Noisy Data 305

8.2 Bias Removal Techniques for Prewhitening and ICA Algorithms 306

8.2.1 Bias Removal for Whitening Algorithms 306

8.2.2 Bias Removal for Adaptive ICA Algorithms 307

8.3 Blind Separation of Signals Buried in Additive Convolutive Reference Noise 310

8.3.1 Learning Algorithms for Noise Cancellation 311

8.4 Cumulants Based Adaptive ICA Algorithms 314

8.4.1 Cumulants Based Cost Functions 314

8.4.2 Family of Equivariant Algorithms Employing the Higher Order Cumulants 315

8.4.3 Possible Extensions 317

8.4.4 Cumulants for Complex Valued Signals 318

8.4.5 Blind Separation with More Sensors than Sources 318

8.5 Robust Extraction of Arbitrary Group of Source Signals 320

8.5.1 Blind Extraction of Sparse Sources with Largest Positive Kurtosis Using Prewhitening and Semi-Orthogonality Constraint 320

8.5.2 Blind Extraction of an Arbitrary Group of Sources without Prewhitening 323

8.6 Recurrent Neural Network Approach for Noise Cancellation 325

8.6.1 Basic Concept and Algorithm Derivation 325

8.6.2 Simultaneous Estimation of a Mixing Matrix and Noise Reduction 328

8.6.2.1 Regularization 329

8.6.3 Robust Prewhitening and Principal Component Analysis (PCA) 331

8.6.4 Computer Simulation Experiments for Amari-Hopfield Network 331

Appendix A. Cumulants in Terms of Moments 333

9 Multichannel Blind Deconvolution: Natural Gradient Approach 335

9.1 SIMO Convolutive Models and Learning Algorithms for Estimation of Source Signal 336

9.1.1 Equalization Criteria for SIMO Systems 338

9.1.2 SIMO Blind Identification and Equalization via Robust ICA/BSS 340

9.1.3 Feed-forward Deconvolution Model and Natural Gradient Learning Algorithm 342

9.1.4 Recurrent Neural Network Model and Hebbian Learning Algorithm 343

9.2 Multichannel Blind Deconvolution with Constraints Imposed on FIR Filters 346

9.3 General Models for Multiple-Input Multiple-Output Blind Deconvolution 349

9.3.1 Fundamental Models and Assumptions 349

9.3.2 Separation-Deconvolution Criteria 351

9.4 Relationships Between BSS/ICA and MBD 354

9.4.1 Multichannel Blind Deconvolution in the Frequency Domain 354

9.4.2 Algebraic Equivalence of Various Approaches 355

9.4.3 Convolution as Multiplicative Operator 357

9.4.4 Natural Gradient Learning Rules for Multichannel Blind Deconvolution (MBD) 358

9.4.5 NG Algorithms for Double Infinite Filters 359

9.4.6 Implementation of Algorithms for Minimum Phase Non-causal System 360

9.4.6.1 Batch Update Rules 360

9.4.6.2 On-line Update Rule 360

9.4.6.3 Block On-line Update Rule 360

9.5 Natural Gradient Algorithms with Nonholonomic Constraints 362

9.5.1 Equivariant Learning Algorithm for Causal FIR Filters in the Lie Group Sense 363

9.5.2 Natural Gradient Algorithm for Fully Recurrent Network 366

9.6 MBD of Non-minimum Phase System Using Filter Decomposition Approach 368

9.6.1 Information Back-propagation 369

9.6.2 Batch Natural Gradient Learning Algorithm 371

9.7 Computer Simulations Experiments 372

9.7.1 The Natural Gradient Algorithm vs. the Ordinary Gradient Algorithm 372

9.7.2 Information Back-propagation 374

Appendix A. Lie Group and Riemannian Metric on FIR Manifold 376

A.0.1 Lie Group 376

A.0.2 Riemannian Metric and Natural Gradient in the Lie Group Sense 379

Appendix B. Properties and Stability Conditions for the Equivariant Algorithm 381

B.0.1 Proof of Fundamental Properties and Stability Analysis of Equivariant NG Algorithm (9.126) 381

B.0.2 Stability Analysis of the Learning Algorithm 381

10 Estimating Functions and Superefficiency for ICA and Deconvolution 383

10.1 Estimating Functions for Standard ICA 384

10.1.1 What is Estimating Function? 384

10.1.2 Semiparametric Statistical Model 385

10.1.3 Admissible Class of Estimating Functions 386

10.1.4 Stability of Estimating Functions 389

10.1.5 Standardized Estimating Function and Adaptive Newton Method 392

10.1.6 Analysis of Estimation Error and Superefficiency 393

10.1.7 Adaptive Choice of [phi] Function 395

10.2 Estimating Functions in Noisy Case 396

10.3 Estimating Functions for Temporally Correlated Source Signals 397

10.3.1 Source Model 397

10.3.2 Likelihood and Score Functions 399

10.3.3 Estimating Functions 400

10.3.4 Simultaneous and Joint Diagonalization of Covariance Matrices and Estimating Functions 401

10.3.5 Standardized Estimating Function and Newton Method 404

10.3.6 Asymptotic Errors 407

10.4 Semiparametric Models for Multichannel Blind Deconvolution 407

10.4.1 Notation and Problem Statement 408

10.4.2 Geometrical Structures on FIR Manifold 409

10.4.3 Lie Group 410

10.4.4 Natural Gradient Approach for Multichannel Blind Deconvolution 410

10.4.5 Efficient Score Matrix Function and its Representation 413

10.5 Estimating Functions for MBD 415

10.5.1 Superefficiency of Batch Estimator 418

Appendix A. Representation of Operator K(z) 419

11 Blind Filtering and Separation Using a State-Space Approach 423

11.1 Problem Formulation and Basic Models 424

11.1.1 Invertibility by State Space Model 427

11.1.2 Controller Canonical Form 428

11.2 Derivation of Basic Learning Algorithms 428

11.2.1 Gradient Descent Algorithms for Estimation of Output Matrices W = [C, D] 429

11.2.2 Special Case-Multichannel Blind Deconvolution with Causal FIR Filters 432

11.2.3 Derivation of the Natural Gradient Algorithm for State Space Model 432

11.3 Estimation of Matrices [A,B] by Information Back

propagation 434

11.4 State Estimator

The Kalman Filter 437

11.4.1 Kalman Filter 437

11.5 Two

stage Separation Algorithm 439

Appendix A. Derivation of the Cost Function 440

12 Nonlinear State Space Models

Semi-Blind Signal Processing 443

12.1 General Formulation of The Problem 443

12.1.1 Invertibility by State Space Model 447

12.1.2 Internal Representation 447

12.2 Supervised-Unsupervised Learning

Approach 448

12.2.1 Nonlinear Autoregressive Moving Average Model 448

12.2.2 Hyper Radial Basis Function Neural Network Model 449

12.2.3 Estimation of Parameters of HRBF Networks Using Gradient Approach 451

Appendix A Mathematical Preliminaries 536

A.1 Matrix Analysis 536

A.1.1 Matrix inverse update rules 536

A.1.2 Some properties of determinant 537

A.1.3 Some properties of the Moore-Penrose pseudo-inverse 537

A.1.4 Matrix Expectations 538

A.1.5 Differentiation of a scalar function with respect to a vector 539

A.1.6 Matrix differentiation 540

A.1.7 Trace 541

A.1.8 Matrix differentiation of trace of matrices 542

A.1.9 Important Inequalities 543

A.2 Distance measures 545

A.2.1 Geometric distance measures 545

A.2.2 Distances between sets 545

A.2.3 Discrimination measures 546.

Notes:

Includes bibliographical references (pages 453-534) and index.

Local Notes:

Acquired for the Penn Libraries with assistance from the Alumni and Friends Memorial Book Fund.

ISBN:

0471607916

OCLC:

249362455