Geoffrey J. McLachlan, Thriyambakam Krishnan9780470191606, 9780471201700, 0471201707
Complete with updates that capture developments from the past decade, The EM Algorithm and Extensions, Second Edition successfully provides a basic understanding of the EM algorithm by describing its inception, implementation, and applicability in numerous statistical contexts. In conjunction with the fundamentals of the topic, the authors discuss convergence issues and computation of standard errors, and, in addition, unveil many parallels and connections between the EM algorithm and Markov chain Monte Carlo algorithms. Thorough discussions on the complexities and drawbacks that arise from the basic EM algorithm, such as slow convergence and lack of an in-built procedure to compute the covariance matrix of parameter estimates, are also presented.
While the general philosophy of the First Edition has been maintained, this timely new edition has been updated, revised, and expanded to include:
New chapters on Monte Carlo versions of the EM algorithm and generalizations of the EM algorithm
New results on convergence, including convergence of the EM algorithm in constrained parameter spaces
Expanded discussion of standard error computation methods, such as methods for categorical data and methods based on numerical differentiation
Coverage of the interval EM, which locates all stationary points in a designated region of the parameter space
Exploration of the EM algorithm’s relationship with the Gibbs sampler and other Markov chain Monte Carlo methods
Plentiful pedagogical elements—chapter introductions, lists of examples, author and subject indices, computer-drawn graphics, and a related Web site
The EM Algorithm and Extensions, Second Edition serves as an excellent text for graduate-level statistics students and is also a comprehensive resource for theoreticians, practitioners, and researchers in the social and physical sciences who would like to extend their knowledge of the EM algorithm.
Table of contents :
0471201707……Page 1
CONTENTS……Page 10
PREFACE TO THE SECOND EDITION……Page 22
PREFACE TO THE FIRST EDITION……Page 24
LIST OF EXAMPLES……Page 28
1.1 Introduction……Page 32
1.2 Maximum Likelihood Estimation……Page 34
1.3.2 Newton-Raphson Method……Page 36
1.3.4 Modified Newton Methods……Page 37
1.4.2 Example 1.1: A Multinomial Example……Page 39
1.4.3 Example 1.2: Estimation of Mixing Proportions……Page 44
1.5.1 EM Algorithm……Page 49
1.5.2 Example 1.3: Censored Exponentially Distributed Survival Times……Page 51
1.5.3 E- and M-Steps for the Regular Exponential Family……Page 53
1.5.4 Example 1.4: Censored Exponentially Distributed Survival Times (Example 1.3 Continued)……Page 54
1.5.6 GEM Algorithm Based on One Newton-Raphson Step……Page 55
1.5.7 EM Gradient Algorithm……Page 56
1.6.1 Maximum a Posteriori Estimation……Page 57
1.6.3 Maximum Penalized Estimation……Page 58
1.7 Brief Summary of the Properties of the EM Algorithm……Page 59
1.8.2 Work Before Dempster, Laird, and Rubin (1977)……Page 60
1.8.3 EM Examples and Applications Since Dempster, Laird, and Rubin (1977)……Page 62
1.8.4 Two Interpretations of EM……Page 63
1.8.5 Developments in EM Theory, Methodology, and Applications……Page 64
1.9 Overview of the Book……Page 67
1.10 Notations……Page 68
2.1 Introduction……Page 72
2.2.1 Example 2.1: Bivariate Normal Data with Missing Values……Page 73
2.2.3 Multivariate Data: Buck’s Method……Page 76
2.3.2 Example 2.2: Linear Regression with Missing Dependent Values……Page 78
2.3.4 Healy–Westmacott Procedure as an EM Algorithm……Page 80
2.4 Example 2.4: Multinomial with Complex Cell Structure……Page 82
2.5 Example 2.5: Analysis of PET and SPECT Data……Page 85
2.6.1 ML Estimation of Multivariate t-Distribution……Page 89
2.7.1 Example 2.7: Univariate Component Densities……Page 92
2.7.2 Example 2.8: Multivariate Component Densities……Page 95
2.7.3 Numerical Example: Red Blood Cell Volume Data……Page 96
2.8.2 Specification of Complete Data……Page 97
2.8.3 E-Step……Page 100
2.8.5 Confirmation of Incomplete-Data Score Statistic……Page 101
2.8.6 M-Step for Grouped Normal Data……Page 102
2.8.7 Numerical Example: Grouped Log Normal Data……Page 103
2.9 Example 2.10: A Hidden Markov AR(1) model……Page 104
3.1 Introduction……Page 108
3.2 Monotonicity of the EM Algorithm……Page 109
3.4.1 Introduction……Page 110
3.4.2 Regularity Conditions of Wu (1983)……Page 111
3.4.3 Main Convergence Theorem for a Generalized EM Sequence……Page 112
3.4.4 A Convergence Theorem for an EM Sequence……Page 113
3.5.2 Two Convergence Theorems of Wu (1983)……Page 114
3.5.4 Constrained Parameter Spaces……Page 115
3.6.1 Example 3.1: Convergence to a Saddle Point……Page 116
3.6.2 Example 3.2: Convergence to a Local Minimum……Page 119
3.6.3 Example 3.3: Nonconvergence of a Generalized EM Sequence……Page 121
3.6.4 Example 3.4: Some E-Step Pathologies……Page 124
3.8.1 Missing Information Principle……Page 126
3.8.2 Example 3.5: Censored Exponentially Distributed Survival Times (Example 1.3 Continued)……Page 127
3.9.1 Rate Matrix for Linear Convergence……Page 130
3.9.2 Measuring the Linear Rate of Convergence……Page 131
3.9.3 Rate Matrix in Terms of Information Matrices……Page 132
3.9.5 Derivation of Rate Matrix in Terms of Information Matrices……Page 133
3.9.6 Example 3.6: Censored Exponentially Distributed Survival Times (Example 1.3 Continued)……Page 134
4.1 Introduction……Page 136
4.2.2 Extraction of Observed Information Matrix in Terms of the Complete-Data Log Likelihood……Page 137
4.2.5 Examples……Page 139
4.3 Approximations to Observed Information Matrix: i.i.d. Case……Page 145
4.4.1 Approximation Based on Empirical Information……Page 147
4.4.2 Example 4.3: Grouped Data from an Exponential Distribution……Page 148
4.5.1 Definition……Page 151
4.5.2 Calculation of J(Ψ) via Numerical Differentiation……Page 153
4.5.3 Stability……Page 154
4.5.5 Difficulties of the SEM Algorithm……Page 155
4.5.6 Example 4.4: Univariate Contaminated Normal Data……Page 156
4.5.7 Example 4.5: Bivariate Normal Data with Missing Values……Page 159
4.6 Bootstrap Approach to Standard Error Approximation……Page 161
4.7.1 Baker’s Method for Standard Error Computation……Page 162
4.7.2 Louis’ Method of Standard Error Computation……Page 163
4.7.3 Oakes’ Formula for Standard Error Computation……Page 164
4.7.5 Example 4.7: Louis’ Method for Example 2.4……Page 165
4.7.6 Baker’s Method for Standard Error for Categorical Data……Page 166
4.7.7 Example 4.8: Baker’s Method for Example 2.4……Page 167
4.8.2 Louis’ Method……Page 168
4.8.3 Example 4.9: Multinomial Data……Page 169
4.8.4 Example 4.10: Geometric Mixture……Page 170
4.9 An Aitken Acceleration-Based Stopping Criterion……Page 173
4.10.2 A Generalized Conjugate Gradient Algorithm……Page 175
4.10.3 Accelerating the EM Algorithm……Page 176
4.11.2 Combined EM and Modified Newton-Raphson Algorithm……Page 177
4.12.1 Derivation of a Condition to be a Generalized EM Sequence……Page 179
4.13 EM Gradient Algorithm……Page 180
4.14.1 The Method……Page 182
4.14.2 Example 4.12: Dirichlet Distribution……Page 184
4.15 Ikeda Acceleration……Page 188
5.1 Introduction……Page 190
5.2.2 Formal Definition……Page 191
5.2.4 Speed of Convergence……Page 193
5.2.5 Convergence Rates of EM and ECM……Page 194
5.2.7 Discussion……Page 195
5.3 Multicycle ECM Algorithm……Page 196
5.4.2 Application of ECM Algorithm……Page 197
5.5.1 Competing Risks in Survival Analysis……Page 199
5.5.3 Observed Data……Page 200
5.5.4 Application of EM Algorithm……Page 201
5.5.5 M-Step for Gompertz Components……Page 202
5.5.6 Application of a Multicycle ECM Algorithm……Page 203
5.5.7 Other Examples of EM Algorithm in Survival Analysis……Page 204
5.6 Example 5.4: Contingency Tables with Incomplete Data……Page 205
5.7 ECME Algorithm……Page 206
5.8.1 Application of the EM Algorithm……Page 207
5.8.3 Application of ECM Algorithm……Page 208
5.8.5 Some Standard Results……Page 209
5.8.6 Missing Data……Page 210
5.8.8 Theoretical Results on the Rate of Convergence……Page 212
5.9.1 A Variance Components Model……Page 213
5.9.2 E-Step……Page 214
5.9.3 M-Step……Page 215
5.9.5 Numerical Example……Page 216
5.10.1 Introduction……Page 217
5.10.2 General Form of Linear Mixed Model……Page 218
5.10.4 Example 5.7: REML Estimation in a Hierarchical Random Effects Model……Page 219
5.10.6 Generalized Linear Mixed Models……Page 222
5.11.1 EM Algorithm for Factor Analysis……Page 224
5.11.4 EM Algorithm in Principal Component Analysis……Page 227
5.12.2 Maximum Likelihood Estimation of t-Distribution……Page 229
5.13 Alternating ECM Algorithm……Page 233
5.14 Example 5.9: Mixtures of Factor Analyzers……Page 235
5.14.2 E-step……Page 236
5.14.3 CM-steps……Page 237
5.14.4 t-Component Factor Analyzers……Page 238
5.14.5 E-step……Page 241
5.14.6 CM-steps……Page 242
5.15 Parameter-Expanded EM (PX-EM) Algorithm……Page 243
5.17 One-Step-Late Algorithm……Page 244
5.18.1 Penalized EM Algorithm……Page 245
5.18.3 Example 5.9: Variance of MPLE for the Multinomial (Examples 1.1 and 4.1 Continued)……Page 246
5.19 Incremental EM……Page 247
5.20 Linear Inverse Problems……Page 248
6.1 Introduction……Page 250
6.2.1 Integration and Optimization……Page 251
6.3.1 Introduction……Page 252
6.3.2 Example 6.2: Monte Carlo EM for Censored Data from Normal……Page 254
6.3.4 MCEM in Generalized Linear Mixed Models……Page 255
6.3.5 Estimation of Standard Error with MCEM……Page 256
6.3.6 Example 6.4: MCEM Estimate of Standard Error for One-Parameter Multinomial (Example 1.1 Continued)……Page 257
6.3.7 Stochastic EM Algorithm……Page 258
6.4.1 The Algorithm……Page 259
6.4.2 Example 6.5: Data Augmentation in the Multinomial (Examples 1.1, 1.5 Continued)……Page 260
6.5.1 Posterior Mode by EM……Page 261
6.5.2 Example 6.6: Bayesian EM for Normal with Semi-Conjugate Prior……Page 262
6.6.1 Introduction……Page 263
6.6.2 Rejection Sampling Methods……Page 264
6.6.3 Importance Sampling……Page 265
6.7.1 Introduction……Page 267
6.7.2 Essence of MCMC……Page 269
6.7.3 Metropolis–Hastings Algorithms……Page 270
6.8.1 Introduction……Page 272
6.8.2 Rao–Blackwellized Estimates with Gibbs Samples……Page 273
6.8.3 Example 6.7: Why Does Gibbs Sampling Work?……Page 274
6.9.1 Example 6.8: M-H Algorithm for Bayesian Probit Regression……Page 276
6.9.2 Monte Carlo EM with MCMC……Page 277
6.9.3 Example 6.9: Gibbs Sampling for the Mixture Problem……Page 280
6.9.4 Example 6.10: Bayesian Probit Analysis with Data Augmentation……Page 281
6.9.5 Example 6.11: Gibbs Sampling for Censored Normal……Page 282
6.10.1 EM–Gibbs Sampling Connection……Page 285
6.10.2 Example 6.12: EM–Gibbs Connection for Censored Data from Normal (Example 6.11 Continued)……Page 287
6.10.4 Rate of Convergence of Gibbs Sampling and EM……Page 288
6.11.1 Introduction……Page 289
6.11.2 Example 6.14: Data Augmentation and Gibbs Sampling for Censored Normal (Example 6.12 Continued)……Page 290
6.11.3 Example 6.15: Gibbs Sampling for a Complex Multinomial (Example 2.4 Continued)……Page 291
6.11.4 Gibbs Sampling Analogs of ECM and ECME Algorithms……Page 292
6.12 Empirical Bayes and EM……Page 294
6.13 Multiple Imputation……Page 295
6.14 Missing-Data Mechanism, Ignorability, and EM Algorithm……Page 296
7.1 Introduction……Page 300
7.3 Quasi-Score and the Projection-Solution Algorithm……Page 301
7.4.1 Introduction……Page 304
7.4.3 Example 7.1: Multinomial Example by ES Algorithm (Example 1.1 Continued)……Page 305
7.5 Other Generalizations……Page 306
7.6 Variational Bayesian EM Algorithm……Page 307
7.7.1 Introduction……Page 309
7.7.2 Methods for Constructing Majorizing/Minorizing Functions……Page 310
7.7.3 Example 7.2: MM Algorithm for the Complex Multinomial (Example 1.1 Continued)……Page 311
7.8 Lower Bound Maximization……Page 312
7.9.2 Example 7.3: Interval-EM Algorithm for the Complex Multinomial (Example 2.4 Continued)……Page 314
7.10.2 Simulated Annealing……Page 315
7.10.3 Comparison of SA and EM Algorithm for Normal Mixtures……Page 316
7.11 The Delta Algorithm……Page 317
7.12 Image Space Reconstruction Algorithm……Page 318
8.1 Introduction……Page 320
8.2 Hidden Markov Models……Page 321
8.3 AIDS Epidemiology……Page 324
8.4.1 Introduction……Page 326
8.4.2 EM Framework for NNs……Page 327
8.4.3 Training Multi-Layer Perceptron Networks……Page 328
8.4.5 An Integration of the Methodology Related to EM Training of RBF Networks……Page 331
8.4.6 Mixture of Experts……Page 332
8.4.7 Simulation Experiment……Page 336
8.4.8 Normalized Mixtures of Experts……Page 337
8.4.9 Hierarchical Mixture of Experts……Page 338
8.4.10 Boltzmann Machine……Page 339
8.5 Data Mining……Page 340
8.6 Bioinformatics……Page 341
REFERENCES……Page 342
AUTHOR INDEX……Page 370
SUBJECT INDEX……Page 378
Reviews
There are no reviews yet.