Applied numerical linear algebra

James W. Demmel9780898713893, 0898713897

Designed for use by first-year graduate students from a variety of engineering and scientific disciplines, this comprehensive textbook covers the solution of linear systems, least squares problems, eigenvalue problems, and the singular value decomposition. The author, who helped design the widely used LAPACK and ScaLAPACK linear algebra libraries, draws on this experience to present state-of-the-art techniques for these problems, including recommendations of which algorithms to use in a variety of practical situations.
If you are looking for a textbook that – teaches state-of-the-art techniques for solving linear algebra problems, – covers the most important methods for dense and sparse problems, – presents both the mathematical background and good software techniques, – is self-contained, assuming only a good undergraduate background in linear algebra,
then this is the book for you.
Algorithms are derived in a mathematically illuminating way, including condition numbers and error bounds. Direct and iterative algorithms, suitable for dense and sparse matrices, are discussed. Algorithm design for modern computer architectures, where moving data is often more expensive than arithmetic operations, is discussed in detail, using LAPACK as an illustration. There are many numerical examples throughout the text and in the problems at the ends of chapters, most of which are written in Matlab and are freely available on the Web.
Material either not available elsewhere, or presented quite differently in other textbooks, includes – a discussion of the impact of modern cache-based computer memories on algorithm design; – frequent recommendations and pointers in the text to the best software currently available, including a detailed performance comparison of state-of-the-art software for eigenvalue and least squares problems, and a description of sparse direct solvers for serial and parallel machines; – a discussion of iterative methods ranging from Jacobi’s method to multigrid and domain decomposition, with performance comparisons on a model problem; – a great deal of Matlab-based software, available on the Web, which either implements algorithms presented in the book, produces the figures in the book, or is used in homework problems; – numerical examples drawn from fields ranging from mechanical vibrations to computational geometry; – high-accuracy algorithms for solving linear systems and eigenvalue problems, along with tighter “relative” error bounds; – dynamical systems interpretations of some eigenvalue algorithms.
Demmel discusses several current research topics, making students aware of both the lively research taking place and connections to other parts of numerical analysis, mathematics, and computer science. Some of this material is developed in questions at the end of each chapter, which are marked Easy, Medium, or Hard according to their difficulty. Some questions are straightforward, supplying proofs of lemmas used in the text. Others are more difficult theoretical or computing problems. Questions involving significant amounts of programming are marked Programming. The computing questions mainly involve Matlab programming, and others involve retrieving, using, and perhaps modifying LAPACK code from NETLIB.

---

Table of contents :
Cover ……Page 1
Contents ……Page 3
Preface to the 1996 Edition ……Page 9
1.2 Standard Problems of Numerical Linear Algebra ……Page 11
1.3 General Techniques ……Page 12
1.3.1 Matrix Factorizations ……Page 13
1.3.3 Effects of Roundoff Error on Algorithms ……Page 14
1.3.4 Analyzing the Speed of Algorithms ……Page 15
1.3.5 Engineering Numerical Software ……Page 16
1.4 Example: Polynomial Evaluation ……Page 17
1.5 Floating Point Arithmetic ……Page 19
1.6 Polynomial Evaluation Revisited ……Page 25
1.7 Vector and Matrix Norms ……Page 29
1.8 References and Other Topics for Chapter 1 ……Page 33
1.9 Questions for Chapter 1 ……Page 34
2.1 Introduction ……Page 41
2.2 Perturbation Theory ……Page 42
2.2.1 Relative Perturbation Theory ……Page 45
2.3 Gaussian Elimination ……Page 48
2.4 Error Analysis ……Page 54
2.4.1 The Need for Pivoting ……Page 55
2.4.2 Formal Error Analysis of Gaussian Elimination ……Page 56
2.4.3 Estimating Condition Numbers ……Page 60
2.4.4 Practical Error Bounds ……Page 64
2.5 Improving the Accuracy of a Solution ……Page 70
2.5.1 Single Precision Iterative Refinement ……Page 71
2.5.2 Equilibration ……Page 72
2.6 Blocking Algorithms for Higher Performance ……Page 73
2.6.1 Basic Linear Algebra Subroutines (BLAS) ……Page 75
2.6.2 How to Optimize Matrix Multiplication ……Page 77
2.6.3 Reorganizing Gaussian Elimination to use Level 3 BLAS ……Page 82
2.6.4 More About Parallelism and Other Performance Issues ……Page 85
2.7.1 Real Symmetric Positive Definite Matrices ……Page 86
2.7.3 Band Matrices ……Page 89
2.7.4 General Sparse Matrices ……Page 93
2.7.5 Dense Matrices Depending on Fewer Than 0(n2) Parameters ……Page 100
2.8 References and Other Topics for Chapter 2 ……Page 102
2.9 Questions for Chapter 2 ……Page 103
3.1 Introduction ……Page 111
3.2 Matrix Factorizations That Solve the Linear Least Squares Problem ……Page 115
3.2.1 Normal Equations ……Page 116
3.2.2 QR Decomposition ……Page 117
3.2.3 Singular Value Decomposition ……Page 119
3.3 Perturbation Theory for the Least Squares Problem ……Page 127
3.4 Orthogonal Matrices ……Page 128
3.4.1 Householder Transformations ……Page 129
3.4.2 Givens Rotations ……Page 131
3.4.3 Roundoff Error Analysis for Orthogonal Matrices ……Page 133
3.4.4 Why Orthogonal Matrices? ……Page 134
3.5 Rank-Deficient Least Squares Problems ……Page 135
3.5.1 Solving Rank-Deficient Least Squares Problems Using the SVD ……Page 137
3.5.2 Solving Rank-Deficient Least Squares Problems Using QR with Pivoting ……Page 140
3.6 Performance Comparison of Methods for Solving Least Squares Problems ……Page 142
3.8 Questions for Chapter 3 ……Page 144
4.1 Introduction ……Page 149
4.2 Canonical Forms ……Page 150
4.3 Perturbation Theory ……Page 158
4.4 Algorithms for the Nonsymmetric Eigenproblem ……Page 163
4.4.1 Power Method ……Page 164
4.4.2 Inverse Iteration ……Page 165
4.4.3 Orthogonal Iteration ……Page 166
4.4.4 QR Iteration ……Page 169
4.4.5 Making QR Iteration Practical ……Page 173
4.4.6 Hessenberg Reduction ……Page 174
4.4.7 Tridiagonal and Bidiagonal Reduction ……Page 176
4.4.8 QR Iteration with Implicit Shifts ……Page 177
4.5.1 Regular Matrix Pencils and Weierstrass Canonical Form ……Page 183
4.5.2 Singular Matrix Pencils and the Kronecker Canonical Form ……Page 189
4.5.3 Nonlinear Eigenvalue Problems ……Page 192
4.6 Summary ……Page 194
4.8 Questions for Chapter 4 ……Page 197
5.1 Introduction ……Page 205
5.2 Perturbation Theory ……Page 207
5.2.1 Relative Perturbation Theory ……Page 217
5.3 Algorithms for the Symmetric Eigenproblem ……Page 220
5.3.1 Tridiagonal QR Iteration ……Page 222
5.3.2 Rayleigh Quotient Iteration ……Page 225
5.3.3 Divide-and-Conquer ……Page 227
5.3.4 Bisection and Inverse Iteration ……Page 238
5.3.5 Jacobi’s Method ……Page 242
5.3.6 Performance Comparison ……Page 246
5.4 Algorithms for the Singular Value Decomposition ……Page 247
5.4.1 QR Iteration and Its Variations for the Bidiagonal SVD ……Page 252
5.4.2 Computing the Bidiagonal SVD to High Relative Accuracy ……Page 256
5.4.3 Jacobi’s Method for the SVD ……Page 259
5.5 Differential Equations and Eigenvalue Problems ……Page 264
5.5.1 The Toda Lattice ……Page 265
5.6 References and Other Topics for Chapter 5 ……Page 270
5.7 Questions for Chapter 5 ……Page 271
6.1 Introduction ……Page 275
6.2 On-line Help for Iterative Methods ……Page 276
6.3.1 Poisson’s Equation in One Dimension ……Page 277
6.3.2 Poisson’s Equation in Two Dimensions ……Page 280
6.3.3 Expressing Poisson’s Equation with Kronecker Products ……Page 284
6.4 Summary of Methods for Solving Poisson’s Equation ……Page 287
6.5 Basic Iterative Methods ……Page 289
6.5.1 Jacobi’s Method ……Page 291
6.5.2 Gauss-Seidel Method ……Page 292
6.5.3 Successive Overrelaxation ……Page 293
6.5.4 Convergence of Jacobi’s, Gauss-Seidel, and SOR(w) Methods on the Model Problem ……Page 295
6.5.5 Detailed Convergence Criteria for Jacobi’s, Gauss-Seidel, and SOR(w) Methods ……Page 296
6.5.6 Chebyshev Acceleration and Symmetric SOR (SSOR) ……Page 304
6.6 Krylov Subspace Methods ……Page 310
6.6.1 Extracting Information about A via Matrix-Vector Multiplication ……Page 311
6.6.2 Solving Ax = b Using the Krylov Subspace 7C& ……Page 316
6.6.3 Conjugate Gradient Method ……Page 317
6.6.4 Convergence Analysis of the Conjugate Gradient Method ……Page 322
6.6.5 Preconditioning ……Page 327
6.6.6 Other Krylov Subspace Algorithms for Solving Ax = b ……Page 330
6.7 Fast Fourier Transform ……Page 331
6.7.1 The Discrete Fourier Transform ……Page 333
6.7.2 Solving the Continuous Model Problem Using Fourier Series ……Page 334
6.7.4 Computing the Fast Fourier Transform ……Page 336
6.8 Block Cyclic Reduction ……Page 338
6.9 Multigrid ……Page 341
6.9.1 Overview of Multigrid on Two-Dimensional Poisson’s Equation ……Page 343
6.9.2 Detailed Description of Multigrid on One-Dimensional Poisson’s Equation ……Page 347
6.10 Domain Decomposition ……Page 358
6.10.1 Nonoverlapping Methods ……Page 359
6.10.2 Overlapping Methods ……Page 362
6.12 Questions for Chapter 6 ……Page 367
7.1 Introduction ……Page 373
7.2 The Rayleigh-Ritz Method ……Page 374
7.3 The Lanczos Algorithm in Exact Arithmetic ……Page 378
7.4 The Lanczos Algorithm in Floating Point Arithmetic ……Page 387
7.5 The Lanczos Algorithm with Selective Orthogonalization ……Page 392
7.6 Beyond Selective Orthogonalization ……Page 395
7.7 Iterative Algorithms for the Nonsymmetric Eigenproblem ……Page 396
7.9 Questions for Chapter 7 ……Page 398
Bibliography ……Page 401
Index ……Page 421