PRINCIPLES OF
To Kristine, Sammy, and Justin, APPLIED Still my best friends after all these years. MATHEMATICS
Transformation and Approximation Revised Edition
JAMES P. KEENER University of Utah Salt Lake City, Utah
ADVANCED BOOK PROGRAM
I ewI I I ~ Member of the Perseus Books Group Contents
Preface to First Edition xi
Preface to Second Edition xvii
1 Finite Dimensional Vector Spaces 1 1.1 Linear Vector Spaces ...... 1 1.2 Spectral Theory for Matrices . . . 9 1.3 Geometrical Significance of Eigenvalues 17 1.4 Fredholm Alternative Theorem . . . . . 24 1.5 Least Squares Solutions-Pseudo Inverses . 25 1.5.1 The Problem of Procrustes .... 41 Many of the designations used by manufacturers and sellers to distinguish their 42 products are claimed as trademarks. Where those designations appear in this 1.6 Applications of Eigenvalues and Eigenfunctions book and Westview Press was aware of a trademark claim, the designations have 1.6.1 Exponentiation of Matrices ...... 42 been printed in initial capital letters. 1.6.2 The Power Method and Positive Matrices 43 45 Library of Congress Catalog Card Number: 99-067545 1.6.3 Iteration Methods Further Reading . . . . 47 ISBN: 0-7382-0129-4 Problems for Chapter 1 49 Copyright© 2000 by James P. Keener 2 Function Spaces 59 Westview Press books are available at special discounts for bulk purchases in the 2.1 Complete Vector Spaces .... . 59 U.S. by corporations, institutions, and other organizations. For more informa 2.1.1 Sobolev Spaces ...... 65 tion, please contact the Special Markets Department at the Perseus Books Group, 2.2 Approximation in Hilbert Spaces 67 11 Cambridge Center, Cambridge, MA 02142, or call (617) 252-5298. 2.2.1 Fourier Series and Completeness 67 All rights reserved. No part of this publication may be reproduced, stored in a 2.2.2 Orthogonal Polynomials . . . . 69 retrieval system, or transmitted, in any form or by any means, electric, mechani 2.2.3 Trigonometric Series . . . . . 73 cal, photocopying, recording, or otherwise, without the prior written permission 76 of the publisher. Printed in the United States of America. 2.2.4 Discrete Fourier Transforms . 2.2.5 Sine Functions 78 Westview Press is a member of the Perseus Books Group 2.2.6 Wavelets .... 79 2.2. 7 Finite Elements . 88 First printing, December 1999 Further Reading . . . . 92 Problems for Chapter 2 . . . 93 Find us on the World Wide Web at http:/ /www.westviewpress.com v CONTENTS vii vi CONTENTS
3 Integral Equations 101 6 Complex Variable Theory 209 3.1 Introduction ...... 101 6.1 Complex Valued Functions ...... 209 3.2 Bounded Linear Operators in Hilbert Space ...... 105 6.2 The Calculus of Complex Functions ...... 214 3.3 Compact Operators ...... 111 6.2.1 Differentiation-Analytic Functions ...... 214 3.4 Spectral Theory for Compact Operators ...... 114 6.2.2 Integration ...... 217 ? 3.5 Resolvent and Pseudo-Resolvent Kernels ...... 118 6.2.3 Cauchy Integral Formula ...... 220 3.6 Approximate Solutions ...... 121 6.2.4 Taylor and Laurent Series ...... 224 3. 7 Singular Integral Equations ...... 125 6.3 Fluid Flow and Conformal Mappings ...... 228 I~ Further R.eading ...... 127 6.3.1 Laplace's Equation ...... 228 Problems for Chapter 3 ...... · ...... 128 I 6.3.2 Conformal Mappings ...... 236 6.3.3 Free Boundary Problems ...... 243 4 Differential Operators 133 6.4 Contour Integration ...... 248 I 6.5 Special Functions ...... 259 4.1 Distributions and the Delta Function ...... 133 I 4.2 Green's Functions ...... 144 ii! 6.5.1 The Gamma Function ...... 259 4.3 Differential Operators ...... 151 I 6.5.2 Bessel Functions ...... 262 4.3.1 Domain of an Operator ...... 151 " 6.5.3 Legendre Functions ...... 268 4.3.2 Adjoint of an Operator ...... 152 6.5.4 Sine Functions ...... 270 4.3.3 Inhomogeneous Boundary Data ...... 154 I Further Reading ...... 273 J Problems for Chapter 6 ...... ~ . 27 4 4.3.4 The Fredholm Alternative ...... 155 ~ 4.4 Least Squares Solutions ...... ~ 157 7 Transform and Spectral Theory 283 4.5 Eigenfunction Expansions ...... -~i 161 7.1 Spectrum of an Operator ...... 283 4.5.1 'Irigonometric Functions ...... 164 I 7.2 Fourier Transforms ...... , ...... 284 4.5.2 Orthogonal Polynomials ...... 167 ~ 7.2.1 Transform Pairs ...... , .. 284 4.5.3 Special Functions ...... ~ 169 § 7.2.2 Completeness of Hermite and Laguerre Polynomials ... 297 4.5.4 Discretized Operators ...... 169 7.2.3 Sine Functions ...... 299 Further R.eading ...... · · · · · · · · · · · · · 171 I Problems for Chapter 4 ...... 171 s" 7.2.4 Windowed Fourier Transforms ...... 300 I 7.2.5 Wavelets ...... 301 5 Calculus of V~iations 177 7.3 Related Integral Transforms ...... 307 i 7.3.1 Laplace Transform ...... 307 5.1 The Euler-Lagrange Equations ...... 177 ~ 5.1.1 Constrained Problems ...... 180 • 7.3.2 Mellin Transform ...... 308 5.1.2 Several Unknown Functions ...... 181 I 7.3.3 Hankel Transform ...... 309 5.1.3 Higher Order Derivatives ...... 184 I 7.4 Z Transforms ...... 310 li 7.5 Scattering Theory ...... 312 5.1.4 Variable Endpoints ...... 184 ~ 5.1.5 Several Independent Variables ...... 185 7.5.1 Scattering Examples ...... 318 5.2 Hamilton's Principle ...... 186 7.5.2 Spectral Representations ...... 325 5.2.1 The Swinging Pendulum ...... 188 Further Reading ...... 327 I Problems for Chapter 7 ...... , . . 328 5.2.2 The Vibrating String ...... 189 I 5.2.3 The Vibrating Rod ...... 189 ! Appendix: Fourier Transform Pairs ...... 335 5.2.4 Nonlinear Deformations of a Thin Beam ...... 193 i J 8 Partial Differential Equations 337 5.2.5 A Vibrating Membrane ...... 194 iii 5.3 Approximate Methods ...... 195 ~ 8.1 Poisson's Equation ...... 339 5.4 Eigenvalue Problems ...... 198 8.1.1 Fundamental Solutions ...... 339 5.4.1 Optimal Design of Structures ...... 201 8.1.2 The Method of Images ...... 343 Further R.eading ...... 202 8.1.3 Transform Methods ...... 344 Problems for Chapter 5 ...... 203 8.1.4 Hilbert Transforms ...... 355 viii CONTENTS CONTENTS ix
8.1.5 Boundary Integral Equations ...... 357 Problems for Chapter 11 ...... 498 8.1.6 Eigenfunctions ...... 359 8.2 The Wave Equation ...... 365 12 Singular Perturbation Theory 505 8.2.1 Derivations ...... 365 12.1 Initial Value Problems I ...... 505 8.2.2 Fundamental Solutions ...... 368 12.1.1 Van der Pol Equation ...... 508 8.2.3 Vibrations ...... 373 12.1.2 Adiabatic lnvariance ...... 510 8.2.4 Diffraction Patterns ...... 376 12.1.3 Averaging ...... , ...... 511 8.3 The a:eat Equation ...... 380 12.1.4 Homogenization Theory ...... , ...... 514 8.3.1 Derivations ...... 380 12.2 Initial Value Problems II ...... , ...... 520 8.3.2 Fundamental Solutions ...... 383 12.2.1 Operational Amplifiers ...... 521 8.3.3 Transform Methods ...... 385 12.2.2 Enzyme Kinetics ...... 523 8.4 Differential-Difference Equations ...... 390 12.2.3 Slow Selection in Population Genetics ...... 526 8.4.1 Transform Methods ...... 392 .~ 12.3 Boundary Value Problems ...... 528 8.4.2 Numerical Methods ...... 395 12.3.1 Matched Asymptotic Expansions ...... 528 I 12.3.2 Flame Fronts ...... 539 Further Reading ...... 400 ~ Problems for Chapter 8 ...... 402 12.3.3 Relaxation Dynamics ...... 542 12.3.4 Exponentially Slow Motion ...... 548 9 Inverse Scattering Transform 411 I Further Reading ...... 551 9.1 Inverse Scattering ...... 411 Problems for Chapter 12 ...... 552 9.2 Isospectral Flows ...... 417 i i Bibliography 559 9.3 Korteweg-deVries Equation ...... 421 ~ ~ 9.4 The Toda Lattice ...... 426 I Selected Hints and Solutions 567 Further Reading ...... 432 I Problems for Chapter 9 ...... 433 Index 596 10 Asymptotic Expansions 437 ~ 10.1 Definitions and Properties ...... 437 I 10.2 Integration by Parts ...... 440 I 10.3 Laplace's Method ...... 442 10.4 Method of Steepest Descents ...... ; 449 I 10.5 Method of Stationary Phase ...... 456 Further Reading ...... ~ ...... 463 'I Problems for Chapter 10 ...... 463
11 Regular Perturbation Theory 469 ! 11.1 The Implicit Function Theorem ...... 469 I 11.2 Perturbation of Eigenvalues ...... 475 * 11.3 Nonlinear Eigenvalue Problems ...... 478 I 11.3.1 Lyapunov-Schmidt Method ...... 482 I 11.4 Oscillations and Periodic Solutions ...... 482 ~ 11.4.1 Advance of the Perihelion of Mercury ...... 483 11.4.2 Vander Pol Oscillator ...... 485 11.4.3 Knotted Vortex Filaments ...... 488 11.4.4 The Melnikov Function ...... 493 11.5 Hopf Bifurcations ...... _...... 494 Further Reading ...... 496 Preface to First Edition
Applied mathematics should read like good mystery, with an intriguing begin ning, a clever but systematic middle, and a satisfying resolution at the end. Often, however, the resolution of one mystery opens up a whole new problem, and the process starts all over. For the applied mathematical scientist, there is the goal to explain or predict the behavior of some physical situation. One begins by constructing a mathematical model which captures the essential fea tures of the problem without masking its content with overwhelming detail. Then comes the analysis of the model where every possible tool is tried, and some new tools developed, in order to understand the behavior of the model as thoroughly as possible. Finally, one must interpret and compare these results with real world facts. Sometimes this comparison is quite satisfactory, but most often one discovers that important features of the problem are not adequately 1 accounted for, and the process begins again. 1 Although every problem has its own distinctive features, through the years it has become apparent that there are a group of tools that are essential to the analysis of problems in many disciplines. This book is about those tools. But more than being just a grab bag of tools and techniques, the purpose of this book is to show that there is a systematic, even esthetic, explanation of how ~ I-?£ these classical tools work and fit together as a unit. Much of applied mathematical analysis can be summarized by the observa tion that we continually attempt to reduce our problems to ones that we already I know how to solve. I This philosophy is illustrated by an anecdote (apocryphal, I hope) of a math t ematician and an engineer who, as chance would have it, took a gourmet cooking I class together. As this course started with the basics, the first lesson was on t how to boil a pot of water. The instructor presented the students with two pots of water, one on the counter, one on a cold burner, while anot_her burner was I already quite hot, but empty. •(1 In order to test their cooking aptitudes, the instructor first asked the engineer to demonstrate to the rest of the class how to boil the pot of water that was already sitting on the stove. He naturally moved the pot carefully from the cold burner to the hot burner, to the appreciation of his classmates. To be sure the class understood the process, the pot was moved back to its original spot on the cold burner at which time the mathematician was asked to demonstrate how to heat the water in the pot sitting on the counter. He promptly and confidently
xi xii PREFACE TO FIRST EDITION PREFACE TO FIRST EDITION xiii exchanged the position of the two pots, placing the pot from the counter onto might try to represent u and I as polynomials or infinite power series in x, but the cold burner and the pot from the burner onto the counter. He then stepped we quickly learn that this guess does not simplify the solution process much. back from the stove, expecting an appreciative response from his mates. Baffled Instead, the ''natural" choice is to represent u and I as trigonometric series, by his actions, the instructor asked him to explain what he had done, and he 00 00 replied naturally, that he had simply reduced his problem to one that everyone u(x) = u~,: sinkx, l(x) 1~.: sin kx. already knew how to solve. L = L k=l k=l We shall also make it our goal to reduce problems to those which we already know how to solve. Using this representation (i.e., coordinate system) we find that the original dif We can illustrate this underlying idea with simple examples, we know that ferential equation reduces to the infinite number of separated algebraic equations to solve the algebraic equation 3x = 2, we multiply both sides of the equation (k2 + 2)u~,: = -1~.:· Since these equations are separated (the kth equation de with the inverse of the "operator" 3, namely 1/3, to obtain x = 2/3. The same pends only on the unknown u~.:), we can solve them just as before, even though is true if we wish to solve the matrix equation Ax = b where there are an infinite number of equations. We have managed to simplify this problem by transforming into the correctly chosen coordinate system. For many of the problems we encounter in the sciences, there is a natural way A=(~!). b=(~)· to represent the solution that transforms the problem ip.to a substantially easier one. All of the well-known special functions, including Legendre polynomials, Namely, if we know A-t, the inverse of the matrix operator A, we multiply both Bessel functions, Fourier series, Fourier integrals, etc., have as their common sides of the equation by A-t to obtain x : A - 1b. For this problem, motivation that they are natural for certain problems, and perfectly ridiculous in others. It is important to know when to choose one transform over another. A-t=! ( 3 -1 ) Not all problems can be solved exactly, and it is a mistake to always look for 8 -1 3 ' x=~(!)· exact solutions. The second basic technique of applied mathematics is to reduce hard problems to easier problems by ignoring small terms. For example, to find H we make it our goal to invert many kinds of linear operators, including the roots of the polynomial x2 + x + .0001 = 0, we notice that the equation is matrix, integral, and differential operators, we will certainly be able to solve 2 many types of problems. However, there is an approach to calculating the very close to the equation x + x = 0 which has roots x = -1, and x = 0, and inverse operator that also gives us geometrical insight. We try to transform the we suspect that the roots of the original polynomial are not too much different original problem into a simpler problem which we already know how to solve. from these. Finding how changes in parameters affect the solution is the goal Fbr example, if we rewrite the equation Ax= bas T-1 AT(T-1x) = T-1b and of perturbation theory and asymptotic analysis, and in this example we have choose a regular perturbation problem, since the solution is a regular (i.e., analytic) function of the "parameter" 0.0001. - ( 1 -1 ) -1- 1 ( 1 1 ) T- 1 1 I T - 2 -1 1 I Not all reductions lead to such obvious conclusions. For example, the poly nomial 0.0001x2 + x + 1 = 0 is close to the polynomial x + 1 0, but the first we find that = has two roots while the second has only one. Where did the second root go? 1 T- AT = ( ~ -~ ) . We know, of course, that there is a very large root that "goes to infinity" as "0.0001 goes to zero," and this example shows that our naive idea of setting all With the change of variables y = T-1x, g = T-1b, the new problem looks small parameters to zero must be done with care. As we see, not all problems like two of the easy algebraic equations we already know how to solve, namely with small parameters are regular, but some have a singular behavior. We need 4yt = 3/2, -2y2 = 1/2. This process of separating the coupled equations to know how to distinguish between regular and singular approximations, and into uncoupled equations works only if T is carefully chosen, and exactly how what to do in each case. this is done is still a mystery. Suffice it to say, the original problem has been This book is written for beginning graduate students in applied mathemat transformed, by a carefully chosen change of coordinate system, into a problem ics, science, and engineering, and is appropriate as a one-year course in applied we already know how to solve. mathematical techniques (although I have never been able to cover all of this This process of changing coordinate systems is very useful in many other material in one year). We assume that the students have studied at an intro problems. For example, suppose we wish to solve the boundary value problem ductory undergraduate level material on linear algebra, ordinary and partial u"- 2u =l(x) with u(O) =u(1r) =0. As we do not yet know how to write differential equations, and complex variables. The emphasis of the book is a down an inverse operator, we look for an alternative approach. The single most working, systematic understanding of classical techniques in a modern context. important step is deciding how to represent the solution u(x). For example, we Along the way, students are exposed to models from a variety of disciplines. xiv PREFACE TO FIRST EDITION PREFACE TO FIRST EDITION XV
It is hoped that this course will prepare students for further study of modem entiate analytic functions at a reasonably sophisticated level is indispensable to techniques and in-depth modeling in their own specific discipline. the remaining text. Section 6.3 (Applications to Fluid Flow) is included because One book cannot do justice to all of applied mathematics, and in an effort it is a classically important and very lovely subject, but it plays no role in the to keep the amount of material at a manageable size, many important topics remaining development of the book, and could be skipped if time constraints were not included. In fact, each of the twelve chapters could easily be expanded demand. into an entire book. The topics included here have been selected, not only for Chapter seven continues the development of transform theory and we show their scientific importance, but also because they allow a logical flow to the that eigenvalues and eigenfunctions are not always sufficient to build a trans development of the ideas and techniques of transform theory and asymptotic form, and that operators having continuous spectrum require a generaliZed con analysis. The theme of transform theory is introduced for matrices in Chapter struction. It is in this context that Fourier, Mellin, Hankel, and Z transforms, one, and revisited for integral equations in Chapter three, for ordinary differ as well as scattering theory for the SchrOdinger operator are studied. ential equations in Chapters four and seven, for partial differential equations in In Chapter eight we show how to solve linear partial differential and differ Chapter eight, and for certain nonlinear evolution equations in Chapter nine. ence equations, with special emphasis on (as you guessed) transform theory. In Once we know how to solve a wide variety of problema via transform theory, this chapter we are able to make specific use of all the techniques introduced so it becomes appropriate to see what harder problems we can reduce to those we far and solve some problems with interesting applications. know how to solve. Thus, in Chapters ten, eleven, and twelve we give a survey Although much of transform theory is rather old, it is by no means dead. of the three basic areas of asymptotic analysis, namely asymptotic analysis of In Chapter nine we show how transform theory has recently been used to solve integrals, regular perturbation theory and singular perturbation theory. certain nonlinear evolution equations. We illustrate the inverse scattering trans Here is a summary of the text, chapter by chapter. In Chapter one, we form on the Korteweg-deVries equation and the Toda lattice. review the basics of spectral theory for matrices with the goal of understanding In Chapter ten we show how asymptotic methods can be used to approx not just the mechanics of how to solve matrix equations, but more importantly, imate the horrendous integral expressions that so often result from transform the geometry of the solution process, and the crucial role played by eigenvalues techniques. and eigenvectors in finding useful changes of coordinate systems. This usefulness In Chapter eleven we show how perturbation theory and especially the study extends to pseudo-inverse operators as well as operators in HUbert space, and of nonlinear eigenvalue problems uses knowledge of the spectrum of a linear so is a particularly important piece of background information. operator in a fundamental way. The nonlinear problems in this chapter all have In Chapter two, we extend many of the notions of finite dimensional vector the feature that their solutions are close to the solutions of a nearby linear spaces to function spaces. The main goal is to show how to represent objects problem. in a function space. Thus, Hilbert spaces and representation of functions in a Singular perturbation problema fail to have this property, but have solutions Hilbert space are studied. In this context we meet classical sets of functions that differ markedly from the naive simplified problem. In Chapter twelve, we such as Fourier series and Legendre polynomials, as well as less well-known sets give a survey of the three basic singular perturbation problema (slowly varying such as the Walsh functions, Sine functions, and finite element bases. oscillations, initial value problems with vastly different time scales, and bound In Chapter three, we explore the strong analogy between integral equations ary value problems with boundary layers). and matrix equations, and examine again the consequences of spectral theory. This book has the lofty goal of reaching students of mathematics, science, This chapter is more abstract than others as it is an introduction to functional and engineering. To keep the attention of mathematicians, one must be system analysis and compact operator theory, given under the guise of Fredholm integral atic, and include theorems and proofs, but not too many explicit calculations. equations. The added generality is important as a framework for things to come. To interest an engineer or scientist, one must give specific examples of how to In Chapter four, we develop the tools necessary to use spectral decompo solve meaningful problema but not too many proofs or too much abstraction. sitions to solve differential equations. In particular, distributions and Green's In other words, there is always someone who is unhappy. functions are used as the means by which the theory of compact operators can In an effort to minimize the total displeasure and not scare off too many be applied to differential operators. With these tools in place, the completeness members of either camp, there are some proofs and some computations. Early of eigenfunctions of Sturm-Liouville operators follows directly. in the text, there are disproportionately more proofs than computations because Chapter five is devoted to showing how many classical differential equations as the foundations are being laid, the proofs often give important insights. Later, can be derived from a variational principle, and how the eigenvalues of a differ after the bases are established, they are invoked in the problem solving process, ential operator vary as the operator is changed. but proofs are deemphasized. (For example, there are no proofs in Chapters Chapter six is a pivotal chapter, since all chapters following it require a sub eleven and twelve.) Experience has shown that this approach is, if not optimal, stantial understanding of analytic function theory, and the chapters preceding it at least palatable to both sides. require no such knowledge. In particUlar, knowing how to integrate and differ- This is intended to be an applied mathematics book and yet there is very xvi PREFACE TO FIRST EDITION little mention of numerical methods. How can this be? The answer is simple and, I hope, satisfactory. This book is primarily about the principles that one uses to solve problems and since these principles often have consequence in numerical algorithms, mention of numerical routines is made when appropriate. However, FORTRAN codes or other specific implementations can be found in many other good resources and so (except for a code for fast Walsh transforms) are omitted. On the other hand, many of the calculations in this text should Preface to the Second be done routinely using symbolic manipulation languages such as REDUCE or MACSYMA. Since these languages are not generally familiar to many, included Edition here are a number of short programs written in REDUCE in order to encourage readers to learn how to use these remarkable tools. (Some of the problems at the end of several chapters are intended to be so tedious that they force the reader to learn one of these languages.) When the first edition of this book was published, l thought that it was an This text would not have been possible were it not for the hard work and up-to-date look at classical methods of applied mathematics. I did not expect encouragement of many other people. There were numerous students who strug the basic tools of applied mathematicians to change dramatically in the near gled through this material without the benefit of written notes while the course future. Indeed the basic ideas remain the same, but there are many ways in was evolving. More recently, Prof. Calvin Wilcox, Prof. Frank Hoppensteadt, which those basic ideas have found new applications and extensions. Gary deYoung, Fred Phelps, and Paul Arner have been very influential in the Some of the most significant new techniques of the last decade are wavelet shaping of this presentation. Finally, the patience of Annetta Cochran and analysis, multigrid methods, and homogenization theory, all of which are exten Shannon Ferguson while typing from my illegible scrawling was exemplary. sions of methods already discussed in the first edition. Additionally, software In spite of all the best efforts of everyone involved, I am sure there are still tools have become much more sophisticated and reliable, and it is not possible typographical errors in the text. It is disturbing that I can read and reread to ignore these developments in the training of an applied mathematician. a section of text and still not catch all the erors. My hope is that those that So the first reason to revise this book is to bring it up to date, by adding remain are both few and obvius and will not lead to undue confusion. material describing these new and important methods of applied mathematics. The second reason for this revision is to make it easier to read and use. To that end, the text has been thoroughly edited, with emphasis on clarity and filling in the gaps. I have tried to eliminate those annoying places where a step was described as obvious, but is far from obvious to the reader. I have also tried to eliminate many of the (mostly typographical, but nonetheless noisome) errors that were in the first edition. I have added equation numbers to those equations that are referenced in the text, and many figures have been added to aid the geometrical interpretation and understanding. Finally, I have added a section of hints and solutions for the exercises. Many of the exercises are difficult and both students and instructors find such a section to be most helpful. Many new exercises have been added, mostly those which are intended to use modern software for symbolic computation and graphical interpretation. The first edition of this book referred to REDUCE, which, sadly, disappeared from widespread use almost immediately after the publication of that edition. The good news is that the new software tools are much better. My personal pref erences are Maple and Matlab, but other choices include Mathematica, Math Cad, Derive, etc. Exercises that are marked with .II!, are designed to encourage computer usage with one of these software packages. I have mentioned Maple throughout the text, but the intent is "generic software package with symbolic computation and graphics capability".
xvii PREFACE TO SECOND EDITION xix xviii PREFACE TO SECOND EDITION
folks at Calvin College for providing me with a quiet place to work for several While the basic philosophy of the text remains unchanged, there are impor months, and to the University of Utah for a sabbatical leave during which this tant ways in which most chapters have been modified. In particular, here are revision was finished. some of the major modifications and additions by chapter: Further corrections or modifications can be found at the Internet site Chapter 1. A section describing the problem of Procrustes (who devised an http://www.math.utah.edu/-keener/AppliedMathBook.html. unusual method for coordinate transformations) has been added. Also, a section on applications of eigenvalues and eigenfunctions James P. Keener of matrices has been added, wherein matrix iterative methods and University of Utah the ranking of sports teams are discussed. June 1999 Chapter 2. A section introducing wavelets has been added, replacing the dis cussion of Walsh functions. Chapter 4. The discussion of extended operators, which I always found to be awkWard, is replaced by a much more transparent and straight forward discussion of inhomogeneous boundary conditions using integration by parts. Chapter 5. An example of how variational principles are used to design optimal structures has been added.
Chapter 7. Further development and construction of wavelets, as well as other transforms (such as windowed Fourier transforms) is included. Chapter 8. A discussion of the Hilbert transform and boundary integral meth ods has been added, along with some interesting new applications. There is also added emphasis on diffusion-reaction equations, with a discussion of the Turing instability, and derivations of the Calm Allen and Calm-Hilliard equations. Finally, a discussion of multi grid methods for the numerical solution of Poisson's equation has been added. Chapter 11. A discussion of the Lyapunov-Schmidt technique for bifurcation problems has been added, as has a description of the Melnikov function. Chapter 12. The discussion of the averaging theorem has been expanded, while discussions of homogenization theory and exponentially slow mo tion of transition layers have been added. The content of Chapters 3, 6, 9 and 10 has changed little from the first edition. As with any project of this size, I owe a debt of gratitude to many people who helped. Eric Cytrynbaum, Young Seon Lee, Todd Shaw and Peter Bates found many of the errors that I missed and made numerous helpful suggestions about the presentation, while David Eyre and Andrej Cherkaev made many helpful suggestions for new material to include. Steve Worcester did the initial 'J.EX-ing of the manuscript in record time. Nelson Beebe is a 'J.EX-pert, whose wizardry never ceases to astound me and who8e help was indispensable. Thanks to the Chapter 1
Finite Dimensional Vector Spaces
1.1 Linear Vector Spaces In any problem that we wish to solve, the goal is to find a particular object, chosen from among a large collection of contenders, which satisfies the governing constraints of the problem. The collection of contending objects is often a vector space, and although individual elements of the set can be ca.lled by many different names, it is common to ca.ll them vectors. Not every set of objects constitutes a vector space. To be specific, if we have a collection of objects S, we must first define addition and sca.la.r multiplication for these objects. The operations of addition and sca.lar multiplication are defined to satisfy the usual properties: H x,y,z E S, then • x + y = y + x (commutative law) • x + (y + z) = (x + y) + z (associative law) • 0 E S, 0 + x = x (additive identity) • -xES, -x + x = 0 (additive inverse) and if xES, a, {3 E IR (or 1 2 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.1. LINEAR VECTOR SPACES 3 Definition 1.1 A set of objects Sis called a linear vector space if two prop 2. A linear combination of two vectors in m?, Clt(Xt,Yl) + Cl2(:Z:2,Y2), is the erties hold: zero vector whenever :z:1 = -/3:z:2 and Yt = -f3y2 where f3 = a2/a1. Geo metrically, this means that the vector (:z:t, yl) is collinear with the vector 1. If x, y E S, then x + y E S (called closure under vector addition), (:z::~, y2), i.e., they are parallel vectors in 1R2. 2. If a E IR (or a E ([;') and x E S, then ax E S (called closure under scalar Important examples of linearly independent vectors are the monomials. That 2 multiplication). is, the powers of x, {1, x, x , ••• , xn} form a linearly independent set. An easy proof of linear independence is to define f(x) = ao + alx + a2x2 + ... + anxn If the scalars are real, S is a real vector space, while if the scalars are complex, and to determine when f(x) is identically zero for all x. Clearly, /(0) = ao and S is a complex vector space. if f(x) = 0 for all x, then /(0) = 0 implies a0 = 0. The kth derivative of f(x) at x = 0, Examples: dk f(x) I = k!ak 1. The set of real ordered pairs (:z:,y) (denoted m?) is a linear vector space d,xk z=O if addition is defined by (:z:1, yt) + (:z:2, y2) = (:z:1 + x2, Yt + y2), and scalar is zero if and only if ak = 0. Therefore, f(x) 0 if and only if ai = 0 for multiplication is defined by a(:z:,y) = (a:z:,ay). = j = 0, 1, 2, ... , n. 2. The set of real n-tuples (:z:t, x2, ... , :Z:n) (denoted lRn) forms a linear vector It is often convenient to represent an element of the vector space as a linear space if addition and scalar multiplication are defined component-wise as combination of a predetermined set of elements of S. For example, we might in the above example. Similarly the set of complex n-tuples (denoted 3. The set {1,z,z2 ,z2 -1} is linearly dependent. However, any of the subsets similarly we discretize g(x) to define G = (g(zl),g(z2), ... ,g(xn)), where 2 2 :z:,. kfn, and it makes sense to define {1,z,z2}, {1,z,z - 1}, or {z,z2,z - 1} forms a basis for the set of = quadratic polynomials. 1 n 4.· The set of all polynomials does not have finite dimension, since any linear (F, G)n = - 2: f(x~c)g(xlc)· n k=l combination of polynomials of degree n or less cannot be a polynomial of degree n + 1. The set of all continuous functions is also infinite since the (While this is an inner product on JR", this is not ap inner product for the polynomials are a subset of this set. We will show later that the polynomials continuous functions. Why?) Taking the limit n -+ oo, we obtain form a "basis" for the continuous functions, but this requires more technical 1 information than we have available now. (/,g} = 1f(x)g(x)dz. (1.2) When you first learned about vectors in an undergraduate Physics or Math It is an easy matter to verify that (1.2) defines an inner product on the ematics course, you were probably told that vectors have direction and magni vector space of continuous functions. 3 tude. This is certainly true in JR , but in more complicated vector spaces the 3. There are other interesting ways to define an inner product for functions. concept of direction is hard to The direction of a vector is always given visualize. For example, if the complex valued functions I and g are continuously relative to some other reference vector by the angle between the two vectors. differentiable on the interval [0, 1], we might define To get a concept of direction and angles in general vector spaces we introduce 1 the notion of an inner product. (/,g)= 1(!(z)g(x) + /'(z)gl(z)) dz. Definition For the inner product called product 1.5 x, y E S, (also scalar More generally, if I and g are n times continuously differentiable on [0, 1] or dot product) of x andy, denoted (x,lf), is a function{-,·): S x S ~ IR, we might define (or (U if Sis a complex vector space) with the properties: (f,g} = di !(~)dig(~)) dx (1.3) 1. (x,y) = (y,z), (overline denotes complex conjugate) r (t lo \j,.,0 dx1 dxJ 2. (ax,y) = a(x,y}, as an inner product. Indeed, one can show that these satisfy the required properties of inner products. 3. (x+y,z)=(x,z)+(y,z), The magnitude of a vector can also be defined for a general vector space. 4. (x,x) > 0 if x :/: 0, (x,x) = 0 if and only if x = 0. Definition 1.6 For x E S, the norm (also called amplitude, magnitude, or A linear vector space with an inner product is called an inner product length) of x, denoted llxll, is a function 11·11: S-+ [O,oo) with the properties: space. 1. llxll > 0 if x :/: 0 and llxll = 0 implies x = 0, Examples: 2. llaxll = lal·llxll for a E IR (or a E = E x,y, 1 lo•l (L:~= 1 1xkiP) 1P, 1 < p < oo. With p = 2, this is the Euclidean norm or "2"-norm, however, for differing values of p the meaning of size changes. For is the usual Euclidean inner product. H x, Sl e (U" I then example, in JR2 the "unit sphere" llxll = 1 with p = 2 is the circle x~ + x~ = 1 n while with p = 1 the "unit sphere" lx1l + lx2l = 1 is a diamond (a square with (:~:, y) = E :l:loYio (1.1) vertices on the axes). H we let p ~ oo, then llxll = maxk lxkl = llxlloo is called lo=l the "infinity" or "sup" (for supremum) norm, and the "unit sphere" llxll = 1 is is an inner product. a square (Fig. 1.1). A linear vector space with a norm is called a normed vector space. Al 2. For two real continuous functions l(x) and g(z) defined for :1: e [0,1], we though there are many ways to define norms, one direct way to find a norm is discretize l(x) to define a vector in mn IF= (f(xt), l(z2), ... ,/(xn)) and 6 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.1. LINEAR VECTOR SPACES 7 If y =f 0, we pick a= fMtt, (thereby minimizing llx- ay!l2) from which the Schwarz inequality is immediate. 1 Using the Schwarz inequality, we see that the triangle inequality holds for any induced norm, since llx + Yll 2 = llxW + 2Re{ (x, y)) + IIYII 2 (a) (b) (c) 2 ~ llxll2 + 2llxll·llvll + llvll Figure 1.1: The unit sphere llxll = 1 for (a) the "1"-norm, {b) the "2"-norm, (c) the sup norm. = (llxll + llviD2, or if we have an inner product space. If x E Sand (x,y} is an inner product for llx +vii ~ llxll + IIYII· S then llxll = J(x,x} is called the induced norm (or natural norm) for S. 2 For example, in (Jln, the Euclidean norm One can show that in JR , cos(}= ~ where (x, y) is the usual Euclidean inner product and(} is the angle between the vectors x andy. In other real inner n )1/2 product spaces, such as the space of continuous functions, we take this to be 2 , {1.4) the definition of cosO. The law of cosines and the Pythagorean theorem are llxll = ( t; lx~~:l immediate. For example, is induced by the Euclidean inner product {1.1), and for real continuous func llx + vW = llxll 2 + 2llxll . IIYII cos(} + IIYW tions with inner product (/,g)= J: f(x)g(x)dx, the induced norm is is the law of cosines, and if cos() = 0 then b )1/2 2 2 2 llx + Yll = llxll + IIYW (Pythagorean Theorem). 11/11 = ( L1f(x)l dx Following this definition of cos (), we say that the vectors x and y are or For n times continuously differentiable complex valued functions, the inner prod thogonal if (x, y) = 0. uct (1.3) induces the norm Examples: 1. The vector (1, 0) ia orthogonal to (0, 1) in JR? using the Euclidean inner 1\/1\ = ldl£,<;l dz (!.' ~ r r product since ((0, 1), (1, 0)} = 0. 2. With inner product (!,g)= J;.,. f(x)g(x)dx, sinx ap.d sin2x are orthogonal For any induced norm, llxll > 0 if x :/: 0, and llaxll = lal· llxll for scalars a. smce· Jor"J1r smxsm· · 2x dx = 0 . To see that the triangle inequality also holds, we must first prove an important 3. With inner product (!,g)= J1 f(x)g(x)dx, the angle 0 between the func result: 0 tions f(x) = 1 and g(x) = x on [0, 1) is 30° since cosO= ~ = .;'3/2. Theorem 1.1 (Schwarz1 inequality) For x,y in an inner product space, Orthogonal vectors are nice for several reasons. If the vectors { ¢1, ¢2, ... , l(x, v>l ~ llxii·IIYII· {1.5) ¢n} E S, cfoi =f 0, are mutually orthogonal, that is, (¢i,¢;) = 0 fori =f j, then they form a linearly independent set. To see this suppose there are scalars a1, a2, ... , an so that a1cfo1 + a2¢2 + · · · + ancfon = 0. Proof: For x, yES, a a scalar, 2 2 2 2 The inner product of this expression with ¢; is 0 ~ llx- ayll = llxll - 2Re((x, ay}) + lai IIYII • 2 1This theorem is also associated with the names of Cauchy and Bunyakowsky. (¢;, 01¢1 + 02¢2 + · · · + ancfon) = (¢;,a;¢;) = 0:; llc/Jj 11 = 0, 8 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.2. SPECTRAL THEORY FOR MATRICES 9 so that ai = 0 for j = 1, 2, ... , n, provided t/J; is nontrivial. H the set {1/Jt. t/J2, ... , t/Jn} forms a basis for S, we can represent any element f E S as a linear combination of 1/J;, .. ~\), n \ I= Lf3itP3· \, q, 2 j=l \ To determine the coefficients /3;, we take the inner product of f with tPi and find a matrix equation B/3 = '7 where B = (bi;), btj = (1/Jj,tPi}, {3 = ({3i), Xz '1i = (/, t/Jt}. This matrix problem is always uniquely solvable since the tPt 's are linearly independent. However, the solution is simplified enormously when Figure 1.2: Graphical representation of the Gram-Schmidt orthogonalization the tPi 's are mutually orthogonal, since then B is a diagonal matrix and easily procedure to produce tP2 from Xt = c/>1 and x2. inverted to yield (!, tPi} {3 Example: i = llt/Jill2 • 2 Consider the powers of x, {1, x, x , ••• , xn}, which on any interval [a, b] (a< The coefficient f3t carries with it an appealing geometrical interpretation in b) form a linearly independent set. m,n. H we want to qrthogonally "project" a vector f onto ¢ we might imag With [a,b] = [-1, 1] and the inner product ine shining a light onto tjJ and measuring the shadow cast by f. From simple 1 trigonometry, the length of this shadow is II/II cosO where 8 is the angle in the (!,g) = [ f(x)g(x)dx, plane defined by the vectors f and 1/J. This length is exactly ~ and the vector 1 of the shadow, ~1/J, is called the projection off onto 1/J. Notice that the the Gram-Schmidt procedure produces quantity llx- 'i'YII is minimized by picking 'i'Y to be the projection of x onto y, if>o = 1, . ~ 1.e., 'i' = TTUfP"Y· l/>1 =x, l/>2 = x 2 -1/3, The Gram-Schmidt orthogonalization procedure is an inductive tech 3 nique used to generate a mutually orthogonal set from any linearly independent l/>3 = x - 3x/5, set of vectors. Given the vectors Xt, x2, ... , Xn, we set t/J1 = Xt. To find a vector and so on. The functions thus generated are called the Legendre poly t/J2 that is orthogonal to t/J1, we set t/J2 = x2 - a¢1, and use the requirement nomials. Other sets of orthogonal polynomials are found by changing the (t/J2, t/J1) = 0 to determine that underlying interval or the definition of the inner product. We will see more of these in Chapter 2. 1.2 Spectral Theory for Matrices so that {x2,tPt) In a vector space, there are many different possible choices for a basis. The main tP2 = X2- llt/Jtll2 tPt• message of this section and, in fact, of this text, is that by a careful choice of basis, many problems can be "diagonalized." This is also the main idea behind In other words, we get tjJ2 by subtracting from x2 the projection of x2 onto tPt transform theory. (See Fig. 1.2}. Proceeding inductively, we find t/Jn by subtracting the successive Suppose we want to solve the matrix problem projections, n-1 ( ) Ax=b, .J. ~ Xn, tPi t/J; 'l'n = Xn - LJ 111/J 112 ' j=1 j where A is an n x n matrix. We view the entries of the vectors x and b as the coordinates of vectors relative to some basis, usually the natural basis. That is, from which it is clear that (1/J~;, t/Jn) = 0 for k < n. We are left with mutually orthogonal vectors c/>1, t/J2, .•. , c/>n. which have the same span as the original set. n x = (x1,x2, ... ,xn)T = L.x;ef. j=l 10 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.2. SPECTRAL THEORY FOR MATRICES 11 The same vector x would have different coordinates if it were expressed relative Definition 1.7 An eigenpair of A is a pair (A, :z:), A E Examples: Proof: The proof of this statement is by induction. For each eigenvalue >..~:, we let x.~: be the corresponding eigenvector. For k = 1, the single vector 1. The matrix x1 =F 0 forms a linearly independent set. For the induction step, we suppose A=(~!) x1,x:l,· .. ,x.~:-1 are linearly independent. We try to find scalars a1, a2, ... ,a.~: 2 so that has eigenpairs ~~ = XI (1,1)T, and ~2 x2 (1,-1)T • The 4, = =2, = a1x1 + a2x2 + · · · + a~cx1c = 0. vectors X1, x2 are real and linearly independent. The matrix If we multiply this expression by A and use that Axi = AiXi, we learn that T= ( 1 1) 1 -1 a1>.1x1 + a2>.2x2 + · · · + a~c>.~cx~c = 0. gives the required change of basis and If we multiply instead by >..~:, we obtain T- 1AT= ( 40 0)2 . >.~ca1x1 + >.1ca2x2 + · · · + >.~ca~cx~c = 0. 2. The matrix Subtracting the second of these expressions from the first, we find that A= ( c?s(} -sin(} ) smO cos(} a1(>.1- >..~:)xl + · · · + a.~:-1(>..~:-1- >..~:)x.~:-1 = 0. transforms the vector (cos¢, sin c/J)T into the vector (cos(¢+0),sin(¢+0))T and so is called a rotational matrix. This matrix has eigenvalues ~ = It follows that a1 = a2 = ··· = a1c-1 = 0, since xl,X2,···•XA:-l are assumed sin(}± icos(} and eigenvectors (=Fi, l)T. There is no real change of basis to be linearly independent and Ale is distinct from >.1, >.2, ..., Aie-l, and finally which diagonalizes A, although it can be diagonalized using complex basis a.~:= 0 since x~c is assumed to be nonzero. vectors. I 3. The matrix There is another important class of matrices for which diagonalization is A=(~~) always possible, namely the self-adjoint matrices, but before we can discuss these, we must define the adjoint of a matrix. has characteristic polynomial ~ 2 = 0, so there is one eigenvalue .>. = 0 with algebraic multiplicity two. However, there is only one eigenvector Definition 1.8 For any matrix A, the adjoint of A is defined as the matrix x = (1, O)T, so the geometric multiplicity is one. There is no change of 1 A* where (Ax,y) = (x,A*y), for all x, yin (i;ffl. basis which diagonalizes A. If (a;J), the of is We need to determine when there are n linearly independent eigenvectors. As Definition 1.9 A= transpose A AT= (a;,). we see from the last example, the dilemma is that even though the characteristic To find the adjoint matrix A • explicitly, note that (using the Euclidean inner polynomial (1.6) always has n roots, counting multiplicity, there need not be an product) equal number of linearly independent eigenvectors. It is this possible deficiency n,n n (n ) that is of concern. For any eigenvalue >., the geometric multiplicity is at least (Ax, y) = L L aiJXJfii = L Xj L aiJili one, since det(A- >.I) = 0 implies there is at least one nontrivial vector x for i=l j=l j=l i=l which (A- >.I)x = 0. In fact, every square matrix A has at least one eigenpair so that A • AT. For example, if (although it may be complex). = Theorem 1.3 If A has n distinct eigenvalues, then it has n linearly independent au a1:l ) A= a21 a22 , eigenvectors. ( aa1 aa2 2Remark about notation: When dealing with vectors and matrices, it is important to then distinguish between row vectors and column vectors. For example, (1, 2) is a row vector and A • = ( au ~~ aa1 ) . ( ~ ) is a column vector. However, it is often convenient to write column vectors as the Cii2 a:l2 aa:l transpose of a row vector, for example, ( ~ ) = (1, 2)T. Definition 1.10 A matrix A is self-adjoint if A • = A. 14 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.2. SPECTRAL THEORY FOR MATRICES 15 H A is self-adjoint and complex, then A is called a Hermitian matrix We observe that if M is an invariant manifold over the complex scalar field whereas, if A is self-adjoint and real, it is symmetric. Notice that by definition, for some matrix A, there is at least one vector x E M with Ax = ..\x for a self-adjoint matrix is square. some ..\ E a::'. To verify this, notice that since M lies in a k-dimensional space, it has a basis, say {xi. x2, ... , xk}· For any x E M, Ax E M so x and Ax Theorem 1.4 If A is self-adjoint the following statements are true: have representations relative to the basis {xi, x2, ... , Xk}· In particular, take 1. (Ax, x) is real for all x, x == I:~=l aixi and Axi = I:7=l f3jiXj· To solve Ax- ..\x = 0, we must have f. All eigenvalues are real, k k 0 = L:ai(Axi)- ..\ L:aixi 9. Eigenvectors of distinct eigenvalues are orthogonal, i=l i=l 4. There is an orthonormal basis formed by the eigenvectors, k (k ) k k 5. The matrix A can be diagonalized. = ~ai j;f3jiXj- ..\xi = t;xi ~(f3ji- AOij)ai. Observing that {XI, x2, ... , Xk} is a linearly independent set, we have The proof of this theorem is as follows: k 1. H A= A*, then (Ax,x) = (x,A*x) = (x,Ax) = (Ax,x), so (Ax,x) is real. L(f3ii- Mii)ai = 0, j = 1,2, ... ,k, 2. H Ax= Ax, then (Ax,x) = (Ax,x} = ,\(x,x). Since (Ax,x} and (x,x} are i=I real, ,\ must be real. which in matrix notation is (B- ,\l)a = 0 where B = ((3ji) is a k x k matrix, 3. Consider the eigenpairs ( ..\, x) and (#-', y). Then and I is the k x k identity matrix. Of course, as we noted earlier, every square matrix has at least one eigenpair, which concludes the verification . ..\(x, y) = (.\x, y) = (Ax, y) = (x, Ay) = (x, 1-'Y} = J.l(x, y) We are now ready to prove item 4, namely that the eigenvectors of an n x n self-adjoint matrix form an n-dimensional orthogonal basis. Suppose A = A*. so that Then there is at least one eigenpair (..\IJ x 1 ) with Axi = ..\1xi and ..\1 real and (,\- J.l){X, y) 0. = the eigenvector x 1 forms a linearly independent set (of one element). For the H ,\and 1-' are distinct(,\ f: J.l), then (x, y) = 0, i.e., x andy are orthogonal. induction step, suppose we have found k - 1 mutually orthogonal eigenvectors Xi, Axi = AiXi with Ai real fori= 1, 2, ... , k- 1. We form the linear manifold The proof of item 4 requires more background on linear manifolds. Mk = {xi (x, Xj) = 0, j = 1, 2, ... , k - 1 }, Definition 1.11 called the orthogonal complement of the k- 1 orthogonal eigenvectors x1, 1. A linear manifold M C S is a subset of S which is closed under vector x2, ... , Xk-l· This manifold is invariant for A since, ifx E Mk, then (x,xj) = 0 addition and scalar multiplication. and 2. An invariant manifold M for the matrix A is a linear manifold M C S (Ax,xj) = (x,Axj) = ,\j(x,x3) = 0 for which x E M implies that Ax E M. for j = 1, 2, ... , k- 1. Therefore, Mk contains (at least) one eigenvector Xk corresponding to a real eigenvalue ..\k and clearly (xk, Xj) = 0 for j < k, since Examples: Xk E Mk. The eigenvalue Ak is real since all eigenvalues of A are real. In 1. N(A), the null space of A, is the set of all x for which Ax= 0. If x andy summary, we can state the main result of this section: satisfy Ax= 0 and Ay = 0, then A( ax+ {3y) = 0 so that N(A) is a linear manifold. If x e N(A), then Ax= 0 so since A(Ax) = AO = 0, Ax is in Theorem 1.5 (Spectral Decomposition Theorem) If A is ann x n self N(A) as well, hence N(A) is an invariant manifold. adjoint matrix, there is an orthogonal basis {x1, x2, ... , Xn} for which 2. R(A), the range of A, is the set of all x for which Ay = x for some y. 1. Axi = AiXi with Ai real. Clearly R(A) is a linear manifold and it is invariant since if x E R(A) then surely Axe R(A). 2. (xi,Xj) = Dij (orthogonality). 1.3. GEOMETRICAL SIGNIFICANCE OF EIGENVALUES 17 16 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES ,.""'""'' ' I 9. The matrix Q with x; as its jth column vector is unitary, that is, I I ,' I q-1 = Q*. ;; I I I -~ b I X I I 4. Q* AQ =A where A is a diagonal matrix with real entries Ai· I I I I I I I ;' - I I r- I Suppose we wish to solve Ax = b where A • = A. We now know that relative I I I to the basis of eigenvectors, this is a diagonal problem. That is, setting x = Qx', I I ,I I b = Qb' we find that Q* AQx' =Ax' = b'. H all the eigenvalues are nonzero, I 1 I the diagonal matrix A is easily inverted for x' = A- b'. Rewriting this in terms I I 1 I of the original basis we find that x = QA- Q*b. These operations can be I I summarized in the following commuting diagram. I I A-l I 1 I Ax=b ~ X= A- b I ; I ; x=Qx' x' =Q*x I ; t I ;; b=Qb' .t. b' = Q*b ~- A-1 1 Ax'= b' ~ x' = A- b' The solution of Ax = b can be found directly by applying the inverse of A Figure 1.3: Graphical solution of the 2 x 2 matrix equation Ax = b with A given (the top of the diagram) or indirectly in three steps by making a change of coor by (1.7), using the eigenvectors x 1 and x2 of the matrix A as coordinates. dinate system to a diagonal problem (the leftmost vertical descent), solving the diagonal problem (the bottom of the diagram), and then changing coordinates back to the original coordinate system (the rightmost vertical ascent). 1.3 Geometrical Significance of Eigenvalues The geometry of diagonalization can be illustrated on a piece of graph paper. Aside from providing a natural basis in JR" in which to represent linear prob Suppose we wish to solve Ax = b where lems, eigenvectors and eigenvalues have significance relating to the geometry of the linear transformation A. A= ( -~ (1.7) -1 1)1 . For a real symmetric matrix A, the quadratic form q(x) = (Ax, x) produces a real number for every vector x in JR". Since q( x) is a continuous function in IR", We first calculate the eigenvalues and eigenvectors of A and find At = with !, it attains a maximum on the closed, bounded set of vectors x with llxll = 1. eigenvector Suppose the maximum is attained at x = x1 • Then for every unit vector x orthogonal to x , q(x) :::; q(xt). But, on the subset (x,x1) 0, llxll 1, q(x) Xt = ( ~) 1 = = again attains a maximum at, say, x = x2. Continuing inductively in this way we and A2 = -1, with eigenvector produce a set of mutually orthogonal unit vectors, x1, X2, ••• , Xn at which the local extremal values of q(x) are attained on the set llxll = 1. We will show that X2 = ( i)' the x, are the eigenvectors of A and that the values q(xi) are the corresponding eigenvalues. We plot these eigenvectors on a piece of rectangular graph paper. Any vector b can be represented· uniquely as a linear combination of the two basis vectorE Theorem 1.6 (Maximum Principle) If A is a real symmetric matrix and XlJ x2, say b = a1x1 + a2x2. To find the solution x of Ax = b (since x = q(x) = (Ax,x}, the following statements hold: 2a1x1 - a2x2) we take twice the first coordinate of b added to the negative of the second coordinate of b with respect to the eigenvector basis. These arE 1. At = maxllzll=l q(x) = q(x1) is the largest eigenvalue of the matrix A and depicted in Fig. 1.3. x1 is the eigenvector corresponding to eigenvalue Al· As was suggested in the preface, this diagonalization procedure is the basif on which most transform methods work, and is reiterated throughout this text. 2. (inductive statement) Let Ak = maxq(x) subject to the constraints Hopefully, this commuting diagram will become emblazoned in your mind beforE a. (x,x;} = 0, j = 1, 2, ... 'k - 1, the end of this text. GEOMETRICAL SIGNIFICANCE OF EIGENVALUES 19 18 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES changed by reversing the sign of h, which is not permitted. Thus, we must have b. llxll = 1. (Az1 - J.'Zl, h) = 0 for all h. Taking h = Azt - J.'Xt, we conclude that Then .X~c = q(x~c) is the kth eigenvalue of A, .Xt 2: .X2 2: · · · 2: .X~~: and x~c is Azt - J.'Zl = 0, the corresponding eigenvector of A. so that Xt is the eigenvector of A. Furthermore, q(xt) = (Axt, Xt) = p(xlt Xt) = · p, since llxtll = 1 so that max q( x) = p is the eigenvalue and the vector that Examples: maximizes q(x) is an eigenvector of A. To verify the inductive step, we again use Lagrange multipliers and maximize 1. Consider the matrix k-1 A=(~ !)• P~c(x) = (.Az,x)- p((x,x) -1)- L qJ(X,xj), then q(x) = (Az,z) = 3vl + 2111112 + 3vt where J=l where X1,x2, ... , Xk-1 are eigenvectors of A corresponcling to known eigenvalues z = ( t/1 ) t/2 • .X1 2: A2 ... 2: Aie-l, and p, Ctj, j = 1, ... , k- 1 are Lagrange multipliers. Notice that 8?Je> = 0 implies that (x,x) = 1 and 8-?a~z) = 0 implies that (x,XJ) = 0. We change to polar coordinates by setting 111 = cos 6, t/2 = sin 6 so that Suppose x~,: maximizes P~.:(x) so that P~.:(x~.: + h) - P~.:(x~.:) ~ 0. Using that q(6) = 3+sin26. Clearly, maxq(6) = 4 occurs at 6 = 1r/4 and minq(6) = 2 occurs at 6 = -'fr/4. It follows that ~1 = 4, z1 = (~, ~)T and ~2 = 2, (x~c,Xj) = 0 for j = 1, 2, ... , k- 1, we find that .., _ ( 1 1 )T .u2 - V'2• -V'2 • P~,:(x~c +h)- P~,:(x~,:) = 2((.Az~,:,h}- p(x~,:,h}) + (Ah,h)- p(h,h} . 2. In differential geometry, the curvature tensor of a surface is a syiilmeiric 2 x 2 matrix. As a result, the directions of maximal and minimal curvature Following the same argument as before, we require (Ax~~:- pz~,:, h) = 0 for all on any two-dimensional surface are always orthogonal. You can check this h. In other words, Ax~~: - J.'Xk = 0 and x~~: is an eigenvector of A. Finally, out by a careful look at the shape of your forearm, for example. q(x~.:) = (Az~~:,x~~:) = p(x~c,x~.:) = p since llx~~:ll = 1 so that maxq(x) is the eigenvalue Ale and the vector that maximizes q(x) is an eigenvector of A. Clearly To understand the geometrical significance of the maximum principle, we Ale ~ Alc-t ~ .. · ~At· I view the quadtatic form q(x) = (.Az,x) as a radial map of the sphere. That is, to each point on the sphere llxll = 1, associate q(x) with the projection of Ax From the maximum principle we can find the kth eigenvalue and eigenvector onto x, q(x)x = (Ax, x)x. H A is positive definite (a positive definite matrix A is only after the previous k - 1 eigenvectors are known. It is sometimes useful to one for which (Ax, x) > 0 for all x :/= 0), the surface q(x)x looks like an ellipsoid have a characterization of Ale that makes no reference to other eigenpairs. This (an American or rugby football) in JRn. (It is not exactly an ellipsoid; the level is precisely the motivation of surface q(x) = 1 is an ellipsoid.) The maximum of q(x) occurs along the major Theorem 1.7 (Courant Minimax Principle) For any real symmetric ma axis of the football-shaped object. H we intersect it with a plane orthogonal to trix A, the major axis, then restricted to this plane, the maximum of q(x) occurs along .X~c =min max (.Az,x), the semi-major axis, and so on. C llzll=l To verify statement 1, we apply the method of Lagrange multipliers to q(x). Cez:O Specifically, we seek to maximize the function Pt(x) = q(x)- p((x,x) -1). H where C is any (k- 1) x n matrix. the maximum occurs at x11 then P1(x1 +h) -Pt(xt) ~ 0 for any h. Expanding Pt(Xt +h) we find that The geometrical idea of this characterization is as follows. For any nontrivial matrix C, the constraint Cx 0 corresponds to the requirement that x lie on Pt(Xt +h)- Pt(Xt) = 2{(Azt, h)- p(x1, h))+ (Ah, h)- p(h, h). = some n - k + 1 dimensional hyperplane in JRn. We find A~,: by first maximizing q(x) with llxll = 1 on the hyperplane and then we vary this hyperplane until Notice that (Az1 , h) - p(x1 , h) is linear in h and (Ah, h) - p(h, h) is quadratic in h. The sign of this quadratic expression cannot be changed by changing the this maximum is as small as possible. sign of h. However, unless the linear expression is identically zero, its sign can For example, imagine slicing a football with a plane through the origin to be changed by reversing the sign of h. Thus, if h is sufficiently small, and the get an ellipse, and finding the principal axis of the ellipse, then varying the orientation of the slicing plane until the length of the principal axis on the slice linear expression is not identically zero, the sign of P 1(x1 +h)- Pt(Xt) can be 20 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.3. GEOMETRICAL SIGNIFICANCE OF EIGENVALUES 21 plane is minimized. The slice that minimizes the principal axis contains the two Proof: According to the minimax principle, for any quadratic form q, the smaller eigenvectors of A, but not the largest, so we find (x2, ..X2). relative maxima are given by Proof: To prove the minimax principle we note that since A is symmetric, there is a unitary matrix Q that diagonalizes so that A = Q AQ* where A is the A, ..Xj =min {max q(x)}, diagonal matrix of eigenvalues ..X1 ? ..X2 • • • ? ..Xn. It follows that 0 llzll=l O:z:=O n where Cx = 0 represents j- 1linear constraints on x. If Xi, i = 1, 2, ... ,j- 1, (Ax,x} = (QAQ*x,x} = (Ay,y} = 'E..Xiy~, are the vectors for which the first j - 1 relative maxima of (j(x) are attained, i=l then the minimum for q(x) is attained by taking the rows of C to be xi, and where y = Q*x and Cx = 0 if and only if CQy =By= 0, where B = CQ. We (restricting x so that llxll = 1) define p. by ~i =min {max q(x)} = Il_!ax q(x) = max q(x) ~ ..Xi+k• 0 Oz=O (z,z;)=O (z,x;)=O Bz=P p. =min { max (Ax,x}} =min {max i: ..XiYl}. 0 11:~:11=1 B 111111=1 i=1 since the combination of Cx = 0 and Bx = 0 places j + k- 1 constraints on Oz=O .811=0 q(x). Similarly, (restricting x so that llxll = 1) If we pick the matrix B so that By = 0 implies that Y1 = Y2 = · · · = Yk-1 = 0, ~i =min { max q(x)} $ max q(x) $ max q(x) = ..Xj, then p. :::; max11 11 11=1l:~=k ..Xiy[ = ..Xk. Now pick a different matrix B. Since B 1/i (z,l/;)=0 (z,x;}=O (x,z;)=O has rank $ k - 1, the solution of By = 0 is not unique, and we can satisfy the equation By = 0 with then- k additional restrictions YH1 = Yk+2 = · · · = where XiJ i = 1, 2, ... , j - 1 are those vectors at which the first j - 1 relative Yn = 0. With these n- k restrictions, the augmented system, call it BaY= 0, maxima of q(x) are attained. 1 has rank$ k- 1 + (n- k) = n- 1 so always has nontrivial solutions. For this For a geometrical illustration of these results, consider a 3 x 3 positive definite restricted set of vectors (i.e., with YH1 = Yk+2 = · · · = Yn = 0) the maximum matrix A and its associated quadratic form q(x) = (Ax,x}. If we view q(x) as for each B might be reduced. Therefore, (restricting y so that IIYII = 1) the "radial" map of the unit sphere llxll = 1, x -t q(x)x, the surface thus constructed is football shaped with three principal axes, the major axis, the semi-major axis, and the minor axis with lengths ..X ~ ..X ? ..Xa, respectively. p. = min {max t ..Xiy[ ~~ min { m~ t ..Xiy[} 1 2 B B11=0 i=l J . B B.,l/=0 i=1 If we slice this football-shaped surface with any plane through the origin, the intersection of q(x) with the plane is "elliptical" with a major and minor axis with lengths :X1 ?. :X2, respectively. It is an immediate consequence of the ? min {~ax ..xk tyr} = ..xk. preceding theorem that these are interlaced, B B.,l/=0 . =1 ..Xt ~ ~1 ?. ..X2 ?. ~2 ~ ..Xa. Since p.? ..Xk and p.:::; ..Xk, it must be that p. = ..Xk, as proposed. I Using the minimax principle we can estimate the eigenvalues of the matrix Example: A. To illustrate the usefulness of these results, suppose we wish to estimate the eigenvalues of the symmetric matrix · Theorem 1.8 Suppo6e the quadratic fonn q(x) = (Ax,x} is constrained by k linear constraints Bx = 0 to the quadratic fonn q in n - k variables. The 1 2 3 ) A= 2 3 6 relative extremal values of q, f!.enot~ ..Xt ? ..X2 ? .... · · · ? ..Xn and the relative ( 3 6 8 extremal values of q, denoted ..X1 ? ..X 2 ~ · · · ? ..Xn-k satisfy the interlacing inequality using the minimax principle. First notice that if we increase the diagonal ..Xj ~ ~j ~ ..xi+k• j = 1, 2, ... , n - k. elements of A a bit, we get the modified Illatrix 1 2 3 ) A= 2 4 6 ( 3 6 9 22 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.3. GEOMETRICAL SIGNIFICANCE OF EIGENVALUES 23 u which is of rank 1 and therefore has two zero eigenvalues. The range of A is j-1 ui __..,uf+l spanned by the vector (1, 2, 3)T and A has the positive eigenvalue ..\1 = 14. It follows (see Problem 1.3.2) that the eigenvalues of A satisfy ..\1 ~ 14, ..\2 ~ 0, - ..\a~ 0. To use the minimax principle further we impose the single constraint :Ct = 0. m With this constraint the extremal values of q(:c) are the eigenvalues of the j-1 mi mf+J diagonal submatrix A=~ ( 36 6) 8 . Figure 1.4: One dimensional lattice of balls connected by springs. For this matrix we calculate that ~ = 12 (with eigenvector (2, 3)T), ~2 = -1 1 where u is the n-vector with components u1, u2, ... , Un and the n x n matrix A (with eigenvector (3,-2)T), so that ..\a~ -1 ~ ..\ ~ 0, and ..\1 ~ 12. 2 is tridiagonal with entries To further improve our estimate of ..\1, observe that q(:c) = 12 for . . - kj+ki-1 . - 1 a3,3 -- mi , J - , .•. , n, kj-1 • 2 aJJ-1 = m , J = , ... , n, ·=_,b(!)· k j aJ,J+l = .;;;, j = 1. ... , n- 1, We try the nearby unit vector and all other entries are zero. Of course, one could envision a more complicated lattice for which connections are not restricted to nearest neighbor interactions in which case the matrix A would not be tridiagonal. Something to try with •=;.. 0) the system (1.8) is to assume that the solution has the form u = 4Jeiwt in which case we must have and find that q(:c) = ¥/ = 13.07 so that 13.07 ~ ..\t ~ 14. A4J = -w24J. An improved estimate of ..\2 is found by imposing the single constraint :c2 = 0 Thus, the eigenvalues -w2 of the matrix A correspond to the natural to get the restricted quadratic form q(:c) = :c~ + 6:ct:ca + 9:c~. For this >. = quadratic form the extremal values are ~· It follows that -0.1098 ~ frequencies of vibration of this lattice, and the eigenvectors 4J determine the shape of the vibrating modes, and are called the natural modes or normal .tfB ~ ..\2 ~ 0. These estimates are actually quite good since the exact eigenvalues are ..\ = -1, -0.076, and 13.076. modes of vibration for the system. The matrix A for the simple lattice here is symmetric if and only if the To understand further the physical significance of eigenvalues, it is useful to masses m; are identical. H the masses are identical we can use the minimax think about vibrations in a lattice. By a lattice we mean a collection of objects principle to draw conclusions about the vibrations of the lattice. (balls or molecules, for example) which are connected by some restoring force H mi = m for j = 1, 2, ... n, we calculate that (such as springs or molecular forces). As a simple example consider a one n n dimensional lattice of balls connected by linear springs (Fig. 1.4). Let m; and m(Au, u} = m L L aiJUiUJ u; denote the mass and displacement from equilibrium, respectively, of the jth i=l j=1 n-1 ball. The equation of motion of the system is given by 2 = -kou~- knu;- L kj(Uj- UJ+1) . cflui j=l ffij dt =k;(Uj+l-Uj)+kj-1(Uj-1-Uj)• 2 Since the constants ki are positive, A is a negative definite matrix (a negative Here we have assumed that the restoring force of the jth spring is linearly pro definite matrix is one for which (Ax,x} < 0 for x ::f: 0). Notice further that portional to its displacement from equilibrium with constant of proportionality increasing any kj decreases the quantity {Au, u} and hence, the eigenvalues of ki > 0 (i.e., Hooke's law). A are decreased. From the minimax principle we have some immediate conse Suppose that the masses at the ends of the lattice are constrained to remain quences: fixed so that u0 = Un+l = 0. This system of equations can be written as 1. H the mass m is increased, all of the eigenvalues of A are increased, that is, made less negative, so that the natural frequencies of vibration are rPu =Au, (1.8) decreased. dt2 24 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.5. LEAST SQUARES SOLUTIONS-PSEUDO INVERSES 25 2. If any of the spring constants k; is increased then the eigenvalues of A The first half of the Fredholm alternative (Theorem 1.9) is easily proven. If are decreased (made more negative) so that the natural frequencies of A*v = 0 and xo satisfies Axo = b then vibration are increased. (b, v) = (Axo, v) = (xo, A"v) = (xo, 0) = 0. The natural frequencies of vibration of a lattice are also called the resonant frequencies because, if the system is forced with a forcing function vibrating To prove the last half, suppose (v, b) = 0 for all v with A*v = 0. We write at that frequency, the system resonates, that is, experiences large amplitude b = br + bo where br is the component of b in the range of A and bp is the oscillations which grow (theoretically without bound) as time proceeds. This component of b orthogonal to the range of A. Then 0 = (bo, Ax) = (A*b0 , x) observation is the main idea behind a variety of techniques used in chemistry for all x, so that A"bo = 0. Since (b, v) = 0 for all v in the null space of A*, (such as infrared spectroscopy, electron spin resonance spectroscopy and nu and since bo is in the null space of A*, we conclude that (b0 , b) = 0. Expanding clear magnetic resonance spectroscopy) to identify the structure of molecules in b = br+bo we find 0 = (bo,br+bo) = (bo,br)+(bo,bo). Since (b0 ,br) = 0 it must liquids, solids, and gases. be that bo = 0 so that b = br is in the range of A, i.e., Ax= b has a solution. 1 It is also the idea behind microwave ovens. If an object containing a water molecule is forced with microwaves at one of its resonant frequencies, the water molecule vibrates at larger and larger amplitudes, increasing the temperature Example: of the object. As you may know, microwave ovens do not have much effect on objects whose resonant frequencies are much different than that of water, but To illustrate these two theorems in a trivial way, consider the matrix on objects containing water, a microwave oven is extremely efficient. Microwave ovens are set at frequencies in the 1011hz range to resonate with the rotational frequencies of a water molecule. A=(!~)· The null space of A is spanned by the vector (2, -l)T so that solutions of 1.4 Fredholm Alternative Theorem Ax = b, if they exist, are not unique. Since the null space of A • is spanned by the vector (3, -l)T, solutions of Ax= b exist only if b is orthogonal to v The Fredholm Alternative Theorem is arguably the most important theo so of the form rem used in applied mathematics (and we will see it many times in this text) as it gives specific criteria for when solutions of linear equations exist. b=a(!), to Suppose we wish solve the matrix equation Ax = b where A is an n x m which is no surprise since (1, 3)T spans the column space of A. matrix (not necessarily square). We want to know if there is a solution, and if so, how many solutions are possible. · One useful restatement of the Fredholm alternative is that the null space of Theorem 1.9 (Fredholm Alternative Theorem) The equation Ax = b has A* is the orthogonal complement of the range of A and that together they span a solution if and only if ( b, v) = 0 for every vector v satisfying A • v = 0. IR.m, so that IRm = R(A) EB N(A*). Stated another way, any vector b E JRm can be written uniquely as b = br + bo where br is in the range of A and b0 is in the null space of A•, and br is orthogonal to b0 (see Fig. 1.5). It is interesting to note that this theorem is true for any inner product on mn I although if one changes the inner product, the adjoint matrix A. changes as do the vectors v for which A*v = 0. 1.5 Least Squares Solutions-Pseudo Inverses Theorem 1.10 A solution of Ax b (if it emts) ~ unique if and only if x = 0 = Armed with only the Fredholm alternative, a mathematician is actually rather ~the only solution of Ax= 0. dangerous. In many situations the Fredholm alternative may tell us that a system of .equations Ax = b has no solution. But to the engineer or scientist Proof: We prove Theorem 1.10 first. Suppose Ax = 0 for some x I- 0. If who spent lots of time and money collecting data, this is a most unsatisfactory Axo = b then Xt = xo +ax also satisfies Ax1 = b for any choice of a so that the answer. Surely there must be a way to "almost" solve a system. solution is not unique. Conversely, if solutions of Ax = b are not unique then Typical examples are overdetermined systems from curve fitting. Suppose there are vectors Xt and x2 with x = x1 - x2 I- 0 satisfying Axt = Ax2 = b. the data points (ti, Yi), i = 1, 2, ... , n, are thought to be linearly related so that Clearly Ax = A(xt - x2) = Axt _: Ax2 = 0. Yi = a(ti -l) + /3, where t = ~ L:~=l ti. This is a system of n linear equations 26 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.5. LEAST SQUARES SOLUTIONS-PSEUDO INVERSES 27 We have control only over the selection of x, so the minimum is attained when we minimize II Ax - brW. Since br is in the range of A, Ax = br always has a solution and minx IIAx- bll = llboW· We know that br is the projection of b onto the range of A and bo is in the orthogonal complement of the range of A. Therefore, 0 = (bo,Ax) = (A•bo,x) for all x so that A•bo = 0. Now Ax= br is equivalent to Ax = b - bo so that Ax - b = -bo must be in the null space of A•, which is true if and only if A• Ax= A•b. Thus the least squares solution of Ax = b is any vector x satisfying A • Ax = A • b. One such x always exists. The equation A•Ax = A•b (1.10) is called the normal equation. Another derivation of the equation (1.10) uses ideas of calculus. Since we wish to minimize IIAx- bll 2 =(Ax- b,Ax- b}, we let x = xo + ay, where xo is the minimizing vector and y is an arbitrary perturbation. Since xo renders IIAx- bll 2 a minimum, (A(xo + ay)- b,A(xo + ay)- b)~ (Axo- b,Axo- b) for ay nonzero. Expanding this expression, we find that we must require Figure 1.5: Graphical interpretation of the Fredholm alternative theorem. a(y, A•(Axo- b)}+ a(A.(Axo- b), y} + a 2 (Ay, Ay} ~ 0. in the two unknowns a and /3 of the form This last expression is quadratic in a, and unless the linear terms vanish, for sufficiently small a, a change of sign of a produces a change of sign of the entire expression, which is not permitted. Thus we require (y, A*(Axo- b)} = 0 for t1 - t 1 ) ( Yl ) ally. That is, we require A• Axo = A*b, which is the normal equation. A(~)= t.~: ~ (~)= ~ . (1.9) We know from our first derivation that the normal equation always has a ( solution. Another check of this is to apply the Fredholm alternative theorem tn-t 1 Yn directly to (1.10). We require (v,A"'b} = 0 for all v with A• Av = 0. (Notice that If there are more than two data points this system of equations does not A• A is a square, self-adjoint matrix). If A* Av = 0 then Avis simultaneously in general have an exact solution, if only because collected data always contain in the null space of A • and in the range of A. Since these two subspaces are error and the fit is not exact. orthogonal, Av must be 0, the only element common to both subspaces. Thus Since we cannot solve Ax = b exactly, our goal is to find an x that minimizes (v, A•b) = (Av, b) = 0, as required. As an example of the least squares solution, consider the curve fitting prob IlAx- bll. The minimal solution is norm dependent. One natural, and common, choice lem described earlier. We seek a least squares solution of (1.9). Premultiplying of norm is the Euclidean norm (1.4) while other choices such as llxlloo or llxll = by A*, we find (use that E~= 1 (t,- f)= 0) E~= 1 lxsl = llxlh are useful as well, but lead to different approximation schemes. n n a '.L:(t,- f) 2 = '.L:Yi(ti- f), Definition 1.12 For an m x n matrix A and vector b E To find the least squares solution, recall from the Fredholm alternative that b which is always solvable for a and /3. Other types of curve fitting are possible by changing the underlying as can always be written as b = br + bo, where br is in the range of A and bo is orthogonal to the range of A. Since Ax - br is in the range of A then (by the sumptions. For example, one could try a quadratic fit Yi = at~+ j3t, + 'Y· Other Pythagorean theorem) choices of basis functions (such as are discussed in Chapter 2) can also be used, but in all cases we obtain a linear least squares problem. For the exponential IIAx- bll2;, IIAx- brll2 + llboll2· fit y1 = aef3t,, the parameters a, /3 occur nonlinearly. However, the equation 28 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.5. LEAST SQUARES SOLUTIONS-PSEUDO INVERSES 29 ln Yi = ln a + /3ti is linear in ln a and /3 and leads to a linear least squares problem. The least squares solution always exists, however, it is unique if and only if R(A*) A* Ax= 0 has only the trivial solution x = 0. H A*(Ax) = 0 then AxE N(A*) and Ax E R(A) so that Ax = 0. Thus, A• A is invertible if and only if Ax = 0 has no nontrivial solutions, that is, if A has linearly independent columns. H A has linearly independent columns, then the least squares solution of Ax = b is given N(A) by x = (A* A)-1 A*b, called the Moore-Penrose least squares solution. The matrix A'= (A* A)-1 A* is called the Moore-Penrose pseudo-inverse of A. Notice that when A is square and invertible, A* is also invertible and (A* A)-1 A* = A-1 A*-1 A* = A-1 so that the pseudo-inverse of A is precisely the inverse of A. Notice also that the linear fit discussed earlier has a unique solution if there are at least two distinct data points ti (so that the columns of X.' Ax=br A are linearly independent). H A does not have linearly independent columns, the least squares problem Figure 1.6: Graphical solution of the least squares problem for the matrix (1.11). does not have a unique solution. Suppose Xp is one solution of the least squares Every vector in the manifold x : Ax = b,. minimizes IIAx- bll· The smallest problem A • Ax = A • b. Since A has linearly dependent columns, there are vectors such vector, denoted x', must be orthogonal to N(A). w that satisfy Aw = 0, and x = Xp + w is also a least squares solution. One reasonable way to specify the solution uniquely is to seek the smallest possible onto a vector in R(A) (orthogonal to N(A*)), and then find the inverse image A* Ax= A*b. x solution of To minimize llxll, we want to satisfy (x,w} = 0 for of this vector somewhere in R(A•) (orthogonal to N(A)). (See Figure 1.6.) all w with Aw = 0. That is, we want x to be orthogonal to the null space of A, That is, x must have the form and therefore, in the range of A •. Definition 1.13 The least squares pseudo-inverse of A is the matrix A' x=a(!) for which x = A'b satisfies: and must satisfy 1. A* Ax = A*b. Ax = ( :~ ) - ~ ( _; ) ( ( _; ) ' ( :~ ) ) = ~ ( ~ ~ ) ( :~ ) . 2. (x, w) = 0 for every w satisfying Aw = 0. Since Example: A(!)=(~). The geometrical meaning of the pseudo-inverse can be illustrated with the it follows that simple 2 x 2 example (~b1 + ~b2) A ( = ~ ( ~ ) + ~ ( ~ ) b2 = ~ ( ~ ~ ) ( ~ ) (1.11) ! ) b1 A=(!!)· so that For this matrix, 2 1 ) ( 1 ) ( 2/5 1/5 ) ( bl ) X = ( Sbl + Sb2 1 = 2/5 1/5 b2 R(A) = span { ( l )} • R(A*) =span { ( ~ ) } ' and that the pseudo-inverse of A is I ( 2/5 1/5 ) A = 2/5 1/5 . N(A*) =span{ ( _; ) } · There are a number of different numerical algorithms one can use to calcu The pseudo-inverse of A must project the vector late .the pseudo-inverse A'. In most situations it is not a good idea to directly calculate (A* A)-1 since, even if A has linearly independent columns, the calcu ·b=(:~) lation of (A* A)-1 is numerically poorly posed. The algorithms that we describe 30 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.5. LEAST SQUARES SOLUTIONs-PSEUDO INVERSES 31 below are included because they illustrate geometrical concepts, and also, in the the k vectors {uj} form an orthonormal basis for the null space of A•. Thus, case of the last two, because of their usefulness for numerical calculations. the projection matrix P is given by P = I- 2::=1 u,uf. Notice that if the null space of A• is empty, then k = 0 and P =I. If the null space of A is empty we Method 1: Gaussian Elimination row reduce the augmented matrix [A, P], but if not, we append the additional constraint (w,, x} = 0 for each linearly independent w, E N(A), and row reduce The first idea is to mimic Gaussian elimination and find A' via ele the augmented matrix mentary row operations. Recall that Gaussian elimination is the process of reducing the augmented matrix [A, I] to [J,A-1] by means of elementary row operations. The goal of reduction is to subtract appropriate multiples of one row from lower rows of the matrix so that only zeros occur in the column directly [fr ~]· wT below the diagonal element. For example, one step of the row reduction of the I 0 matrix (equivalent to the system of equations Ax = Pb, {w1, x) = 0, i = 1, ... , l), au a12 B1n discarding all zero rows, which leads to [I, A']. a22 B2n To see that this can always be done, notice that the projection P is guaran teed by the Fredholm alternative to project onto the span of the columns of A. Furthermore, the rows of A span the range of A • and the vectors {Wi} span the A= BA:A: BA:n null space of A, so that, from the Fredholm alternative JRn = R(A•) E9 N(A), BA:+1,A: BA:+1,n the rows of A augmented with {wf} span mn and hence can be row reduced to an identity matrix (with possibly some leftover zero rows). 0 BnA: ann Example: is accomplished by premultiplying A by the matrix (only nonzero elements are Consider the matrix displayed) 2 4 6) 1 A= ( 1 2 3 . Since A has linearly dependent columns, A has no Moore-Penrose pseudo inverse. The null space of A is spanned by the two vectors 1 M.~:= I -O'A:+l 1 w,=(_D· --=( -0· -0'1:+2 1 and the null space of A • is spanned by the single unit vector -O'n 1 u= ~ ( -~ )· where O'J = BJA:/a,~:,~:. Specifically, the nonzero entries of M.~: are ones on the AB a result, diagonal and -ui in the kth column, below the diagonal. The numbers ui 1 ( 1 -2 ) 1 ( 4 2 ) are called multipliers for the row reduction. The result of a sequence of such p = I- 5 -2 4 = 5 2 1 . row operations is to triangularize A, so that MnMn-1 · · · M1A = U is an upper We row reduce the augmented matrix triangular matrix. The product of elementary row operations MnMn-1 · · · M1 = L-1 is a lower triangular matrix which when inverted gives LU. This 4/5 A = A representation of A is called an LU decomposition. p) (2 4 6 2/5 1/52/5) wr 0 - 1 2 3 0 . By analogy, our goal is to find some way to augment the matrix A with ( wf 0 - 1 1 -1 0 another matrix, say P, so that the row reduction of [A, P] yields the identity 2 -1 0 0 0 and the pseudo-inverse [I, A']. The least squares solution we seek must satisfy Elementary row operations reduce this matrix to Ax= br, subject to the conditions {w,,x) = 0 where {wi} spans the null space 1 0 0 2/70 1/70 ) of A, and the vector br is the projection of the vector b onto the range of A. We 0 1 0 4/70 2/70 1 can represent. br as br = Pb = b.- Li=tA: {u,, b)u, = ( I- Li=tA: u,u,T) b, where ( 0 0 1 6/70 3/70 32 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.5. LEAST SQUARES SOLUTIONS-PSEUDO INVERSES 33 (we have discarded one identically zero row) so that Eliminating the last row of U and last column of L (which are superfluous) we determine that A' = _!_ ( ~ ~ ) . 70 6 3 A = LU = ( ) ( 1 2 ) . ~5 2~ 0 -2 It is easy to check that A • AA' = A •. Notice that since L has linearly independent columns, the null space of A is the same as the null space of U and similarly, since the rows of U are linearly independent, the null space of A* is the null space of L*. The range of A is Method 2: L U Decomposition spanned by the columns of L and the range of A • is spanned by the columns of U*. If A A •, then U o:L * for some scalar o:. Any matrix A can be written as A = LU where L is lower triangular and = = Once the LU decomposition is known, the pseudo-inverse of A is easily found. U is upper triangular (with the proviso that some row interchanges may be necessary). In practice, A is usually not reduced to a diagonal matrix, but Theorem 1.11 If A = LU, then simply to triangular form. The decomposition is unique if we require L to have ones on the diagonal and U to have linearly independent rows, except for A'= U*(UU*)-1 (L*L)-1U. possible zero rows. To see why the LU decomposition is desirable, suppose we want to solve Ax= band we know that A= LU. We let Ux = y so that Ly =b. Solving This statement can be directly verified by substituting x:::: A'b into A* Ax= Ly = b involves only forward substitution, and solving U x = y involves only A*b and using that A= LU. Since U has linearly independent rows and L has backward substitution. Thus, if both L and U are full rank, the solution vector linearly independent columns, UU* and L* L are both invertible. Notice also x is easily determined. that if Aw = 0, then Uw = 0 and The LU decomposition is found by direct elimination techniques. The idea {w, A'b) {w, u•wu•)-1 (L* L)-1 L*b) is to reduce the augmented matrix [A, I] to the form (U, L -l] by lower triangu = (Uw, (UU*)-1(L* L)-1 L*b) 0, lar elementary row operations. When implementing this on a computer, to save = = memory, one can simply store the multipliers from the row reduction process so that x is the smallest possible least squares solution. The usefulness of this in the sub diagonal element that is zeroed by the row operation (see Prob representation of A' is primarily its· geometrical interpretation. lem 1.5.12). The sub-diagonal half of the resulting matrix is the sub-diagonal part of L and the diagonal elements of L are all ones. If A has linearly in Method 3: Orthogonal Transformations dependent rows, then zero rows of U and corresponding columns of L can be harmlessly eliminated. Another important way to solve least squares problems is to transform the matrix A to upper triangular form using carefully chosen transformations. The advantage of transforming to a triangular system is that if the diagonal elements Example: are nonzero, it can be solved by backward substitution, which is not much more difficult than solving a diagonal system. However, there is an additional bonus Consider the matrix if the matrix transformations we use are orthogonal matrices. Definition 1.14 A square complex matrix Q is unitary (also called orthog 1 A=u n onal) if Q* = Q- , i.e., Q*Q = I. By row reduction, storing multipliers below the diagonal, we find that A row (Recall that unitary matrices first appeared in Thm. 1.5.) Unitary matri reduces to ces have two important geometrical properties, namely they preserve angles and lengths, that is, for any vectors x and y, (Qx, Qy) = (x, Q*Qy) = (x, y), (: -~) and IIQxll = llxll. In other words, unitary transformations are rotations and reflections. so that The transformation we seek represents A as A= QR where Q is unitary and L= 13 0 1 0)0 . R is upper triangular, and this is called the QR decomposition of A. This de U=O -D· ( 5 2 1 composition can always be found because this is exactly what the Gram-Schmidt 34 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.5. LEAST SQUARES SOLUTIONS-PSEUDO INVERSES 35 I I I I I X u = x+ llxlle u = x -llxlle I 1 I ____ ... - I llxll e I llxlle e 1 I I I I ------Uv I -Pv Figure 1.8: Sketch of the geometry of a Householder transformation applied to Figure 1. 7: Sketch of the geometry of an elementary reflector, where Pv = f 1:ii~ u the vector u = x±llxllel. The two choices of a= ±llxll give different reflections. is the projection of v onto u. Proof: 2 2 procedure does, namely, it represents a vector as a linear combination of pre Since x f. -cre1, u f. 0 and U is an elementary reflector. Since cr = llxll , viously determined orthogonal vectors plus a new mutually orthogonal vector. 2 2a2 2crxl. Here, the columns of Q are the orthogonal basis that results from applying the llull = + Gram-Schmidt procedure to the columns of A. The matrix R indicates how to where X1 = (x, e1) is the first component of x, and express the columns of A as linear combinations of the orthogonal basis Q. To carry this out we could use the Gram-Schmidt procedure, however, it uu*x (x + cre1,x)u U X = X - 2 = X - = X - U = -CTel. turns out that this is a numerically unstable process. Instead, we use a method U + O"Xl CT2 + CTX1 that looks similar to Gaussian elimination, which, instead of using elementary row operations as in Gaussian elimination, uses an object called an elementary It is wise (for numerical reasons, to avoid cancellation errors) to choose a to reflector, or more commonly, Householder transformation. have the same sign as Xt, so that x f. -cre1 unless x = 0 already, in which case, the transformation is not needed. I Definition 1.15 An elementary reflector (Householder transformation) is a The geometry of Theorem 1.12 is depicted in Fig. 1.8. matrix of the form We now see that we can place zeros in the first column of A, below the diag 2uu• onal, by premultiplying A by the appropriate Householder transformation. To provided u ':/: 0. U =I -llull2' carry out the transformation on the entire matrix, we do not actually calculate the matrix U but rather note that for any column vector y It is easy to verify that U = u• and u•u = I. The identification of the Householder transformation as a reflector comes from the fact that if v is parallel Uy = Y _ 2uu•y = y _ 2(u,y) u, to u, then Uv = -v, while if vis orthogonal to u, then Uv = v. Thus, if Pv is u*u (u,u) the projection of v onto u, then v = Pv +(I- P)v, and Uv = -Pv +(I- P)v. In other words, U "reflects" the vector v through the plane that is orthogonal so that U y can be viewed as a linear combination of two column vectors with to u. A sketch of this is shown in Fig. 1. 7. a multiplier related to the inner product of the two vectors. Premultiplying A useful property of elementary reflectors is that, as is also true of elementary by a sequence of Householder transformations, we sequentially triangularize the row operations, they can be chosen to introduce zeros into a column vector. matrix A. We first apply the Householder transfor:rp.ation that places zeros in the first column below the diagonal. Then we apply a second Householder Theorem 1.12 For z E IEr', let u = ±llzll, and suppo1e z ':/: -ue1. If u = transformation that leaves the first row and first column undisturbed and places z + ue1, then U = I - ~ u an elementary reflector whose action on z u zeros in the second column below the diagonal, and so on sequentially. Note Ux -ue1 (e1 u the first natural basis vector). that if = 2UkUk uk = h- llukll 2 36 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.5. LEAST SQUARES SOLUTIONS-PSEUDO INVERSES 37 is a k x k Householder transformation that places zeros in the first column below this cannot happen, since the transformation that diagonalizes Mo is not or the diagonal of a k x k matrix, then the n x n matrix thogonal. However, if Mo is not symmetric, then Mn converges to a "nice" matrix whose eigenvalues are easy to find. In-k 0 ) Householder transformations provide a very efficient way to find the QR V~c = ( 0 U~c decomposition of a matrix. Good software packages to do these calculations on a computer are readily available, and a good description of the corresponding is the Householder transformation for an n x n matrix that transforms only the theory is found in [103). lower diagonal k x k submatrix and leaves the remaining matrix unscathed. H Vi, V:l,· · ·, Vn are the chosen sequence of transformation matrices, then Method 4: Singular Value Decomposition {SVD) with Q* = Vn Vn-1 · · · V1 we have The final method by which A' can be calculated is the singular value IIAx- bll = IIQ* Ax- Q*bll = II&- Q*bll, decomposition. This method extends the idea of the spectral decomposition of a matrix to nonsquare matrices and is often used in applications. where As we will see, the singular value decomposition of a matrix A is similar to 1 the spectral representation of a square matrix A = TAT- • Once the spectral R= ( ~)' decomposition is known (if it exists), the inverse is easily found to be A-1 = TA-1T-1 , provided the diagonal matrix A has only nonzero diagonal entries. with R upper triangular. H the columns of A are linearly independent (the This does not, however, work to find a pseudo-inverse. same condition as required for the Moore-Penrose pseudo-inverse) the matrix R If A is self-adjoint, then the spectral decomposition of A is A= QAQ* where is square, has nonzero diagonal elements, and is therefore invertible. Q is orthogonal. To find the pseudo-inverse of A, we want to minimize H we denote the vector IIAx- bll:l = IIQAQ*x- bll = IIAQ*x- Q*bll = IIAy- Q*bll, (1.12) Q*b= (~)I where y = Q*x. It is an easy matter to verify (see Problem 1.5.2) that the where the vector b1 has the same number of elements as R has rows, then pseudo-inverse of an m x n diagonal matrix D of the form the least squares solution is found by solving Rx = b1 and for such an x, IIAx- bll = 11~11, and this error cannot be made smaller. It follows that the D=(D 0) pseudo-inverse of A is given by A'= R'Q*, where R' is the pseudo-inverse of R. 0 0 ' This method is most useful in the case that the matrix A has linearly inde where D is a (square) diagonal matrix with nonzero diagonal elements is the pendent columns so that R is invertible and the solution of Rx = b1 is found by back substitution. H, however, R is not invertible, one must find the smallest n x m diagonal matrix solution of 1ix = i1]., which, unfortunately, cannot be found by back substitution. D' Dt = ( 0 0)0 . There is another important use for Householder transformations that should be mentioned here. As is well known, there is no way to find the eigenvalues It follows that the pseudo-inverse of the n x n diagonal matrix A is of a matrix larger than 4 x 4 with a finite step algorithm. The only hope for larger matrices is an iterative procedure, and one of the best is based on the A'= ( x-l o) Q R decomposition. 0 0 ' Two matrices have the same eigenvalues if they are related through a simi larity transformation. The goal of this algorithm is to find an infinite sequence where A is the nonzero part of A. Therefore we take of similarity transformations that converts a matrix Mo into its diagonal matrix of eigenvalues. y = Q*x = A'Q*b The method is to use Householder transformations to decompose the matrix andy= Q*x is the smallest minimizer of (1.12). However, Mn into the product of an orthogonal matrix and an upper triangular matrix Mn = QnRn· To make this into a similarity transformation we reverse the order X= QA'Q*b of matrix multiplication and form Mn+l = RnQn so that Mn+l = Q;; 1 MnQn. the same norm as Q*x and so is the smallest least squares solution of (1.12). Now the amazing fact is that if M0 is a real symmetric matrix, then Mn con has verges to the diagonal matrix of eigenvalues. Of course, if Mo is not symmetric, It follows that the pseudo-inverse of A is A'= QA'Q*. 38 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.5. LEAST SQUARES SOLUTIONS-PSEUDO INVERSES 39 The geometrical interpretation of this pseudo-inverse is as follows. H A = Suppose A* A bas k positive eigenvalues A1, A2 1 ••• , Ak > 0, and Ak+l = QAQ*, then Q represents the basis relative to which A has a diagonal represen ).k+2 = · · · =An = 0. Let Yi = $t. i = 1, 2, ... , k, and notice that tation. H one of the diagonal elements of A is zero, then of course, the inverse of A does not exist. To find A'b, we form Q*b which expresses b relative to the (y;,yi} = .JX~\f>;;"(Ax;,Axi} = .JXIJ>:;(xi,A'"Axi} new coordinate system. The expression A'Q*b projects Q*b onto the range of A and determines the smallest element whose image under A is Q* b. Finally = .:§f(x;,xi} = 5ii· Q(A'Q*b) expresses this vector in terms of the original coordinate system (see Problem 1.5.6). Since y; E mm I this guarantees that k :::; m. We can use the Gram-Schmidt The change of coordinates Q identifies both the range and null space of A. procedure to extend the y;'s to form an orthonormal basis for !Rm, {y;}~ 1 . Let What we have done here is to project the vector b onto the range of A and U be the matrix with columns Yl, Y2, ... , Ym. The vectors y; are the orthonormal then find another vector in the range of A whose image is this projection. Since eigenvectors of AA •. We now examine the entries of U* AV. For i ::; k, the ijth A* =A, the matrix Q is appropriate for both of these transformations. entry of u• AV is This construction of A' can work only for symmetric matrices where R( A •) = (y;,AxJ} = -it-(Ax;,Axj} = +(x;,A*Axi} R(A). Unfortunately, this construction fails for nonsymmetric matrices for two 'S.~' v Ai important reasons (see Problem 1.5.9}. First, since A* :f. A, R(A*) :f. R(A) so = ~(Ax;,Axi) = .J>:iaiJ· projecting onto R(A) does not also put us on R(A*). Furthermore, if A* :f. A, Fori> k, note that AA*y; 0 so that A*y; is simultaneously in the null space the basis provided by the diagonalization of A is not an orthogonal basis and = of A and the range of A*. Therefore (the Fredholm alternative again}, A*y; 0 hence projections are not orthogonal. = and (y;,Axi} = (A*y;,xi} = 0 fori> k. It follows that U*AV = E where What is needed is an analogous construction for nonsymmetric matrices. !: (O';j), O'ij = .J>:i5;i, i 1,2, ... , m, j 1, 2, ... ,n, and this is equivalent This construction is provided by the singular value decomposition, which we = = = to (1.13). now state. It is noteworthy that the orthogonal vectors {x1 , x2 , ••• , xk} span the range Theorem 1.13 Suppose A is any m x n matrix. of A • and the vectors { x k+l, ... , Xn} span the null space of A. Similarly, the vectors {y1, Y2, ... Yk} span the range of A and {Yk+l• ... , Ym} span the null 1. A can be factored as space of A*. Thus the orthogonal matrices U and V provide an orthogonal A= UEV*, (1.13} decomposition of !Rm and !Rn into !Rm = R(A) EB N(A*) and !Rn = R(A"') EB where U (m x m) and V (n x n) are unitary matrices that diagonalize AA• N(A), respectively. Furthermore, the singular value decomposition A= UEV* and A • A respectively, and E is an m x n diagonal matrix {called the shows the geometry of the transformation of X by A. In words, v· X decomposes x into its orthogonal projections onto R(A•) and N(A), then EV*x rescales the matrix of singular values) with diagonal entries O'ii = ...;>:i1 where Ai 1 i = 1,2, ... ,min(m,n), are the eigenvalues of AA* {see Problem 1.B.3}. projection of x onto R(A*), discarding the projection of x onto N(A). Finally, multiplication by U places !:V*x back onto the range of A. B. The pseudo-inverse of A is It is now clear that to minimize A'= V!:'U., (1.14) 2 2 IlAx- bll = IIU!:V*x- bll , where !:' 1 the pseudo-inverse of !:1 is an n x m diagonal matrix with we take 1 diagonal entries -(1' =-it- provided O'ii :f. 0 and 0 otherwise. o11 = II V AI X= V!:'U*b so that the pseudo-inverse of A is as stated in (1.14). I The factorization A = U!:V* is called the singular value decomposition of The singular value decomposition gives a nice geometrical interpretation for A. the action of A. That is, first there is a rotation by V*, followed by rescaling Proof: To show that this decomposition always exists, notice that the matrix by E and finally a second rotation by U. Similarly, to find the least squares A• A is a symmetric n x n matrix with non-negative eigenvalues A1, A2 1 ••• , An· solution of Ax = b, we first use U to decompose b into its projection onto R(A) Let V be the n x n matrix that diagonalizes A • A. The columns of V are the and N(A*). The component on R(A) is rescaled by E' and then transformed orthogonal eigenvectors x1,x2, .•. ,xn of A* A, normalized to have length one. onto R(A*) by V. Then This again illustrates nicely the importance of transform methods. As we A* A= VAV*. know, the least squares solution of Ax = b can be found by direct techniques. 40 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.5. LEAST SQUARES SOLUTIONS-PSEUDO INVERSES 41 However, the change of coordinates provided by the orthogonal matrices U, V in the direction orthogonal to the nearly parallel straight lines of the original transforms the problem Ax= b into ~y = b where y = V'"x, b = U'"b, which is problem. In other words, the singular value decomposition identified the stable a diagonal (separated) problem. This least squares problem is easily solved and and unstable directions for this problem, and by purposely replacing € by zero, transforming back we find, of course, x = V~'U'"b. This transform process is we were able to keep the unstable direction from contaminating the solution. illustrated in the commuting diagram In a more general setting, one can use the singular value decomposition to filter noise (roundoff error) and stabilize inherently unstable calculations. This A' Ax=b --t x=A'b is done by examining the singular values of A and determining which of the y=V'"x x=Vy singular values are undesirable, and then setting these to zero. The resulting .t. b=U'"b t b=Ub pseudo-inverse does not carry with it the destabilizing effects of the noise and E' roundoff error. It is this feature of the singular value decomposition that makes ~y=b ~ y= ~'b it the method of choice whenever there is a chance that a system of equations where the top line represents the direct solution process, while the two verti has some potential instability. cal lines represent transformation into and out of the appropriate coordinate The singular value decomposition is always recommended for curve fitting. representation, and the bottom line represents solution of a diagonal system. The SVD compensates for noise and roundoff error prppagation as well as the The singular value decomposition also has the important feature that it fact that solutions may be either overdetermined or underdetermined. In a allows one to stabilize the inversion of unstable (or ill-conditioned) matrices. least squares fitting problem it is often the case that there is simply not enough To illustrate this fact, consider the matrix experimental evidence to distinguish between certain basis functions and the resulting least squares matrix is (nearly) underdetermined. The SVD signals A=(1+-Jfo this deficiency by having correspondingly small singular values. The user then 2-710 1-1fo),2+-fto has the option of setting to zero small singular values. The decision of how small is small is always up to the user. However, with a little experimentation where € is a small number. We can visualize what is happening by viewing the one can usually learn what works well. equation Ax = b as describing the intersection of two straight lines in JR 2 by noting that for € small the lines are nearly parallel. Certainly small changes in b give large changes to the location of the solution. Thus, it is apparent that 1.5.1 The Problem of Procrustes the inversion of A is sensitive to numerical error. The Householder transformation solves the problem of finding an orthogonal We calculate the singular value decomposition of A to be transformation that transforms one vector into another, provided the lengths of the two vectors are the same. A generalization of this problem, called the prob A--1 (1 2)(v'10 0)(1 1) lem of Procrustes3 is to seek an orthogonal transformation that transforms -1 0 € 1 -1 ' - v'10 2 one set of vectors into another, as best possible. and the inverse of A to be Suppose we have two sets of vectors in JRn, represented as the columns of two n x k matrices A and B, and we wish to find the best orthogonal transformation A-1 --L(1 1)(fto 0)(1 2) Q that aligns A with B. That is, we wish to minimize - v'IO 1 -1 0 € 2 -1 2 = ( 0.1 + eJro 0.2 - e{to ) . IIQA-BII • 0.1 - eJrn 0.2 + e:;/i6 Before we can solve this problem we must define an appropriate norm for 1 matrices. A useful matrix norm for this problem is the Frobenius norm It is clear that when the singular value € is small, the matrix A- is very large 1 and the solution x = A- b is unstable. IIAII~ = Tr(A* A) =Tr(AA*). If instead of using the full inverse, we replace € by zero and use the singular value decomposition to find the pseudo-inverse, (see Problem 1.5.14). In terms of this norm, the problem is to minimize IIQA- Bll~ = Tr(AA'") + Tr(BB'")- 2Tr(QAB*). A' __1 ( 1 1 ) ( 'fto 0 ) ( 1 2 ) _ 1:_ ( 1 2 ) - v'iO 1 -1 0 0 2 -1 - 10 1 2 ' 3 Procrustes was a legendary Greek outlaw, living in Athens, who would offer his hospitality to travelers, and then force them to lie on an iron bed, stretching them on a rack if they were the solution is now stable. That is, small changes in the vector b translate into too short or lopping off limbs if they were too long for the bed - an unusual method of only small changes in the solution vector x = A'b and, in fact, x varies only coordinate transformation,--to be sure. 42 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES ).6. APPLICATIONS OF EIGENVALUES AND EIGENFUNCTIONS 43 Thus, it is equivalent to maximizing Tr(QAB*). where eAt is the matrix whose only nonzero components are the diagonal ele Now suppose that the matrix AB* has the singular value decomposition ments e).it, j = 1, 2, ... , n. It follows that 1 AB* =U~V*. x(t) = T- eAtTx(O} Then, so that eAt ~ T-leAtT. Tr(QAB*) = Tr(QU~V*) = Tr(V*QU~) = 'Eziiqi $ L:ai, i i 1.6.2 The Power Method and Positive Matrices where Z = V*QU. This inequality is valid since the diagonal elements of a There are many situations where the only eigenvalue of interest is the one with real unitary matrix cannot be larger than one, and the singular values ai are largest magnitude. It is easy to calculate the largest eigenvalue and correspond non-negative. However, it is also clear that equality is obtained if and only if ing eigenvector using a method called the power method. The power method Z = I, in which case involves "powering" a matrix. We start with some initial vector x0 , and then Q=VU*. form the sequence of vectors XI; (1.16} X1;+1 = Allx~;ll" 1.6 Applications of Eigenvalues and Eigenfunc Theorem 1.14 If the largest (in magnitude) eigenualue At of A is unique with tions algebraic multiplicity one, and has a one-dimensional eigenspace spanned by ifJ,., and if (:eo, l/11) 0, lll/Jtll 1, then It is not possible to overstate the importance of eigenvalues and eigenfunctions in :f: = applications. In this section, we give a short introduction to a few of the ways lim XI; = AtlPl· in which an understanding of eigenvalues and eigenfunctions gives important k-+oo insight or capability in applied problems. 1.6.1 Exponentiation of Matrices Proof: We suppose that At is the largest eigenvalue of A and that t/11 is the corresponding eigenvector of A •, (l/Jt. tPt) = 1. We represent the initial vector The solution of the scalar ordinary differential equation ~ = au with u(O) = Uo as xo = a1l/Jt + r1 with a1 = (xo, t/11) and note that rt is orthogonal to tPI. is u(t) = uae'". It would be nice if we could make a similar statement for the (rt, tPt) = 0. The orthogonal complement of tPt is an invariant subspace of A vector differential equation since (Ar, t/Jt) = (r, A*t/Jt) = (r, At tPt) = 0 if (r, tPt) = 0. Furthermore, dx=Ax (1.15) dt IIArll $ Kllrll where A is ann x n matrix and x(t) is a vector in m.n. The problem, however, for some constant K < IAtl, whenever (r,t/Jt) = 0. is that we do not know how to interpret the matrix eAt. Now we calculate that The answer is readily found if the matrix A can be diagonalized. If there is a similarity transformation so that A= T-tAT, then (1.15) becomes Akxo = a1Atl/J1 + r~; dx where T- T AT-1Tx ATx dt = = llr~;ll = IIAAiroll $ Kkllroll, so that so that 1 dy lim , ~;A"xo-+ atlPl· -=Ay k-+oo "l dt Thus, if a1 :f: 0, our result is established. Furthermore, the rate of convergence where Tx = y. Since A is a diagonal matrix, the solution of the system !!If = Ay is approximately f , which is a measure of the separation between the largest is readily written as eigenvalue At and till other eigenvalues of A. 1 y(t) = eAty(O), 44 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.6. APPLICATIONS OF EIGENVALUES AND EIGENFUNCTIONS 45 Example: The Ranking of Teams are all strictly positive. FUrthermore, the largest eigenvalue of the matrix A+ p.I is ,\ Every Fall a controversy rages throughout the United States trying to decide + p., where tjJ is the positive eigenvector and ,\ the corresponding positive which is the best collegiate football team {division I-A). The dilemma is that eigenvalue of A. Hence, if we power the matrix A + JLl, convergence to tjJ is there are over a hundred teams vying for the title, but no team plays more assured (provided A is irreducible), and tjJ is the positive ranking vector we than 12 or 13 opponents, so there are often good teams that have not been seek. matched up head to head. It is certain that this same controversy occurs in many other sports and in many other countries. 1.6.3 Iteration Methods A possible resolution to this controversy is to devise a ranking scheme that assigns a rank to each team based on its performance. One idea (there are The power method is one of the many iteration methods that allow rapid extrac several) for how to calculate a team's rank is as follows: We assume that tion of important information about a matrix. In fact, matrix iteration is the team i has rank r;, a number between 0 and 1. The team earns a score based only practical method by which to solve many important problems associated on its games with other teams with partial differential equations. Direct methods, such as Gaussian eliminar tion, are certain to work if the problem is small enough, but direct methods 8; = _!_ L a;Jrh have serious difficulties when matrices are large and sparse (a sparse matrix is n; J'f.i one which has mostly zero entries), as they often are with the simulation of where a;J is a number that reflects the result of the head to head contest partial differential equations. between teams i and j. The score depends on the product of a;; and r; in The main idea of many iteration methods is to split the matrix A into two order to give some weight to the strength of opposition. It is reasonable to parts, A = A1 + A2, where A1 is invertible and easily solved, and then to write take a;i + ai; = 1 with a;J > aii if team i beat team j in their head to head the equation Ax = b as contest. The division by n;, the total number of games played by team i, is A1x b- A2x. necessary to normalize the score 8;. The entry a;; is zero if team i has not = played team j, and of course, a;t = 0. We then define an iterative procedure by It is now reasonable to suppose that the score and the rank of a team should be related by ..\r, = s;, so that the determination of rank reduces to an AlXHl = b- A2Xk· (1.17) eigenvalue problem Ar = ..\r, where the entries of A are the weighted elements a;; /n;. The question is to determine if this equation has a useful solution. The convergence of this iteration is easily seen to depend on the eigenvalues 1 of the matrix A1 A2 . That is, if xis a solution of the problem Ax= b, then This leads us to consider an important class of matrices called positive ma 1 trices. A positive matrix is a matrix A all of whose entries are positive. Sim Xk+l - x = A1 A2(xk - x). ilarly, a nonnegative matrix is one all of whose elements are nonnegative. The If A1 1 A2 has eigenvectors t/J; with corresponding eigenvalues J.l.j, and if Xo -x = matrix that we constructed to rank football teams is nonnegative. The classi cally important statement about these matrices is called the Perron Frobenius Ej=l a;t/J;, then Theorem. Xk- X=~ OjJ.l.Nj· j Theorem 1.15 (Perron Frobenius) A nonnegative matrix A has a nonneg 1 The iteration converges provided the largest eigenvalue of A1 A2 is smaller than ative eigenfunction with corresponding positive eigenvalue. Furthermore, if the 1 in absolute value. matrix is irreducible, the eigenfunction is unique and strictly positive, and the Now the trick is to split A in such a way as to make the largest eigenvalue corresponding eigenvalue is the largest of all eigenvalues of the matrix. 1 of A1 A2 as small as possible, thereby making convergence as fast as possible. The easiest split of A is perhaps We leave the proof of this theorem to the interested reader, which can be A=D+L+U , found in several sources, including [56]. There are several equivalent ways to de scribe irreducibility. Perhaps the most geometrical is that a matrix is irreducible where D is the diagonal part of A, and L and U are lower and upper triangular if for every pair of integers i, j, there is a sequence k1, k2, ••• , km for which the (off-diagonal) parts of A, respectively. Then, if we choose A1 = D, and A2 = product Oik1 OA: 1 A: 2 ••• akmi is nonzero. In practical terms for football teams, this L + U, iterates are called Jacobi iterates. These iterates are certainly easy to means that between any two teams there is a path of common competitors. calculate, since they only involve solving the system The implementation of the Perron Frobenius theorem uses the power method. There is a positive number, say p.,_ so that the eigenvalues of the matrix A+ p.I Dxn+l = b- (L + U)xn, 46 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES 1.6. APPLICATIONS OF EIGENVALUES AND EIGENFUNCTIONS 47 which, if the diagonal elements are all nonzero, is easy and fast. However, it is Now suppose that !Jmsn and /Jmaz are the smallest and largest eigenvalues of 1 not much harder to take At= D + U and A2 = L, in which case the iterates A1 A2 , and that -1 < l'min < l'maz < 1. Then the optimal relaxation pa rameter is found by balancing the numbers 1- w(JJj + 1) between -1 and 1, (D + L)xn+l = b- Uxn taking 2 Wopt = 1 require solution of a lower triangular system of equations at each step. This 2 + Jlomln + Jloma3l method goes by the name of Gauss-Seidel iterates. in which case the eigenvalue that determines the rate of convergence for the The key determinant to the success of these methods is the size of the eigen 1 SOR method is values of A} A2 • While these cannot be calculated easily for many matrices, it /Jmaz - Jlomln is illustrative to see what these are for some simple examples. ..\soR = 2 + l'maz + Jlomln Suppose D is a scalar multiple of the identity matrix, D = dl. H t/J is an This is an improvement unless JLmsn = -l'maz· eigenvector of A with eigenvalue ..\, At!J = ..\t/J, then t/J is also an eigenvector of D-1(L + U). That is, Multigrid Methods n-1(£ + U)t/J = (~ -1)€/J. The solution of large systems of equations which come from the discretization of partial differential equations has been revolutionized with the discovery of a It is an immediate consequence that the Jacobi method converges whenever all new approximate solution technique called the multigrid method. As we will eigenvalues ..\ of A satisfy 0 < ~ < 2. Thus, for example, if A is diagonally see, the success of this method is completely dependent upon its relationship to dominant (a matrix A= (atJ) for which IBid~ L#t latJD , Jacobi iterates are the eigenvalues and eigenfunctions of the underlying problem. convergent. However, at this point in our development of transform theory, it is not It is always possible to try to accelerate the convergence of an iterative yet appropriate to discuss this important solution technique, so we defer the method. Acceleration is the goal of discussion of multigrid methods to Chapter 8. Atz~+l = b- A2Zn 1 Zn+l = wx~+l + (1 - w)xn Further Reading for some parameter w. Notice that if w = 1, the iterations are unchanged. With w i- 1, this method is called successive over-relaxation or SOR, and There are numerous linear algebra books that discuss the introductory material the parameter w, usually taken to be bigger than 1, is called the relaxation of this chapter. Three that are recommended are: parameter. Notice that if the relaxation parameter is less than one then the new guess Zn+t is an interpolation between z~+l and Zn, whereas if the relaxation • P. R. Halmos, Finite Dimensional Vector Spaces, Van Nostrand Reinhold, parameter is greater than one, the new guess is an extrapolation. New York, 1958. It known that for many matrices the Jacobi method converges more slowly is • G. Strang, Linear Algebra and its Applications, 3rd ed., Academic Preas, than the Gauss-Seidel method, and with careful choice of the relaxation param New York, 1988. eter, the SOR modification of Gauss-Seidel iterates converges fastest of them all. It is not hard to see how to choose the relaxation parameter to get improved • R. A. Horn and C. R. Johnson, MatriX Analysis, Cambridge University convergence from the SOR method. Press, Cambridge, 1985. 1 Suppose the eigenvalues of A} A2 are known, with The geometrical and physical meaning of eigenvalues is discussed at length in Ai1 A2tPJ = l'itPi· the classic books The convergence of the SOR method is assured if the eigenvalues of • R. Courant and D. Hilbert, Methods of Mathematical Physics, Volume I, Wiley-Interscience, New York, 1953. 1 1 A} ( -wA2 + {1 ·~ w)A1) = -wA} A2 + (1- w)I • J. H. Wilkinson, The Algebraic Eigenvalue Problem, Oxford University are less than one in magnitude. However, if tPi is an eigenvector of Ai1 A2, then Preas, London, 1965. ( -wAi1 A2 + (1- w)I)t!JJ = (1- w(JJJ + 1))tPi· Numerical aspects of linear algebra are discussed, for example, in 48 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES PROBLEMS FOR CHAPTER 1 49 • G. W. Stewart, Introduction to Matrix Computations, Academic Press, Problems for Chapter 1 New York, 1980, Problem Section 1.1 • G. E. Forsythe, M.A. Malcolm and C. B. Moler, Computer Methods for Mathematical Computations, Prentice-Hall, 1977, 1. Prove that every basis in a finite dimensional space has the same number of elements. • G. H. Golub and C. F. Van Loan, Matrix Computations, 3rd ed., Johns 2. Show that in any inner product space Hopkins University Press, Baltimore, 1996, 2 2 llx+vll + llx -vll = 2llxW + 2llvW. • L. N. 'frefethen and D. Bau, Numerical Linear Algebra, SIAM, Philadel phia, 1997, Interpret this geometrically in JR?. and for least squares problems in 3. (a) Verify that in an inner product space, • C. L. Lawson and R. J. Hanson, Solving Least Squares Problems, SIAM, 1 2 Re (x, y} = 4(11x + -llx- vll ). Philadelphia, 1995, vW while many usable programs are described in (b) Show that in any real inner product space there is at most one inner product which generates the same induced norm. • W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling, Nu 11 (c) In IRn with n > 1, show that llxiiP = CE~= IxkiP) P can be induced merical .Fbrtran Art of Computing, Cambridge Uni 1 Recipes 1n 77: The · by an inner product if and only if p = 2. versity Press, Cambridge, 1992. 4. Suppose f(x) and g(x) are continuous real valued function defined for This book (in its many editions and versions) is the best selling mathematics x E [0, 1). Define vectors in IRn, F = (f(xt), j(x2), ... , f(xn)) and G = book of all time. (g(xt), g(x2), ... , g(xn)), where Xk = kjn. Why is The best and easiest way to do numerical computations for matrices is with Matlab. Learn how to use Matlabl For example, 1 n (F, G}n = nL f(xk)g(xk) • D. Hanselman and B. Littlefield, Mastering Matlab, Prentice-Hall, Upper k=l Saddle River, NJ, 1996. with Xk = ~'not an inner product for the space of continuous functions? While you are at it, you should also learn Maple or Mathematica. 5. Show that 1 • B. W. Char, K. 0. Geddes, G. H. Gonnet, B. L. Leong, M. B. Monagan, (!,g)= 1(!(x)g(x) + f'(x)g'(x)) dx. and S. M. Watt, Maple V Language Reference Manual, Springer-Verlag, New York, 1991, is an inner product for continuously differentiable functions on the interval [0, 1). • A. Heck, Introduction to Maple, 2nd ed., Springer-Verlag, New York, 1996. 6. Show that any set of mutually orthogonal vectors is linearly independent. • S. Wolfram, Matbematica, 3rd ed., Addison-Wesley, Reading, MA, 1996. 7. (a) Show that IRn with the supremum norm llxlloo = maxk{lxkl} is a Finally, the ranking of football teams using a variety of matrix algorithms normed linear vector space. is summarized in (b) Show that IRn with norm llxl It = :E~=l lxk I is a normed linear vector. 2 • J.P. Keener, The Perron Frobenius Theorem and the ranking of football 8. Verify that the choice-y =~minimizes llx--yyll • Show that l(x, y}l 2 = teams, SIAM Rev., 35, 80-93, 1993. llxll 2 ·llvll2 if and only if x andy are linearly dependent. 50 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES pROBLEMS FOR CHAPTER 1 51 9. A Starting with the set {1, x, x2 , ••• , xk, ... }, use the Gram-Schmidt pro (c) Is the converse true? cedure and the inner product 3. (a) Show that if A is ann x m matrix and B is an m x n matrix, then AB and BA have the same nonzero eigenvalues. (f,g} = f(x)g(x)w(x)dx, w(x) > 0 ib (b) Show that the eigenvalues of AA• are real and non-negative. to find the first five orthogonal polynomials when 4. Show that the eigenvalues of a real skew-symmetric (A = -AT) matrix are imaginary. (a) a= -1, b = 1, w(x) = 1 (Legendre polynomials) (b) a= -1, b = 1, w(x) = (1- x2)-112 (Chebyshev polynomials) 5. Find a basis for the range and null space of the following matrices: (c) a= 0, b = oo, w(x) = e-~ (Laguerre polynomials) (a) (d) a= -oo, b = oo, w(x) = e-:~~ 2 (Hermite polynomials) Remark: All of these polynomials are known by Maple. 2 A~o D· 10. ,!;!. Starting with the set {1, x, x , ••• , xn, ...} use the Gram-Schmidt pro cedure and the inner product (b) 1 (f,g} = /_ (f(x)g(x) + f'(x)g'(x))dx 1 2 3) 1 A= ( 3 1 2 . to find the first five orthogonal polynomials. 1 1 1 6. Find an invertible matrix T and a diagonal matrix A so that A = TAT-1 Problem Section 1.2 for each of the following matrices A: 1. (a) Represent the transformation whose matrix representation with re (a) spect to the natural basis is ( 1 0 0 ) A= 12 1 2)3 1/4 1/4 1/2 ( 1 0 1 0 0 1 relative to the basis {(1, 1, O)T, (0, 1, l)T, (1, 0, l)T}. (b) (b) The representation of a transformation with respect to the basis {(1,1,2}T,(1,2,3)T,(3,4,1)T} is ( -~ ~) 1 1 1 ) (c) A= 2 1 3 . ( 1 0 1 1 0 2 0)0 Find the representation of this transformation with respect to the c 2 1 3 basis {(1, 0, O)T, (0, 1, -l)T, (0, 1, 1)T}. 2. (a) Prove that two symmetric matrices are equivalent if and only if they (d) have the same eigenvalues {with the same multiplicities). (b) Show that if A and B are equivalent, then c0 1 0)0 detA = detB. 0 1 1 52 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES pROBLEMS FOR CHAPTER 1 53 (e) Problem Section 1.3 1. Use the minimax principle to show that the matrix 1/2 1/2 ~/6 ) 1/2 1/2 ~/6 . ( 2 4 5 1 ) ~/6 ~/6 5/6 4 2 1 3 7. Find the spectral representation of the matrix ( 5 1 60 12 1 3 12 48 A= ( _; ~). has an eigenvalue~ < -2.1 and an eigenvalue At > 67.4. illustrate how Ax = b can be solved geometrically using the appropriately 2. (a) Prove an inequality relating the eigenvalues of a symmetric matrix cliosen coordinate system on a piece of graph paper. before and after one of its diagonal elements is increased. 8. Suppose P is the matrix that projects (orthogonally) any vector onto a (b) Use this inequality and the minimax principle to show that the small manifold M. Find all eigenvalues and eigenvectors of P. est eigenvalue of 9. The sets of vectors {1/>i}f.:tJ {,P,}f::t are said to be biorthogonal if A= 84 4 8 -44) (1/>,,P;} = 6,;. Suppose {1/>slr:t and {,P,}f:t are biorthogonal. ( 4 -4 3 (a) Show that {1/>i}f::t and {1/li}f::t each form a linearly independent set. is smaller than -1/3. (b) Show that any vector in JRn can be written as a linear combination · of {t/>i} as n :Eait/>i x= 12 2 43) i=t ( 3 4 3 where a, = (x, 1/Js)· (c) Express (b) in matrix form; that is, show that is not positive. n 4. The moment of inertia of any solid object about an axis along the unit x= LP,x vector x is defined by i=t J(x) = L~(y)pdV, where P1 are projection matrices with the properties that P( = P, and P,P; = 0 fori¥:. j. Express the matrix Pi in terms of the vectors where d11 (y) is the perpendicular distance from the point y to the axis t/>1 and ,p,. along x, p is the density of the material, and R is the region occupied by 10. (a) Suppose the eigenvalues of A all have algebraic multiplicity one. the object. Show that J(x) is a quadratic function of x, J(x) = :z;T Ax Show that the eigenvectors of A and the eigenvectors of A • form where A is a symmetric 3 x 3 matrix. a biorthogonal set. 5. Suppose A is a symmetric matrix with eigenvalues At ~ ..\2 > As ~ •.•• (b) Suppose At/>i = Ast/>i and A•,p, = >..,,p,, i = 1, 2, ... , n and that Show that Ai ¥:. A; for i ¥:. j. Prove that A = E~1 A,P, where P, = tf>,,p; is a max(u,v)=O(Au, u} + (Av, v} = At + ..\2 projection matrix. Remark: This is an alternate way to express the spectral decomposition theorem for a matrix A. where !lull = llvll = 1. (c) Express the matrices C and c-t, where A= CAc-t, in terms of t/>1 and ¢ 1• Problem Section 1.4 (d) Suppose At/> = At/> and A,P = X,p and the geometric multiplicity of A 1. Under what conditions do the matrices of Problem 1.2.5 have solutions is one. Show that it is. not necessary that (I/>, ,P} ¥:. 0. Ax = b? Are they unique? 54 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES pROBLEMS FOR CHAPTER 1 55 2. Suppose P projects vectors in JRn (orthogonally) onto a linear manifold 3. (a) For any two vectors x,y E mn with llxll = IIYII find the Householder M. What is the solvability condition for the equation Px = b? (orthogonal) transformation U that satisfies Ux = y. 3. Show that the matrix A= (lliJ) where a,i = ( 0 for tions are generally preferable) to find the QR representation of the matrix all x :/: 0. Use the Fredholm alternative to prove that a positive definite matrix is invertible. 2 l 3) A= ( 4 2 1 . Problem Section 1.5 9 1 2 1. Use any of the algorithms in the text to find the least squares pseudo 5. For the matrix inverse for the following matrices: 5 -3) (a) A= ( 0 4 I illustrate on a piece of graph paper how the singular value decomposition 0 1 0 A = UEV* transforms a vector x onto Ax. Compare this with how c0 0) A= TAT- 1 transforms a vector x onto Ax. 1 1 0 6. For the matrix (b) A=(~!), ( ~ -~) illustrate on a piece of graph paper how the least squares pseudo-inverse -1 3 A' = Q-1 A'Q transforms a vector b into the least squares solution of Ax=b. (c) 7. For each of the matrices in Problem 1.2.6, \lSe the QR algorithm to ( 3 1 2) form the iterates An+l = Q;;1 AnQn, where An = QnRn· Examine a few -1 1 -2 of the iterates to determine why the iteration works and what it converges to. (d) 8. For the matrices ( -1 2/30 2/31) . A=(~~) -1 -2/3 7/3 and A ( 1.002 0.998 ) (e) Jl. The linear algebra package of Maple has procedures to calculate = 1.999 2.001 I the range, null space, etc. of a matrix and to augment a matrix with another. Use these features of Maple to develop a program that uses illustrate on a piece of graph paper how the least squares pseudo-inverse exact computations to find A' using the Gaussian elimination method A' = VE'U* transforms a vector b onto the least squares solution of (Method 1), and use this program to find A' for each of the above Ax = b. For the second of these matrices, show how setting the smallest matrices. singular value to zero stabilizes the inversion process. 2. Verify that the least squares pseudo-inverse of an m x n diagonal matrix 9. For a nonsymmetric matrix A= T-1 AT, with A a diagonal matrix, it is T-1 D with ~i = t1i6i; is the n x m diagonal matrix D' with ~i = ; 1 6,; not true in general that A' = A'T is the pseudo-inverse. Find a 2 x 2 whenever q' :/: 0 and d~i = 0 otherwise. example which illustrates geometrically what goes wrong. 56 CHAPTER 1. FINITE DIMENSIONAL VECTOR SPACES PROBLEMS FOR CHAPTER 1 57 10. Find the singular value decomposition and pseudo-inverse of Problem Section 1.6 1. A Write a computer program (in MATLAB) to find the largest eigen 2v'S -2v'5) value of a matrix using the power method. Use this program to solve the A= 3 3 . following problems: ( 6 6 (a) Find the largest eigenvalue of the matrix in Problem 1.3.1. 11. Find the least squares solution of then linear equations (b) Find the smallest eigenvalue of the matrix in Problem 1.3.1. (Shift the eigenvalues by subtracting an appropriate constant from the di a;x + b;y = Ci i = 1,2, ... ,n, agonal.) where a;b1 - a1b, :f: 0 for i :f: j. H r1, j = 1, 2, ... , k = n(n- 1)/2 are 2. ~ Develop a ranking scheme for your favorite sport. First collect the the solutions of all possible pairs of such equations, show that the least data for all the games played between teams (or individuals). squares solution (a) Take a;i = 1 if team i beat team j and zero otherwise. Does this r = (:) give a reasonable ranking? is a convex linear combination of r i, specifically (b) Take aii to be the percentage of points earned by team i in the contest between team i and j. Does this give a reasonable ranking? k (c) Propose your own method to assign values to the matrix A. r = LPJTj, J=l 3. Prove that a diagonally dominant matrix has a nonnegative eigenvector. where D2 P - i i- """L..ti=l D2'i and Di is the determinant of the jth 2-equation subsystem. Interpret this result geometrically. 12. The matrices M~c, k = 1, 2, ... , n, represent elementary row operations if they have the form fflij = 1 if i = j m,k = -aile if i > k m;1 = 0 otherwise. Suppose L is a triangular matrix with entries l;j = 1 if i = j, l,i = a,i if i > j, l;j = 0 otherwise. Show that (MnMn-1 · · · Mt)-1 = L. 13 . Find the SVD of then x n Hilbert segment A= (at ),au = .1!. 1 ,~.,for3 several values of n. Why is this matrix ill-conditioned? 14. (a) Show that IIAIIF = Tr(A* A) is a norm on the space of n x k matrices. (b) Show that if A and B are related through an orthogonal similarity transformation, then IIAIIF = IIBIIF· (c) Show that IIAIIF = IIA*IIF, even though A and A* may have different dimensions. I·~ i:iJ ~ •'ffi t -~• I:}§ -~ J Bibliography j ·~ 1t ~ [1] M. J. Ablowitz, P. A. Clarkson, and M. A. Ablowitz. Solitons, Nonlin ear Evolution Equations and Inverse Scattering, volume 149 of London Mathematical Society Lecture Notes. Cambridge University Press, 1991. I {432} ~ [2] M. J. Ablowitz and A. S. Fokas. Complex Variables: Introduction and ·~::1 t Applications. Number 16 in Cambridge Texts in Applied Mathematics. .~ };~ Cambridge University Press, Cambridge, 1997. {273} .~ ~ [3] M. J. Ablowitz and H. Segur. Solitons and the Inverse Scattering Thins ~ •';! form. SIAM, Philadelphia, 1981. { 432} 'i, -_'4 .~ [4] M. Abramowitz and I. A. Stegun. Handbook of Mathematical Functions. I Dover, New York, 1974. {402} I [5] K. Aki and P. G. Richards. Quantitative Seismology, Theory and Methods: I Vol. JJ. Freeman, 1980. {132} 1" '$ [6) R. Askey. Orthogonal Polynomials and Special Functions. SIAM, Philadel phia, 1975. {93} !'$1 [7] K. E. Atkinson. A Survey of Numerical Methods for the Solution of Fred holm Integral Equations of the Second Kind. SIAM, Philadelphia, 1976. {127} [8] G. K. Batchelor. An Introduction to Fluid Dynamics. Cambridge Univer sity Press, Cambridge, 1967. {488, 497} [9] M.S. Bauer and D. S. Chapman. Thermal regime at the Upper Stillwater dam site, Uinta Mountains Utah. Tectonophysics, 128:1-20, 1986. {408} [10] C. M. Bender and S. A. Orszag. Advanced Mathematical Methods for Scientists and Engineers. International Series in Pure and Applied Math ematics. McGraw-Hill, 1978. { 463} [11] P. G. Bergmann. Introduction to the Theory of Relativity. Dover, New York, 1976. {203} 559 560 BIBLIOGRAPHY ~ BIBLIOGRAPHY 561 ·~" [12] N. Bleistein and R. A. Handelsman. Asymptotic Expansions of Integrah. ' [29] I. Daubechies. Ten Lectures on Wavelets. SIAM, Philadelphia, 1992. {93, Dover, New York, 1986. {463} 1 328} [13] A. Brandt. Multi-level adaptive solutions to boundary-value problems. [30] C. deBoor. A Pmctical Guide to Splines, volume 27 of Applied Mathemat Math. Comp., 31:333-390, 1977. {402} I ical Sciences. Springer-Verlag, New York, 1978. {93} [14] W. L. Briggs. Multigrid Thtorial. SIAM, Philadelphia, 1987. {402} [31] W. Eckhaus and A. Van Harten. The Inverse Scattering Transformation and the Theory of Solitons. North Holland, Amsterdam, 1981. {432} [15] J. Buckmaster and G. S. S. Ludford. Theory of Laminar Flames. Cam 1il ?l bridge University Press, New York, 1982. {551} i ·' [32] A. Erdelyi. Asymptotic Expansions. Dover, New York, 1987. {463} ·~ [16] G. F. Carrier, M. Krook, and C. E. Pearson. Functions of a Complex ' [33) P. C. Fife. Models for phase separation and their mathematics. In Variable. McGraw-Hill, New York, 1966. {273} ~~ M. Mimura and T. Nishiuda, editors, Nonlinear Partial Differential Equa j tions and Applications. Kinokuniya Publishers, 1991. {402} ll [17] K. M. Case. On discrete inverse scattering problems. J. Math. Phys., ·J 14:916-920, 1973. { 431, 432} 1 [34] H. Flaschka. On the Toda lattice, II-inverse scattering solution. Prog. ~ . Theor. Phys., 51:703-716, 1974. {431, 432} [18] B. W. Char, K. 0. Geddes, G. H. Gonnet, B. L. Leong, M. B. Monagan, 3 ~1 and S.M. Watt. Maple V Language Reference Manual. Springer-Verlag, << [35] G. E. Forsythe, M.A. Malcolm, and C. B. Moler. Computer Methods for 1991. {48} ~j Mathematical Computations. Prentice-Hall, 1977. ~~ {48} ;j [19] A. Cherkaev. Variational Methods for Structuml Optimization. Springer, New York, 1999. {203} ~ [36] A. C. Fowler: Mathematical Models in the Applied Sciences. Cambridge :l Texts in Applied Mathematics. Cambridge University Press, Cambridge, [20) Y. Choquet-Bruhat and C. DeWitt-Moretti. Analysis, Manifolds and 1997. {551} Phusics. North-Holland, Amsterdam, 1982. { 497} j [37) B. Friedman. Principles and Techniques of Applied Mathematics. Dover, [21] S. N. Chow and J. K. Hale. Methods of Bifurcation Theory. Springer New York, 1990. {171, 328, 401} Verlag, New York, 1982. {496} .; [38] P. R. Garabedian. Partial Differential-Equations. Chelsea Publishing Co., ~ [22] S. N. Chow, J. K. Hale, and J. Mallet-Paret. An example of bifurcation ,') New York, 1998. {401} to homoclinic orbits. J. Diff. Eqns., 37:351-373, 1980. I {497} ~ ;j (39) I. M. Gel'fand and S. V. Fomin. Calculus of Variations. Prentice-Hall, [23] C. K. Chui. An Introduction to Wavelets. Academic Press, 1992. {93, ~ Englewood Cliffs, 1963. {202} 307, 328} ~ [40) I. M. Gel'fand and B. M. Levitan. On the determination of a differential [24) E. T. Copson. Theory of Functions of a Complex Variable. Oxford at the l'P equation from its spectral function. Am. Math. Soc. Transl., pages 259- Clarendon Press, 1935. {273} 1 309, 1955. Series 2. {432} 'j [25) E. T. Copson. Asymptotic Expansions. Cambridge University Press, Cam J [41] I. M. Gel'fand and G. E. Shilov. Genemlized Functions, volume 1. Aca bridge, 1967. {463} demic Press, New York, 1972. {171} -~·• [26) A. M. Cormack. Representation of a function by its line integrals, with [42) H. Goldstein. Cl£Usical Mechanics. Addison-Wesley, Reading, MA, 2nd some radiological applications. J. Appl. Phys., 34:2722-2727, 1963. {128} 1~ edition, 1980. {203} ! [27] R. Courant and D. Hilbert. Methods of Mathematical Physics, volume 1. oW [43] G. H. Golub and C. F. Van Loan. Matrix Computations. Johns Hopkins Wiley-Interscience, New York, 1953. {47, 171} ~'1 University Press, 3rd edition, 1996. {48} [28] S. J. Cox and M. L. Overton. On the optimal design of columns against [44) D. Gottlieb and S. A. Orszag. Numerical Analysis of Spectral Methods; buckling. SIAM J. Appl. Math., .23:287-325, 1992. {203} Theory and Applications. SIAM, Philadelphia, 1977. {401} 562 BIBLIOGRAPHY BIBLIOGRAPHY 563 (45] J. Guckenheimer and P. Holmes. Nonlinear Oscillations, Dynamical Sys [62] 0. Kappelmeyer and R. Haenel. Geothermics with Special Reference to Ap tems, and Bifurcations of Vector Fields, volume 42 of Applied Mathemat plication. Number No 4 in Geoexploration Monographer, Ser. 1. Gebriider ical Sciences. Springer-Verlag, New York, revised edition, 1990. {496} Borntraeger, Berlin, 1974. {401} (46] R. Haberman. Elementary Partial Differential Equations. Prentice-Hall, [63] J. Keener and J. Sneyd. Mathematical Physiology. Springer-Verlag, New 3rd edition, 1997. {401} York, 1998. {551} (47] J. K. Hale. Ordinary Differential Equations. Wiley-Interscience, New (64] J.P. Keener. Analog circuitry for the van der Pol and Fitzhugh-Nagumo York, 1969. {513, 551} equations. IEEE 7rans on Sys., Man, Cyber., pages 101Q-1014, 1983. (48] P.R. Halmos. Finite Dimensional Vector Spaces. Van Nostrand Reinhold, SMG-13. {497} New York, 1958. {47} (65] J. P. Keener. The Perron Frobenius theorem and the ranking of football (49] D. Hanselman and B. Littlefield. Mastering Matlab. Prentice-Hall, Upper teams. SIAM Rev., 35:8Q-93, 1993. {48} Saddle River, NJ, 1996. {48} [66] J. B. Keller. Inverse problems. Am. Math. Monthly, 83:107-118, 1976. [50] A. Heck. Introduction to Maple. Springer-Verlag, New York, 2nd edition, {432} 1996. {48} [67] J. B. Keller and S. Antman. Bifurcation Theory and Nonlinear Eigenvalue [51] P. Henrici. Applied and Computational Complex Analysis. Wiley and Problems. Benjamin, New York, 1969. {496} Sons, 1986. {273} [68] J. Kevorkian and J. D. Cole. Multiple Scale and Singular Perturbation [52] H. Hochstadt. Integral Equations. Wiley-Interscience, New York, 1973. Methods, volume 114 of Applied Mathematical Sciences. Springer-Verlag, {127} Berlin, 1996. {546, 551} [53] H. Hochstadt. The FUnctions of Mathematical Physics. Dover, New York, (69] S. Kida. A vortex filament moving without change of form. J. Fluid Mech., 1986. {93} 112:397-409, 1981. { 497} . [54] M. H. Holmes. Introduction to Perturbation Methods. Springer-Verlag, (70] D. J. Korteweg and G. deVries. On the change of form of long waves New York, 1995. {551} advancing in a rectangular canal, and on a new type of long stationary waves. Phil. Mag., 39:422-443, 1895. {432} (55] F. Hoppensteadt. Mathematical Theories of Populations: Demographics, Genetics and Epidemics. SIAM, Philadelphia, 1975. {127} [71] Y. Kuznetsov. Elements of Applied Bifurcation Theory. Springer-Verlag, New York, 2nd edition, 1998. {496} [56] R. A. Hom and C. R. Johnson. Matrix Analysis. Cambridge University Press, Cambridge, 1985. {44, 47} (72] G. L. Lamb, Jr. Elements of Soliton Theory. John Wiley and Sons, New York, 1980. {328, 432, 497} [57] P. Horowitz and W. Hill. The Art of Electronics. Cambridge Press, Cam bridge, MA, 2nd edition, 1989. {497} [73] L. D. Landau and E. M. Lifshitz. Mechanics. Butterworth-Heineman, 3rd edition, 1976. {203} [58] C. M. Hutchins. The acoustics of violin plates. Scientific American, 245:17Q-187, 1981. {401} [74] C. L. Lawson and R. J. Hanson. Solving Least Squares Problems. SIAM, Philadelphia, 1995. {48} [59] F. M. Phelps Ill, F. M. Phelps IV, B. Zorn, and J. Gormley. An experi mental study of the brachistochrome. Eur. J. Phys., 3:1-4, 1984. {204} [75] P. Lax. Integrals of nonlinear equations of evolution and solitary waves. Comm. Pure Appl. Math., 21:407-490, 1968. {432} (60] G. Iooss and D. Joseph. Elementary Stability and Bifurcation Theory. Springer-Verlag, New York, 1989. {496} (76] G. A. Lion. A simple proof of the Dirichlet-Jordan convergence theorem. Am. Math. Monthly, 93:281-282, 1986. {165} (61] E. Jahnke and F. Emde. Tables of FUnctions. Dover, New York, 1945. {402} (77] W. V. Lovitt. Linear Integral Equations. Dover, New York, 1950. {127} 564 BIBLIOGRAPHY BIBLIOGRAPHY 565 [78] J. Lund and K. L. Bowers. Sine Methods for Quadrature and Differential [94] T. D. Rossing. The physics of kettledrums. Scientific American, 247:172- Equatiom. SIAM, Philadelphia, PA, 1992. {93, 274} 179, Nov. 1982. {401} [79] W. S. Massey. Algebraic Topology: An Introduction. Springer-Verlag, New [95] W. Rudin. Principles of Mathematical Analysis. McGraw-Hill, New York, York, 1990. { 497} 3rd edition, 1976. {92, 165} [80] S. F. McCormick, editor. Multigrid Methods. Frontiers in Applied Math [96] W. Rudin. Real and Complex Analysis. McGraw-Hill, New York, 3rd ematics. SIAM, Philadelphia, 1987. {402} edition, 1987. {164} [81] L. M. Milne-Thomson. Theoretical AerodJmamics. Dover, New York, 1973. [97] D. R. Smith. Singular Perturbation Theory. Cambridge University Press, {274} Cambridge, 1985. {551} [82) J. D. Murray. Asymptotic Analysis. Springer-Verlag, New York, 1984. [98] I. N. Sneddon. Fourier Transforms. Dover, New York, 1995. {328} {463} [99] I. Stakgold. Branching of solutions of nonlinear equations. SIAM Rev., [83) J. D. Murray. Mathematical Biology. Springer-Verlag, New York, 1989. 13:289-289, 1971. { 496} {402} [100] I. Stakgold. Boundary Value Problems of Mathematical Physics. Macmil [84) F. Natterer. The Mathematics of Computerized Tomography. Wiley, New lan, New York, 2nd edition, 1998. {171, 328} York, 1986. {128} [101] F. Stenger. Numerical methods based on Whittaker cardinal, or sine [85] A. W. Naylor and G. R. Sell. Linear Operator Theory in Engineering and functions. SIAM Rev., 23:165-224, 1981. {274} Science. Holt, Rinehart and Winston, New York, 1971. {127} · [102] F. Stenger. Numerical Methods Based on Sine and Analytic F'u.nctions, [86) F. W. J. Olver. Asymptotics and Special F'u.nctions. A. K. Peters, Ltd., volume 20 of Springer Series in Computational Mathematics. Springer London, 1997. {463} Verlag, New York, 1993. {93, 274} [87] R. E. O'Malley, Jr. Introduction to Singular Perturbations. Academic [103) G. W. Stewart. Introduction to Matrix Computations. Academic Press, Press, New York, 1974. {551} New York, 1980. {48} [88) M. Pickering. An Introduction to Fast Fourier Transform Methods for [104] J. J. Stoker. Nonlinear Elasticity. Gordon and Breach, New York, 1968. Partial Differential Equations, with Applicatiom. Wiley and Sons, New {203} York, 1986. {401} [105) J. J. Stoker. Differential Geometry. Wiley-Interscience, New York, 1969. [89] D; L. Pqwers. Boundary Value Probletm. Academic Press, 4th edition, l {202, 497} 1999. {401} [106] G. Strang. Linear Algebra and its Applications. Academic Press, New ., (90] W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling. l 'i York, 3rd edition, 1988. {47} Numerical Recipes in Fortran 77: The Art of Computing. Cambridge .:t University Press, Cambridge, 1992. {48, 77, 402} [107) G. Strang and G. Fix. An Analysis of the Finite Element Method. Prentice l Hall, 1973. {93} ~ [91] R. H. Rand. Topics in Nonlinear Dynamics with Computer Algebra. Gor ~ [108] G. Strang and T. Nguyen. Wavelets and Filter Banks. Wellesley don and Breach, Langhorne, PA, 1994. {497} ~~ 1 Cambridge Press, Wellesley, MA, 1997. {93, 328} [92] R. H. Rand and D. Armbruster. Perturbation Methods, Bifurcation The ~ (109] A. F. Timin. Theory of Approximation of Functions of a Real Variable. ory and Computer Algebra, Applied Mathematical Sciencu, volume 65. ~~ Springer-Verlag, New York, 1987. {497} :1 Macmillan, New York, 1963. {328} :1J'l (93] G. F. Roach. Green's FUnctions: Introductory Theory with Applications. J [110) E. C. Titchmarsh. Theory of F'u.nctions. Oxford University Press, 1939. Van Nostrand Reinhold, London, 1970. {171} ~ {273} 566 BIBLIOGRAPHY [111] E. C. Titchmarsh. Eigenfunction Expansions Associated with Second Order Differential Equations. Oxford University Press, Oxford, 1946. {328} [112] L. N. Trefethen and D. Bau. Numerical Linear Algebra. SIAM, Philadel phia, 1997. {48} [113] F. G. 'Iiicomi. Integral Equations. John Wiley and Sons, New York, 1957. Appendix A {127} (114) A. M. Turing. The chemical basis of morphogenesis. Phil. 7rans. Roy. Soc. Lond., B237:37-72, 1952. {390} Selected Hints and [115] M. Van Dyke. Perturbation Methods in Fluid DynamiC8. Academic Press, New York, 1964. {551} Solutions [116) M. Van Dyke. An Album of Fluid Motion. The Parabolic Press, Stanford, 1982. {274} (117) R. Wait and A. R. Mitchell. Finite Element Analysis and Applications. j Wiley and Sons, New York, 1968. {93} [jt.l.l; Hint: What if one basis was smaller than another? [118] J. W. Ward and R. V. Churchill. Fourier Series and Boundary Value i 1.1.3; (a) Follows from direct verification. Problems. McGraw Hill, New York, 5th edition, 1993. {92} , (b) Follows from (a). H the norm is known to be induced by an inner '! product, then (a) shows how to uniquely calculate the inner product. (119] H. F. Weinberger. Variational methods for eigenvalue approximation. SIAM, 1974. {203} (c) Suppose llxll = c~=~llxkiP)l/p. (-¢:::) If p = 2, then (x, y) = l':~=l XkYk induces the norm. [120] H. F. Weinberger. A First Course in Partial Differential Equations. Dover, (:::}) If the norm is induced by an inner product, then from (a) New York, 1995. {401} (121} G. B. Whitham. Linear and Nonlinear Waves. Wiley and Sons, New 1 ( n . n 2 ) (x, y) = 4 ixk + YkiP) 21P- ixk- YkiP) 1P . York, 1974. {401} (L (L k-1 k=l [122) J. H. Wilkinson. The Algebraic Eigenvalue Problem. Oxford University Take x = (1, 0, 0, ... , 0), and y = 0, 1, 0, ... , 0). Then (x, x) = Press, London, 1965. {47} 1, (x, y) = 0, and (x, x + y) = ! ((2P + 1?1P - 1). Since for an inner 2 [123] F. A. Williams. Combustion Theory. Addison-Wesley, Reading, Mass., product, (x, x + y) = (x, x) + (x, y), it must be that (2P + 1) /P = 5. 1965. {551} Since (2P + 1) 2/P is a monotone decreasing function of p which ap proaches 1 for large p and is unbounded at the origin, the solution of [124) T. J. Willmore. An Introduction to Differential Geometry. Oxford Uni (2P + 1)2/P = 5 at p = 2 is unique. We conclude that p = 2. versity Press, London, 1959. {203} 1.1.4; (F,F) = 0 does not imply that f(x) = 0. [125) S. Wolfram. Mathematica. Addison-Wesley, Reading, MA, 3rd edition, 2 1996. {48} 1.1.8; Observe that with {3 = (x, y) /llvll , x- {3y is orthogonal toy, so that (126] E. Zauderer. Partial Differential Equations of Applied Mathematics. Wiley ilx- avW = llx- f3vW + li(a- f3)vW, and Sons, New York, 2nd edition, 1998. { 401} which is minimized when a= {3. Clearly, if x = ay, then [127) A. H. Zemanian. Distribution Theory and 7ransform Analysis: An Intro 2 duction to Generalized Functions, with Applications. Dover, New York, i(x,v)l = llxWIIvW· 1987. {171} If so, then we calculate directly that llx- {3yjj 2 = 0, so that x = {3y. 567 566 BIDLIOGRAPHY [111] E. C. Titchmarsh. Eigenfunction Expansions Associated with Second Order Differential Equations. Oxford University Press, Oxford, 1946. {328} [112] L. N. Trefethen and D. Bau. Numerical Linear Algebra. SIAM, Philadel phia, 1997. {48} [113) F. G. 'llicomi. Integral Equations. John Wiley and Sons, New York, 1957. Appendix A {127} [114] A. M. Turing. The chemical basis of morphogenesis. Phil. Thlns. Ro1J. Soc. Lond., B237:37-72, 1952. {390} Selected Hints and [115] M. Van Dyke. Perturbation Methods in Fluid Dynamics. Academic Press, New York, 1964. {551} Solutions [116] M. Van Dyke. An Album of Fluid Motion. The Parabolic Press, Stanford, 1982. {274} [117) R. Wait and A. R. Mitchell. Finite Element Analysis and Applications. 1.1.1; Hint: What if one basis was smaller than another? Wiley and Sons, New York, 1968. {93} [118] J. W. Ward and R. V. Churchill. Fourier Series and Boundary Value 1.1.3; (a) Follows from direct verification. Problema. McGraw Hill, New York, 5th edition, 1993. {92} (b) Follows from (a). If the norm is known to be induced by an inner product, then (a) shows how to uniquely calculate the inner product. [119] H. F. Weinberger. Variational methods for eigenvalue approximation. 1 SIAM, 1974. {203} (c) Suppose llxll = o=~=llxkiP) 1P. (-<=) If p = 2, then (x, y} = L:~=l XkYk induces the norm. (120] H. F. Weinberger. A Firat Course in Partial Differential Equations. Dover, (:::::?)If the norm is induced by an inner product, then from (a) New York, 1995. {401} 2 (121] G. B. Whitham. Linear and Nonlinear Waves. Wiley and Sons, New (x, Y} = 41 ( (Ln lxk + YkiP) 21P- (Ln lxk- YkiP) 1P ) . York, 1974. {401} k-1 k=l [122] J. H. Wilkinson. The Algebraic Eigenvalue Problem. Oxford University Take x = (1, 0, 0, ... , 0), and y = 0, 1, 0, ... , 0). Then (x, x} = Press, London, 1965. {47} 1, (x, y} = 0, and (x, x + y} = ~ ( (2P + 1) 2/P - 1). Since for an inner 2 [123] F. A. Williams. Combustion Theory. Addison-Wesley, Reading, Mass., product, (x, x + y} = (x, x} + (x, y), it must be that (2P + 1) /P = 5. 1965. {551} Since (2P + 1)2/P is a monotone decreasing function of p which ap proaches 1 for large p and is unbounded at the origin, the solution of [124) T. J. Willmore. An Introduction to Differential Geometry. Oxford Uni (2P + 1) 2/P = 5 at p = 2 is unique. We conclude that p = 2. versity Press, London, 1959. {203} 1.1.4; (F, F} = 0 does not imply that f(x) = 0. [125) S. Wolfram. Mathematica. Addison-Wesley, Reading, MA, 3rd edition, 2 1996. {48} 1.1.8; Observe that with {3 = (x, y) /llvll , x- {3y is orthogonal toy, so that 2 2 2 [126) E. Zauderer. Partial Differential Equations of Applied Mathematics. Wiley llx- ayll = llx- f3Yll + ll(o:- f3)yjj , and Sons, New York, 2nd edition, 1998. {401} which is minimized when a= {3. Clearly, if x = ay, then [127] A. H. Zemanian. Distribution Theory and Thlnsform Anal1Jais: An Intro 2 2 duction to Generalized FUnctions, with Applications. Dover, New York, l(x,y)j = llxii IIYW· 1987. {171} If so, then we calculate directly that llx- {3yjj 2 = 0, so that x = {3y. 567 568 APPENDIX A. SELECTED HINTS AND SOLUTIONS 569 2 3 1.1.9; (a) ¢o(x) = 1, c/J1(x) = x, c/J2(x) = x - ~, ¢a(x) = x - ~x, ¢4(x) = (d) T does not exist. /x4- li.x2 .1... 7 + 35. 1 1 2 1 (b) ¢o(x) = 1,¢1(x) = x,¢2(x) = x - ~ = ~cos(2cos- x),¢3 (x) = 1 3 1 4 2 (e) T = -1 1 1)} , T- AT = (000)0 ~ ~ . x - ~x =-! cos(3cos- x), ¢4(x) = x - x + k = kcos(4cos- 1 x). ( 0 -v'3 ~ 0 0 3 2 3 (c) ¢o(x) = 1,¢1(x) = x -1,¢2(x) = x - 4x + 2,¢3 (x) = x - 9x2 + 18x- 6, ¢4(x) = x4 - 48x3 + 72x2 - 96x + 24. 1.2.7; T= _1 _ 2 ) ,T-1 AT= ( O3 0) . 2 3 ( 2 1 (d) ¢o(x) = 1,¢1(x) = x,¢2(x) = x - !,¢a(x) = x - ~x,¢4(x) = 6 x4- 3x2 + i· 1.2.8; If x is in M, then Px = x, and if x is in the orthogonal complement of 2 1.1.10; ¢o(x) = 1,¢1(x) = x,¢2(x) = x - ~,¢a(x) = x3 - t x,¢4(x) = x4 - M, the Px = 0. Therefore, P has two eigenvalues, A= 0, 1. 33 2 27 ... ( ) 5 1930 3 4411 0 28 X + 140' '1'5 X = X - 'i359X + 1057X. 1.2.10; (d) Examine the eigenvectors of A and A* where A= ( ~ ~ ). 2 3 1.2.1; (a) Relative to the new basis, A= 1 1 ( 0 0 n 1.3.1; Hint: Minimize (Ax,x) with a vector of the form xT = (1, -1,z,O). 1 1 3 ) ( 1 1.3.2; (a) Prove that if the diagonal elements of a symmetric matrix are in (b) Set C = 1 2 4 , and D = 0 01 0) 1 . Then the repre- creased, then the eigenvalues are increased (or, not decreased) as ( 2 3 1 0 -1 1 well. sentation of A in the new basis is 4 4 ) (b) Find the eigenvalues and eigenvecto"' of B = ( : 8 -4 , and 53 19 4 ) 1 "¥, -4 8 A'= v-'cAc-'v = ( ~ ~'~ =! . use them to estimate the eigenvalues of A. i 1 2 3) 1.2.2; (c) A = 1o) , and B = (lo) have the same determinant but 1.3.3; The matrix 2 2 4 has a positive, zero, and negative eigenvalue. ( 0 1 0 2 ( 3 4 7 are not equivalent. l Apply 1.3.2a. 1.2.3; (a) Notice that if ABx =Ax, then BA(Bx) = A(Bx). ·~ 1.4.1; (a) b must be orthogonal to (1, 1, -1)T, and the solution, if it exists, is (b) If AA•x =AX, then A(x, x) = (AA*x, x) = (A*x, A*x) :2: 0. unique. 1.2.4; If Ax= AX then A(x, x) = (Ax, x) = (x, ATx) = -(x, Ax) = -X(x, x}. (b) The matrix A is invertible, so the solution exists and is unique. 1.2.5; (a) R(A) = {(1,1,2)T,(2,3,5)T},N(A) = O,R(A*) = m?,N(A*) = 1.4.2; b must be in the range of P, namely M. {(1, 1, -1)T}. 3 1.4.3; ( =>) Suppose A is invertible, and try to solve the equation ~i ai 0 for all x =J:. 0, the null spaces of A and A • must be empty. Hence, (b, x) = 0 for all x in N(A*) so that Ax = b has (c) T= ( ~1 01 00) , T-1AT= coo)0 2 0 . a solution. Similarly, the solution is unique since the null space of A is -1 1 . 0 0 3 empty. 570 APPENDIX A. SELECTED HINTS AND SOLUTIONS 571 . F1 - ~n 1 th .l!Ll' 1 1 _ ~m 1 1 ~m-1 1 2 •.1 3 , or Xn- ~1.:= kf• e wuerence Xn-Xm - ~k=n+t kf ~ (n+ ) ~k=O ~ 1.5.1; (a) A'= i -12 -12 1) 1 1 1 1 (n!t)l is arbitrarily small for and n large. ( 0 0 0 ~ (n~t)l 1 ~'! < m 2.1.4; The functions {sinmrx}~= are mutually orthogonal and hence linearly (b) A'= ( 1 2~ i ! ~ ) independent. 2.1.5; maxt lfn(t) - fm(t)l = !(I- '!!;) if m > n, which is not uniformly small (c) A' = 1\ ( : ~ ) 2 0 -4 form and n large. However, J; lfn(t)- fm(t)!Zdt = (~2:':~ < 1 ~n. -6 2 2) 2.1.6; (a) Hint: Show that l(lul-lvl)l ~ ju- vi. (d) A' = i ( -5 4 1 -4 2 2 2.1.8; J; X(t) = 0. 1.5.3; (a) Take u = :c -v. 2.1.11; J; (I; f(:c,y)dx) dy = - J; (I; f(:c,y)dy) dx = -~. Fubini's theorem fails to apply because lf(x, y)ldx) dy = a:'!v' dx) dy = 1.6.4; Q = (~,.P,,.P,), where~ = ;ky (:) ,¢>. = }.;. ( ~0 ) ,.P, = J; (I; f; (I; J;; tan-1 ;dy does not exist. 2 2 3 2 2.2.1; With w(x) = l,p(z) = \:x + 16 , with w(:c) = v'f='?,p(x) = 1:1T (6x + 75 ~1 ) , and R- ~ 2 1( -(v'ffi *1Ji5 1f¥) . 1), and with w(x) = ~,p(:c) = ~(4x + 1). 3 1.5.5; Use that 2.2.2; Define the relative error e~,: by e~ = 1 - ~· where /A:(x) is the kth 1 ( 1 -2 ) ( ViO 0 ) ( 1 -1 ) partial sum, f~.:(x) =~!::a ant/>n(x). Then A = ViQ 2 1 0 2v'iQ .. 1 1 or (a) H(:c) - 2H(x -?r) = ~ ~::0 2 n~l sin(2n + l)x, e5 = 0.201, A = ( i ~ ) ( ~ ~ ) ( ~ !a ) · (b) :c -?r = -2~~ 1 ~ sinnx, e5 = 0.332, X 0 < X < 11" 1r 4 ~oo 1 ( 1) - 2 (c ) 211" - x 11" < x < 2?r = 2 - 1T ~n=o (2n+t)!l cos 2n + x, e5 - 1.5.6; Use that A = i ( ~ ~ ) ( ~ { ~ ) ( !2 ~ ) ,so that 0.006, AI (d) x2 (2?r- x)2 = 8 ~· -48 ~:: ~ cosnx, es = 0.002. = 15 ( -2} -2)4 . l 1 1 ~ 1.5.8; For A = ( ~:: ~::~ ) , singular values are ViO and eVlO with E = j D.J 1 . f:t'n I ( 0.} + Jw 0.2 - ~ ) ( 200.1 -99.8 ) ' 0.0 0.00 v LV, A = 0.1 - :Jrn 0.2 + :Jrn = -199.8 100.2 . ' ~ ·1 ':1 .... -~ Using instead singular values ViO and 0, A' = ( ~:~ ~:; ) . ·1.0 ,.. 2.0 0.0 D.J 1.0 1.1 2.0 0.0 D.J !.0 l :til no: ~ j 1.5.10; A= UJJV where U = ~ v'50 -1o -2 o ) , I:= ( 2V100 3VWo ) , ( Figure A.1: Left: Partial sum f~.:(x) for Problem 2.2.2a with k = 5; llight: 0 -2 1 0 0 Partial sum f~.:(x) for Problem 2.2.2b with k = 5. 1 V = 1 _ -1_ ) , so that A I = ( _rl..Yi20 SO1 .!.U 1 ) . ~ ( 1 1 1 . 20 30 15 2.2.3; f(x) =x(x-?r)(x-211") = 12~~ 1 ;?s-sinnx. 573 572 APPENDIX A. SELECTED HINTS AND SOLUTIONS :u 100 2.2.15; Suppose the transforms of j,g, and hare fJ_, and F• respectively. Use 1 21 :u 10 direct substitution and the fact that -k L,i=~ e riJk/N = 1 if k is an 1.0 10 integer multiple of N (including 0), and = 0 otherwise to show that hj = u 40 1.0 h9i· o.a 10 2.2.16; Rewrite the definition of the discrete Fourier transform as a matrix mul 0.0 . tiplication. Show that the matrix is orthogonal. 0.0 o.a 1.0 1.1 1.0 CLO CL6 1.0 ... 1.0 .til .til 2.2.19; See Fig. A.4. Figure A.2: Left: Partial sum fk(x) for Problem 2.2.2c with k = 5; llight: Partial sum /k(x) for Problem 2.2.2d with k = 3. 0.4 0.2 2.2.4; With the additional assumption that the function f(x) to be fit intersects the straight line at the two points x = x1 and x = x2, x1 < x2, we must o.o-1---~ have that x2- Xt =!,and x2 + x1 = 1, so that a=£/(})- !f(~),p = .0.2 2/(f)- 2f(i>· .(),4 2.2.6; (a) You should have a procedure that generates the Legendre polynomi 3 -2 ·1 0 als from Problem 1.1.9, or use Maple, which knows about Legendre X polynomials. Figure A.4: The wavelet generated by the B-spline N3 (x), for Problem 2.2.19. I.Q I.Q o.a ... 2.2.24; Solve the equation Aa = P where 0.0 0.0 2 1 0) /o- h) ...... A= 1 4 1 , P= -3 ( /o -h · ( 0 1 2 h-h •1.0 .... 0.0 CL6 1.0 ·1.0 .... 0.0 CL6 1.0 X X The solution is _2 Figure A.3: Left: Legendre polynomials P 1 (x),P2(x) and P3 (x); llight: Legen ! -1! ) ( a= _1 0 ~ lo)h . dre polynomials P4 (x),P5(x). ( 12 i -! i h 3 5 4 2 (b) g(x) = ax+bx +cx , where a= ~(1r -15311" + 1485) = 3.10346, 2.2.26; (a) Solve Aa = P where 4 2 4 2 b = -m<11" - 12511" + 1155) = -4.814388, c = ~(11" _ 10511" + 945) = 1.7269. 1 N 11 aij = ¢~'1//jdx, Pi=-~h t/J~'t/ljdx. (c) g(x) = 3.074024x-4.676347x3 + 1.602323x5 • A plot of g(x) is barely 10 i=O 0 distinguishable from sin 'II"X on the interval -1 :$ x :$ 1. Use Problem 2.2.23 to evaluate the coefficients, and find that the ith 2.2.7; Use integration by parts to show that the Fourier coefficients for the two equation is at-1 + 4at + ai+l = *(fi+l- /i-t), fori ::fi 0, N. representations are exactly the same for any function which is sufficiently {b) Solve Aa = P where smooth. 1 · N 1 2.2.9; (b) Write tPn+l - AnXtPn = r:,:=o PktPk, and evaluate the coefficients by aij = 1¢~¢jdx, PJ = - L: /i 1t/J~t/Jjdx. taking inner products with tPb and using part (a), and the fact that llt/Jkll = 0 ~0 0 1. Show that Bn = -An(XtPn 1 tPn) 1 On= An/An-1· Use Problem 2.2.23 to evaluate the coefficients, and find that the ith equation is -ai-1 + 8ai- at+l = lUi+l- /i-t), fori ::fi O,N. 2.2.14; Direct substitution and integration yields h(t) = 211" r:,:._ 00 fkgkeikt. 574 APPENDIX A. SELECTED HINTS AND SOLUTIONS 575 (c) Require ai-l+ 4ai + Gi+t = *Ui+t -/i-t), fori =f 0, N. 3.4.3; An = n281r2 is a double eigenvalue with ¢n(x) =sin n;x, '1/Jn(x) =cos n~x (d) Solve Aa = f3 where for n odd. 1 3.4.4; k*(x,y) = k(y,x):~~~· Uii = 1 '1/J~'(x)'l/J'j(x)dx, 3.5.1; (a) u(x) = f(x) +fox ex-tf(t)dt =ex when f(x) = 1. = 1 N 1 (b) u(x) f(x) + J; sin(t- x)f(t)dt = cosx when f(x) = 1. /3i = 1 g(x)'l/Ji(x)dx- 2.:::: /i 1 . + f0 f(t)dt = x + 2 ( 1~>.) when f(x) = x, provided 2.2.27; Observe that the equations for a , ... , GN-1 from Problem 2.2.26a and c 1 1 A# 1. are identical. 1 5 (b) u(x) = f(x) + ~ f xtf(t)dt = x when f(x) = :. 1 0 3.1.1; Use Leibniz rule to differentiate the expression u(x) = I y(x-l)f(y)dy+ 0 (c) u(x) = p(x) + a(x) I:o exp (-I: b(t)a(t)dt) b(y)p(y)dy. I: x(y- l)f(y)dy twice with respect to x. 1 II>.Kr+ "[" 3.2.1; Find a sequence of functions whose £ 2 norm is uniformly bounded but 3.6.1; (a) Show that llu- unll < 1_ j>.KII · whose value at zero is unbounded. There are plenty of examples. (b) Use that IIKII :::; to conclude that n 2: 2. The exact solution is ! 2 u(x) = 3~~ and some iterates are u0 (x) = 0, u1(x) = 1, u2(x) = {, 3.2.2; The proof is the same for all bounded linear operators; see page 107. ua(x) = 9 . 3.2.3; (a) The null space is spanned by u = 1 when A= 2, therefore solutions 3.6.2; If f(x"') = 0, for convergence to x*, require II-f'(x"')l < 1 or 0 < f'(x"') < exist and are unique if A =f 2, and solutions exist (but are not unique) 2. 112 if A= 2 and I f(t)dt = 0. 0 D • • 1f(x·v" (c) The null space is spanned by ¢(x) = cosjx if A = *· Therefore, 3.6.5; Show that Un(x) = 2:::~=0 ~ · if A =f *for j = 1, · · ·, n, the solution exists and is unique, while if 2 3.7.1; y = y(x) where a2x = sin-1 y'y + yJl- y2, a=~· for some j, then a solution exists only if '/r cos = 0. A = * J0 f (x) j xdx 2 2 3.7.2; (a) A straight line. 3.3.1; u(x) = f(x)+A I0 '/r 2:}= 1 i-~'lr cosjtcosjxf(t)dt = sin x- ~ 2_1r>.1r cos 2x, (b) y = y(x) where x = ln(y+ Jy2 -1), provided y 2: 1, provided A =f :. For A=:, the least squares solution is u(x) = ~· hJY2=1- ~ which means this problem is not physically meaningful, since y cannot reach zero. 3.3.2; u(x) = 3 _!6>.Po(x) + 3!2>.P1(x) + 15 ~6 >.P2(x), provided A =f ~~~· It is 2 helpful to observe that x + x = !Po(x) + P1(x) + 1 ~P2(x). 3.7.3; (a) T(p) = I;(p) V ~dz. c(z) 1-c~(z)p 3.4.1; (a) Eigenfunctions are ¢n(x) = sinn1l'x for An= n 2~ 2 . (b) Let y = ljc2(z), then use Abel's technique to show that z(c) = IJl/c ~ (b) f(x) = -;!-r E~o (~~~\)2 sin(2n- 1)11'x. -:; 1/co yc2p2-t' 3.4.2; (b) (c) There are no eigenvalues or eigenfunctions. Remark: The existence tsin(k+~)n d 4.1.1; a) Use that Sk ( x) = 2 sin for -1 x 1, an then observe that of eigenfunctions is only guaranteed for self-adjoint operators. ( if < < 2 1 siV¥ Sk(x) = (k + !) sinc((k + !)x) is a delta sequence. (d) ¢n(x) = sinanx, An= ai• where an= n:f . 577 576 APPENDIX A. SELECTED illNTS AND SOLUTIONS 1 4.2.12; g(x,y) = -!e-lv-xl, u(x) = ~ J.:O e-lv-xlu(y)dy- f~oo !e-lv-xlf(y)dy. 4.1.4; Observe that X= f~oo 1/l(x)dx is a test function, x(O) = 0, so that X= x(e- y))e = (g*(e,x),Len(e,v))e 4.1.10; In the sense of distribution, x'(x) = 5(x)- 5(x- 1), since (x'(x) 4.2.1; g(x,y) = -x for 0 ~ x ~ y,g(x, y) = g(y,x). 4.3.7; M*y =!fit+ AT(t)y, with the vector ( ~~~~~ ) E N..L([L, R]). 4.2.2; U(x) =xis a solution of the homogeneous problem. There is no Green's 4.3.8; (a) The operator is formally self-adjoint, but not self-adjoint. function. (b) Require u'(O) = u'(l) = 0, for example. cosa(;-lx-v\) 'ded ..J.. 4.2.3; g ( x, y ) = 111n f prov1 a .,... 2mr. 4.3.9; Require J:1r f(x) sinxdx =a, and J:1r f(x) cosxdx = {3. -(1- y)2x for 0 $ x < y ~ 1 4.2.4; g(x, Y) = { (2 _ y)yx- y for 0 ~ y < x ~ 1 1 ~.3.10; Require f0 f(x)dx = -{3. 4.2.5; g(x,y) = lny for x < y, g(y,x) = g(x,y). = 4.3.11; Require J0! f(x) sin 1rxdx {3 + 1ra. _x(3y5/2+2) for 0 ~ x < y 4.3.12; Require Jg f(x)dx =a- {3. 4.2.6; g(x, y) = { 113tf( -a/2 ) - 5 3x+2x for x > y 4.4.1; g(x,y) = yx+~(y-x)+l2 -~(x2 +y2 ),for0 ~ x < y ~ l,g(x,y) = g(y,x). 4.2.7; u(x) = -£:. f~oo e-alx-eiJ(e)cJe. 4.4.2; g(x, y) = (~ + y2 (4y- 3))(x- ~) + t- ~-X: - xH(y- x). 4.2.8; The functions sin 2x and cos 2x are elements of the null space of the op erator. 4.4.3; g(x, y) = -~ cos21r(x- y)- ~sin 21r(x- y)- 4~ sin 21r(y- x). 1 4.2.9; u(x) = J; g(x, y)f(y)dy-).. J0 g(x, y)u(y)dy+a(1-x)+{3x, where g(x, y) = 4.4.4; g(x, y) = ~ cosxsiny+~ sinxcosy- ~ sinxsin y-2H(y-x) sinxcos y- x(y- 1) for 0 ~ x < y ~ 1, g(x, y) = g(y, x). 2H(x - y) cos x sin y. 2 2 4.2.10; u(x) = fg g(x,y)f(y)dy-)..jg g(x,y)u(y)dywhereg(x,y) = i(x+1)(y-2) 4.4.5; g(x,y) = ~xy- x- -T(x + y ) for x < y,g(x,y) = g(y,x). for 0 ~ x < y, and g(x, y) = g(y, x). 4.4.6; g(x, y) = ~ ln(l-x)+ ~ ln(l +y)+ ~for -1 ::; x < y S 1, g(x, y) = g(y, x). 1 4.2.11; u(x) = f g(x, y)f(y)dy- ).. g(x, y)u(y)dt where g(x, y) = xn(yn - 0 J; 2~ y-n) for 0 ~ x < y ~ 1,g(x,y) =g(y,x). 4.4.7; u(x) = l cos2x + ({3- a)~;+ ax-~~ (a+~). 578 APPENDIX A. SELECTED HINTS AND SOLUTIONS 579 4.4.8; u(x) = cosx sin3x- /; sinx. -~xcosx + + f2 5.1.9; The Euler-Lagrange equations are y" = z, z" = y. 4.4.9; u(x) = 0. 5.1.10; The Euler-Lagrange equation is ~ - y = 0. 4.5.1; u(x) = aiJ3 - f +ax + 0;a x2 + n~:-2 cos mrx, where E:.:1 5.1.12; y(x) = 2sink1rx. bn = -2 J: (f(x) +a - /3) cos mrxdx. 4.5.2; Use Fourier series on [0, 211"]. 5.2.1; Uz = 0 at X= O,l. 4.5.3; Use Fourier series on [0, 1]. 5.2.2; Uzz = 0 and f..l.tUzzz- f..1.2Uz = 0 at X= 0, 1. 4.5.4; No eigenfunction expansion solution exists. 5.2.3; H u is the vertical displacement of the string, then T = ! J~ pu~dx + muHO, t) + mu~{l, t), U = ~ J~( .;(1 + u!)- 1)dx + fu2(0, t) + ~u 2 (l, t). 4.5.5; u(x) =ax+ /3- a1r + E:., Cln cos(2n- 1)~, 1 Then, require puu = f..l./z ~ subject to the boundary conditions where yl+u;: _ 8 1-coe(:ln-1)! mUtt + ku f..I.--J!.&...; at x = 0, and mUtt + ku -f..I.--J!.&...; at x l. Cln - ll'(:ln 1)2 (:ln-1)2/:1-1 • = yl+u;: = yl+u;: = 4.5.6; u(x) = -~(Jx:l- !) - !x. Solution is exact if a+! = 0. 5.2.4; puu = f..I.Uzz on 0 1 2 2 (c) iz- il < 2, iz- il > 2. 5.4.1; Use the functional D(¢) = J0 p(x)¢' (x)dx + ap(1)¢ (1), a 2:: 0, subject 1 2 (d) iz- il < 1,1 < iz- il < v'2, iz- il > J2. to H(¢) = J0 w(x)¢ (x)dx = 1 and (e) iz- il < 1, 1 < iz- il < v!z, v'2 < iz- il < 2, iz- il > v!z. (a) ¢(0) = ¢(1) = 0, (b) ¢(0) = 0, 6.2.6; ~zl=l/2 z21z~l dz = 0. (c) ¢(0) = 0, and a= 0. 6.2.7; ~zl=l/ 2 exp[z 2 ln(1 + z)]dz = 0 (There is a branch point at z = -1). Show that >.(a) 2:: >.(b)~ >.(c). 6.2.8; ~zl=l/ arcsin zdz = 0 (There are branch points at z = ±1). 1 2 2 2 5.4.2; Minimize J0 (x¢' (x) + ~ ¢ (x))dx, subject to J; x¢(x)dx = 1, with . r sin z d . h 1 ¢(0) = ¢(1) = 0, or ¢(0) = 0 with ¢(1) unspecified. 6 ' 2 ' 9 ' Jjzl=1 2z+i Z = 7r Sill 2· 5.4.3; Minimize D(¢) = ¢12 (x)dx subject to ¢ 2 = 1. Use ¢(x) 2 J; J; (x)dx = 6.2.10; ~zl=l ln~~~ )dz = 0. 2 v'30x(l- x) to find D(¢) = 10 2:: 1r • 6.2.11; flzl=l cot zdz = 27ri. 5.4.4; The first eigenvalue is approximated by 10, from Problem 5.4.3. The second eigenfunction is approximated by ¢ 2 (x) = V84ox(x- 1)(x- !), 6.2.13; Hint: Use the transformation z = e where p = ~ and apply the Phrag 2 2 and D(¢2) = 42. The exact values are .>. 1 = 1r , .>. 2 = 471' . men-Lindelof theorem to g(O = f(z). 6.1.1; (a) f(-3) = -iVM,f(!) = -ji,J(5) = -.,fiO. 6.2.14; The function G(z) = F(z)ei(a+<)z satisfies IG(iy)i :S A and jF(x)l :S SUP-oo arithmic branch point on the negative branch of the square root at 6.3.4; !), = R-q - 13 r2-R' Z - 12' . A. ·.J,_ 2(b ) 4i(b-a)l cz-1) 6.1.2; (e 21ri)z is not single valued. I 635• • ' 'f' + t'f' - a - 3 -a - 311" n z+l .g 2 6.1.3; (a) z = ! + 2n7r- i In(2 ± v'3). 6.3.6; (b) w( z) = Az + C~i - i)2, Fx - iFy = -8p1riA. i 2 (b) z = (2n + 1)7r- i In( v'2 + 1), z = 2n7r- i In( J2- 1). :2 (c) Fx- iFv = p7r(4rA- 8A i). / (c) No such values exist. 6.3.7; The upper half.; plane. 6.1.4; ii = e-(n/2+2n?r) for all integer n; ln(l + i}i1r = -7r2(t + 2n) + i; ln2; 6.3.8; Flow around a corner with angle(}= {31r. This makes sense only for (3 < 2. arctanh 1 has no value. . F 2 2 ia 6 • 3 • 9 , x - t·p y - - 1r pU2 a sm. ae . 2 3 2 6.1.6; It is not true. For example, consider z 1 = e1ril , z2 = e 1ri/ • 2 6.3.11; f(z) = UJZ2 + a • 6.1.7; The two regions are lzl < 1 and izi > 1; There are branch points at w = ±1. 6.3.12; Show that 1; = iJ (e-wf2k + vfl + e-w/k ). . f( ) - 15-8i 6 • 2 • 1 ' Z - 4(z-2)2(z-!)' oo dx 271' 6.4.1; J-oo ax2+bx+c = -/4ac-bl' 3 6.2.2; fc f(z)dz = -27rv'I9(15)11 e-i1r/3. . roo xsinx dx = .l!:e-lal. 6 •4 • 2 ' Jo al+x2 2 6.2.3; The integral is independent of path. fc z-113dz = !z 2 1 3 i~+~ = -3(2)1f3ei1r/ roo dx - ~. 6.4.3; Jo l+xk - ksin.,. 6.2.4; Find the real part of f(z) = z 112 and use that f is an analytic function. roo dx - _71'_. 6.4.4; Jo (x+l)xP - sin 1rp 6.2.5; (a) iz- il < v'2, lz- il > v'2. . Joo x dx = 71' • (b) iz- il < 2, iz- il > 2. 6.4.5, 1 (x2+4)-/x'Ll ' 2J5 582 APPENDIX A. SELECTED HINTS AND SOLUTIONS 583 6.4.6; fooo i!t dx = 72· 6.5.5; (a) W(Jv, Yv) = ;z. b) W(J. H(t)) 2i 4 ( 1/t II = 11"% • 6.4.7; J~ £~~dx = 211'H(ka)e-k , (c) W(H~1 >,H~2 >) = -~!· 6.4.8; Consider f0 4:~1. on the rectangular contour with comers at z = ±11' and z = ±11' - iR, and let R -+ oo. Show that f~w _,. !!~~~ .. dx = 6.5.6; (b) Yn(z) = - ~: r(~~l) + higher order terms. ~(H(a -1)1na -In( a+ 1)). 6.5.8; Use that '.E~=-oo Jn(z)tn = e 2 - 1 roo ln(1+111 ) .J- In 2 7.1.3; Use improper eigenfunctions to show (L4 A)- is unbounded. Show that 6 • 4 • 17; Jo , ...L-2 u;.G = w • {Xn} with Xn = sin n9 is an improper eigenfunction. 2 6.4.18; /~00 co::2 a:dz = '8 • 7.1.4; Show that f/>(x) = sinpz is ·an eigenfunction for all p. Notice that the operator is not self-adjoint. 6.4.20; Evaluate f0 P~(z) on some contour C that contains all the roots. 7.2.1; (a) o(x-~)=2:E~=lsin( 2 n2-l1Tx)sin( 2 n2-l11'~). 2 2 2 2 6.4.22; The change of variables p2 = r cos 9+z sin 9 converts this to an integral 00 for which complex variable techniques work nicely. {b) o(x- ~) = ~ fo coskxcosk~dk. 00 00 1 2 (c) o(x- ~) = ~ fo sink(x + 4>) sink(~+ l/>)dk where tan=~· 6.4.23; f0 t- 1 e'"'dt =.[fir exp(iifi[r). 7.2.6; (a) ~2~42· 6.4.24; tl/2ei"tdt = J:O vrre::!' •. (b) l!i~. - :1 6.4.27; f~oo (.. ~:~)1! dz = 2e0111 e-wi/J/'J sin ~IPI/3-t r(1- /3)H( -p). (c) .j!e=if-. 7.2.7; H(p+1r) -H(p-1r). 6.4.28; Use that V'z+lat!v'z+id = Yif{fa:'fJf"', but be careful to apply Jordan's 00 1 110 7 2 8· ..!.. - -(iuF(u))e-'":cdu- roo e:c-•J'(s)ds 1 1 Th oo f"", - ../i -iw/4e- -e-""H( ) • • ' 211" f-oo l+iiS ,.. ,.. ,.. - J:c ' emma correct y. en, J_00 v'z+at+OI+Vl - ~e 01 -jJ P • 7.2.9; Use the convolution theorem to find u(x) = f(x)- f~oo f(t)e-31:z:-tldt. 6.6.4; Examine the expression fo" (:l:JE. .. , make the change of variables x = t 2 2 ycos () + '7sin 6 and then use the beta function to show that f(y) = 7.2.10; f~oo J(e)J(x- e)cie. ~Ju Jt (,~Jjt .. dz. For another way to solve this problem, see Prob lem 7.3.4. 7.2.11; (a) E:z:E~ = ~ which is optimal. 584 APPENDIX A. SELECTED HINTS AND SOLUTIONS 585 2 2 (b) E 111 Ep = ti = 0.274. 7.5.5; (a) RL = RRe- ik 7.2.12; (a) J~ eiJ.Iz Jo(ax)clx =f. J:,.. eilll(p+aaln8)dx) d(J 7 5 6. R = -e2ikta ~~:a cos kaa+ik, sin kaa where k· = .!!!. 00 (J:O ' ' ' ka coa kaa-illt aln ll:~a ' c1 • = 6(1-4 sin 8)d8 = if < and = 0 otherwise. 2 fo~,.. + a J a; -p2 II' I lal 7.5.7; The gene~al solution is u(z) = aeikz(k + 1 + 3iktanhx- 3tanh~ x) + 2 (b) Take the inverse Fourier transform of the answer to part (a). ,Be-ikz(k2 + 1-3iktanhx- 3tanh x). Tr(k) = ~!Z+~H!Z+!~. The bound 2 states are ¢1(x) = tanhx sechx fork= i 1 and ¢J2(x) = sech x fork= 2i. 7.2.15; Show that Ne(1) = 1/120, N6 (2) = 13/60, and N8 (3) = 11/20. Then - _L ~ 1l + 11 13 + 1 Es( z ) - wZ + eoZ 20 + GoZ 120.:2 ' 7.3.2; (a) 8-(a+I>r(a + 1). (b) provided Re 8 > a. a.:. 7.5.13; Use (7.37) to show that cu = 2 ~~:(Li), Ct2 = c~1 = 2~~$~) 1 C22 = - 2A:(}H) · 1 ez/-./2 z < 0 7.3.3; u(z) = J; f(y)K(z- y)dy where K = L- ( +thl,n)· 1 There is a bound state having f/J(z) = { e-zf../2(1 + ~tanhx), x > 0 1 7.3.4; f(t) = linw7r(! 1t f~ Ta- T(t- T)dT. at k= ~· 7.3.6; (a) M[H(z)- H(z- 1)] = ~· ~ 1inh~ 2 b. 8.1.2; (a) Vn(x) =an~ +bn sllih , where an= -1i f am Tg(y)dy, (b) M[(1 + z)-1] = aln,..n' 0 bn f(y)dy. 111 = ~ J: sin T (c) M[e- ] = r(3). (b) u. ( ) = aalnh(Df{u-11)) ~~~sinh~!: (e)t~e+ aalnh ~ t sinh(!!!!:(e- (d) M = 1:,. n Y nwalnh li£1 0 a n mrslnh • 11 a b))F(e)de where Fn({) = !!!!:(!({) - ( -1)ng({)). In the special (e) M[eilll] = i'r(8). 1 a a 2 case f(y) = y(b - y) 1 g(y) = y (y - b), an = - n'Y',..s (1 + 2( -l)n)), (f) M[cosz] =! COB11'8r(s). bn = n1~1 (1- ( -1)n). The first representation converges much faster 00 7.3.7; F(J-4) = f r/(r)sinJ.4rdr,rf(r) = ~ f 00 F(J.4)sinJ.4rdJ.4. than the second and gives a good representation of the solution with 0 0 only a few terms. 00 7.3.8; (a) Show that J Jo(z)dx = 1, and then G(p) = ~· 0 8.1.3; (a) Require f~oo f~oo z( cosz,z > 0 { sinz1 z > 0 8.1.5; G(z,Zo) = - 2~Inl:.:~:_o1 1· 7•5•1i Ut(z)= { coshz,z if and only if Aa~ > "": . 8.1.8; u(x,y)=En=t00 ( aniiiilillf£+bn~ ~h~)•in cosT, where 2111 (d) C _ 1 + 2ik(a+P}+agce- "-1) c _ -e2ika (2i/!k-,1a)+~2iak+,8a) 2 4 4 11 - 4k I 12 - 41; ' an =- ~?!~ (120( -l)n- 60n ~ + n 1r + 600), 2 2 2 >. = -p where tanh ap = - (a+p)(p(a+J-+2P) +JA)+p2. bn = ~?!~ {120 + 600(-l)n- 60n 1r ( -l)n + n411'4). 586 APPENDIX A. SELECTED HINTS AND SOLUTIONS 587 8.1.9; G(x,y;~,r~). , = ""'ooL.,.,n= ..Lsin~sin~e-~lx-elnn- a a . 1 8.1.29; Eigenfunctions are u(r,fJ, 8.1.12; If f(O) = l::':0 ancosn(8- 8.1.16; Hint: Evaluate L:f= 1 rJ cosj.= v'2~, whereas where F(k) = Iooo f(r)eiklnrd;, G(k) =It g(-r)eiklnrdr. for a circle of radius R the fundamental eigenvalue is.\= ~· Take 1r R2 = L2 and use that 2 ~w = .\. The fundamental frequency for the circle 8.1.18; u(r,8) = 2::'=-oo J:{<:R)einll, where an= ~ I:,. e-inllf(8)d8, and In(x) is smaller than for the square, >.ftrc!e = 0.959>.~Iuare. is the modified Bessel function of first kind.2 8.2.12; G(r) -~H~ 1 )(r) is outgoing as r-+ oo. 8.1.20; u = a2!jj2 (af- /3H(f)). = 1 00 8.2.13; Construct the Green's function from H~ ) (>.jr - el) with A = ~, and 8 1 21· u(r 0) = ""' n_ ~-rlnleinll where a = .1.. f 2,. g(O)e-inll df) ' • ' ' L..,n=-oo ...... ci+lnli3 ' n 2,. Jo • then the solution is proportional to (up to a scalar constant) 1/J(r, 0) = 1 ei>.r sin(>.asln 9) for large r. 8.1.22; -v(~') = Inn . v~G(x, e)J(x)dVx - Ian n . veG(x, e)v(x)dSx, where 7r >.ainiJ _ 8u V - lfn· . C8V _ 1 8 2 V V 8 ' 3 ' 1 1 8t - R1 8z'I" - R2 . 8.1.24; Eigenfunctions are ~ 8.1.25; Eigenfunctions are Jn(J.'n!cfi)sinnfJ for n > 0. Thus, eigenvalues are the 8.3.4; G(x, t) = J;te-tf-at. same as for the full circle, with n = 0 excluded. 2 2 2 4 8.1.26; Eigenfunctions are (b) 2Jt = e-Un-1 _ e-Un+l, 8.3.9; t = At~ In ~ = 5.5 hours. At this time the temperature at the bottom of the cup is 116° F. This seems like a long time. What might be wrong 9.2.3; qxt = sinh q. with the model? 9.3.2; The bound state has J.L = -4· There is a single soliton with amplitude 2 2 8.3.10; (a) Require ut = Di"iJ2u on regions i = 1,2 subject to the conditions !A2 and speed c = 4J.L = A . that u and n · kiVu be continuous at the interface. 64 9.3.3; Use that r(x) = 6e-x+Bt + 12e-2x± t and find a solution of the form (b) In a spherical domain, eigenfunctions are (9.18) with~= x- 4t, 'f1 = 2x- 32t. 0 < r < rp cp(r) = /.lain~ r-h' 9.3.4; One soliton will emerge. { ram , rp < r < R . !l£u. i !ku - 0 9 • 3 • 5 ' dt = - 2k cu' dt - • The tempera! behavior is and the values I" must satisfy the cp(r)e-1h 9.4.3; Verify (9.20) and (9.21). transcendental equation ~tan~ = --jt; tan e~>. 2 = = 2 4 9.4.4; (a) a= ~~ 2 z- i, (3 1- a, R ~· (c) /-' j::::j 6.6 X 10- fa. 1 = _.! (4a5-l~f4a5zl-1) (3 _ 1 zl(16a~-4a5+1)-4afi Th t (b) a 4 a5z z~-1) ' - 4 a5(zL1) . ere are wo 8.3.11; Solve the transcendental equation Jh:ie-~-ot = (J fort. The velocity is 2 4a2 h rootsof(3(z)=Oatz = ._, ?~ ... t' (c) a= _ 2B (3 = zl±2Bz-1 8.3.12; Propagation occurs provided h3(J < /(l) = 0.076, where ~~ z2-1 · f(y) (41ry)-312e-1/ 4v. The propagation velocity is v where /( lg.) = = 9.4.5; anWn = a~z) (z- ~). h30, and the minimal velocity is i~· 2 9.4.6; With q > 0, set a = !e-qo/2 , a !eqo/ , and then use Problem 9.4.4b 8.3.13; 0 0 1 = to show that two solitons moving in opposite directions emerge. u{:c, t) = ei<.lt (1 iw: ia:-y)eio(L-z) - (1 iw _ ia:-y)eio(z-L) + + 9.4.7; Set b = -~ =F 0 and then use Problem 9.4.4c to show that one soliton (1 + IW + ia:-y)eioL _ (1 + iw _ ia-y)e-iaL , 1 emerges. 2 2 where a2 = 1 + iw. 9.4.8; Choose spring constants kn with 4k; = 1 + sinb wsech nw. 8.3.14; p(:c, t) = e"-'t P(:c), where P(:c) = exp( -q(w):c), q:l(w) = k+~""'. 10.2.1; En(x) = r(:) L~o(-1)krtltf>. 8.4.1; Un(t) exp( -e•lniw/k) )2) sin(.ar). H we set n and h = we = = llf, l, 10 2 2· r1(cosm + t2)eixtdt = (-i. + 2 + 2i )eix _ 2i + i(1 + eix) '\'oo (2!:)2k. • ' ' JO X X L.Jk=O X - have in the limit k -+ oo, u(:c,t) = exp(-!jf.)sin(ar), which is the ~ ~ ~ correct solution of the continuous heat equation with periodic initial data. 10.2.3; I; eixtt-lf2dt = Iooo eixtrtf2dt- It eixtt-ll2dt _ (;i + ei"' '\'oo (-1)k-1 r(k±l/2) 8.4.2; Un(t) = Jn(-f). - V x 7/r L.Jk=O xk+t · 2 2 2 8.4.5; Un(t) = J2n(f). 10.2.4; C(x) = !A- P(x) cosx + Q(x)sinx , S(x) = h/I- P(x) sinx - 2 8.4.6; k(w) = cos-1 (1- ~~~~h 2 ). Q(x)cosx , where P(x) = ~ (~- ~ + · · ·), Q(x) = 12 (lx - ~4x + ~16x + .. ·) . 8.4. 7; For stability, require -2 < 6t.\ < 0, for all eigenvalues ,\ of A. 10.3.1; E1(x) =e-xl:~ 0 (-1)k Jb. 8.4.8; For stability, require that all eigenvalues of A be negative. roo e-•t '\'00 ( )k (4k)l 2 10.3.2; Jo l+t' dt = L.Jk=O -1 z'41'+T. 9.1.3; q(:c) = -2H(:c)sech :c. (See Problem 7.5.13.) rl -xtt-ll2dt /Jf -x r;; '\'00 1 9.2.2; (a.) !l9f = ian(a~+l -a~). 10• 3 • 5 i Jo e = V z - e Y 1r L.Jk=O xk+it(l/2-k) · 590 APPENDIX A. SELECTED HINTS AND SOLUTIONS 591 00 10.3.6; f e:ctt-tclt = y'21jiell (1 - - ~3 + · · · ), where 71 = ez-1. 2 3 4 0 2!11 6 112 11.1.3; X1 = 1- E + 3E - 12c: + 0(E ), X2,3 = ±~ -! ± ~iE + !e2 + 0(~). 2 10.3.7; I: F sin2 1rtdt = ~ - 1 ~~ + (50 - 81r2);; + .. ·. 11.1.4; ¢ + i.,P = U(z- a:)+ iEa(i)20 + 0(E2). 1 10 3 8. rff ezt2t-1/3 cos tdt - 1 2 (b) 2:~=0 ( ~ ) kl..\" = Iooo e-z(1 + ..\x)ndx,.., (n..\)ne-n+! ex:1 >fii· 11.1.8; u(r,O} = rcosO + c:(£{1- r 4 ) + -fi(r - r 4 ) cos28) + O(e). 2 10.3.13; Iooo flee-t Intdt = lnx( 11.1.9; u(r,O) = rcosO- !c:r sin29 + O(e2). v'21rxz+te-z 2zb - 2..;m + · · ·). 2 2 4 4 6 11.1.11; sfv = c: k /2 + c: k /4 + O(c: ). 1 13 10.3.16; ( ~ ) ,..., 1.82 X 10 . 2 2 2 u h 1 f •d h 2l 2wk sinh 2kl+2k 1 1 11112• . ; ror a c anne 0 Wl t , 8 = E T _, L!l nLI nL'ii'I + .. ·. 3 4 10.4.1; Io e:~~~;> dx = 21rie- "1 - 2ie-• l:J!o( -4)i ri~ttfl>. 11.2.1; (a) Eigenvalues are 1, 2, 3, independent of E. Eigenvectors are ( -2, 0,1- c:}T, (0, 1,0)T, and (0,0, l)T. 2 3 10.4.3; I(x) =I: exp[ix(t + t3 /3)]dt = J!"e- z/ (1- ~ + 48~\~ 2 + O(x-3)). (b) The eigenvalue ,\ = 1 has algebraic multiplicity 2, but geometric mul tiplicity 1, so the "standard" perturbation method fails. The eigen 10.4.4; Jn(z) = {£ cos(z- Df- f)- ~{J; sin(z- Df- f)+ O(z-612 ). values are ..\ = ±~ = 1 ± (c: 112 - !c:3/ 2 -}E6/ 2 + 0(c:7/ 2)), with corresponding eigenvectors (1,z2)T where x2 = ±.j6 = ±(c:1/ 2 + 10.4.5; l'nlc = (2n + 4k- 1)f - t,; ~:;!Z~ + O(b ). 2 2 1 1/2c:3/2 + 3/&6/2 + O(e7/2)). 1 1 10.4 6· r cos(xt")dt = ,1(.!)1/Prl)- ie • • , Jo ,. z ,. PIC • 11.2.2; y(x) = sinnx + ~(-xsinnx -1rnxcosnx + nx2 cosnx), ,\ = n2 - E~ + 2 112 o(e). 10.4.7; Ioff/ {1- ~} cos(:ccosO)dD 11.2.3; (b) For X1 sufficiently large (since f(x) has compact support), ,\ ""Ao + =& {ew(~-~(~+iv'ID+···)} -yt~+···. e-zt J/(~tofgj)dz J 0 :» dz 10.4.8; H x > 0, the change of variables s = converts this to !v'zi(x312), VZf 2 where I(x) is defined in Problem 10.4.3. H :c < 0, Iooo cos(sx + s8 /3)ds"' 11.3.1; Steady solutions are at 8 = 0 and at cosO= ;&r provided 0 > f. 2(-~1/4 cos('f +f). 11.3.2; y(x) = c:sinn1rx 2 + €2(1-coemrz) +2zcoamrz -E2( 2 _ 1)s;nn'/r:C +2E2e\nmrz-n7rZCOinlTZ 10.4.9; Use the answer to Problem 10.4.8 to show that the kth zero of the Airy 3~211'2 Srtll* ~ 8nlirJ cos mr ' ..\ = n21r2 + t,i;r(1- cosn1r) + 0\~). function is at x, -J(-4k + 1)1r. 2 2 2 11.3.3; Bifurcation points occur at solutions of ~(1-/:4) = n 1r and there are oo -•'1 . fl[ 1 3 10.4.10; (a) f-co wdq,..., v Q"(l- ~ + ici2' •• ·), for large a. a finite number of such solutions. The nth solution exists if ~ > n21r2. 2 2 2 2 2 4 10.5.1; I: f(x)e'"'(:c)dx = f(a)r(~)("'<~ca) )2/3eiltg(a). 11.3.4; u(z, y) = c:sinn1rxsin m;v + 0(E ), ,\ = -(n + !m )1r - !6 E a + 0(E ). 2 4 6 10.5.4; Set u(x, t) = e-d/ w(x, t) so that Wtt = W:cz + ~w. 11.3.5; u = e¢ + O(c: ), ..\ = .\o - ~a4 f ::: if a4 f:. 0 whereas u = c:¢ + O(c: ), 11.1.1; X1 = -0.010101, X2,3 = -0.49495 ± 0.86315i. ,\ = Ao - c: 4 a5 f 3 "" 11.4.8; Require a= (¢) = - . ~w~ cos(¢). 15 2 stn :l .. 10 11.5.1; There are two Hopf bifurcation points, at A= ! ± ~- 11.5.3; Steady state solutions have s = >., x = (>. + a)(l- >.). A Hopf bifurcation occurs at >. = !(a+ 1). -o.• -o.2 0.0 0.2 o.• ), 11.5.4; Suppose '1/Jo satisfies AT '1/Jo = 0, and 'l/;1 satisfies AT '1/Jt = -iA'l/;1. Then a small periodic solution takes the form x = e(a¢o + bei>.t¢1 + be-i>.t¢)1) + Figure A.5: Plot of ifJ as a function of A for Problem 11.3.6. 2 2 2 O(e ) provided 2lbi (Q(¢1, ¢}1), '1/Jo) + a(B¢o, '1/Jo) + a (Q(¢o, ¢o), '1/Jo) = 0, where 2a(Q(¢o, ¢1), 'l/;1) = -(B¢1, 'l/;1). 11.3.7; {a) u(x) = Aa(x), where A satisfies the quadratic equation 12.1.1; u(t, E)= a(1 + ~) cos((l- ~t- E)+ O(e3). A2 fol a3(y)dy- ~ +! fol a(y)dy = 0. 12.1.2; u(t) = A( Et) sin( (1 + e2w2)t) + eB( d) cos( (1 + e2w2)t) + 0( e2) where At = (b) u(x) = Aa(x), where 1 = Asin(A) a 2 (y)dy. J; 2 2 2 !A(1- ~A ), Bt = !B(1- ~') + {!,(A - 4)(6A -1), and w2 = -l6 • "" a.o 12.1.3; The solution is u(t) = eA(r) sin(t) + O(e2 ) where AT= !A(1J(r)- ~A 2 ), 2.11 2 15 and f.L = E 1J, T = d. 2.0 -.: 10 3 ~ ... c 12.1.4; (a) The Landau equation is AT= -iA , with solution A(r) = ,; 3 ]t~,,, 1.0 o.o where A(O) = Ao, so that u(t) = V Af sin(t + ¢o) + O(e). 1+ 4 A oft 01 0.0 1 I I I i I :p=:, I 2 3 o.o o.z 0.4 o.e o.a 1.0 1..2 1.4 1.1 ... .. -:z 0 (b) The Landau equation is AT = - ~ A , with solution A( r) = , , 1r_Ap~_ ), ), 3 where A(O) = Ao. 2 2 Figure A.6: Left: Plot of A as a function of A for Problem 11.3.7a with a(x) = 12.1.5; Take a= 1 + E a2, take the slow time to beT= E t and then the solution 5 2 sin(x); Right: Plot of A as a function of A for Problem 11.3.7 with a(x) = sin(x). is u(t) = A(r) sin(t + ¢(r) + O(E) where f/JT = !(a2 - 12 + ! cos ¢), AT = ~sin ¢cos¢. 11.3.0; u(x) = exp(Acosx), where A= Af~1r eAcoau cosydy. 12.1.6; The leading order solution is u(t) = A(r)sin(t + ¢(r)) where r = d and AT= iA2 cos(¢), ifJT =~A sin¢. 11.3.10; Solve the system of equations v" +n2 rr 2v = e( 1 (b) u(x) = 4::~:-:_1 - 2tanh(z;t + tanh- (i)) 12.1.12; Write the equation (12.16) as the system u 111 = r(~)v, Vz = J(x, ~),and 3 then make the exact change of variables u = Uo EW(~)z, v = z EZ, 1 + + (c) u(x) = H(x- i> 51:+h + t(l- H(x- i)) ia:-:!3 - ~ tanh(lf(x -!)) where~= r(u)-r, and Z = J; J(y,u)du-u J: f(y,u)du, with u = ~· where H(x) is the usual Heaviside function. 2 12.1.13; Set v = z + Eh(~)u where h' = g- g, and then !!if = z + eh(~)u, ~ = 12.3.8; u(x) = a+~-l + dn(cosh( z-<;!.B-l)) + O(e). -gu- Eh(~)z -·f2h(~) 2 u. L2.3.11; (a) T"' 2ln 2 ~a. 12.1.15; In dimensionless time, rc has units of length-2• A good dimensionless (b) T(v) = ln !~ where c(v) = v':~v!i· parameter is E = ~, r = r c + r m/ h. The function W is a piecewise linear -r!1 !+-r :, "sawtooth" function with slope - roh?frm • 12.1.16; Write the equation as the system u' = v,v' = u- (1 + g'(~))u8 + Eav. Then the transformation v = z + Eg( ~) transforms this into a system to which Melnikov's method can be applied. The requirement is that af~oo U/?(t)clt = J~00 9'(!.¥)U8(t)Ub(t)clt, where Uo(t) = v'2sech t. 12.2.2; The device is a hysteretic switch. There are two stable steady outputs, Vo = V~ if V VR+ for v > AvR+ 0 12.2.3; (a) i = "Ri' where v0 = i for AVR- < v < AvR+ , where A = { VR- for v < AVR- Bl~'iz;· (b) The current in the left branch of the circuit satisfies C RpR• ~ + R.i2 = v- v1 , thus the effective inductance is L = CRpR•. 12.2.5; u(t) = rh + e-t/e + O(E). 12.2.6; u(t) = 1 ~1 + O(E}, v(t) = (t+!>:~ + e-t/'- + O(E). 11 12.3.2; To leading order in E, u(t) = -ln(~) - ln(2)e-t/ 2 '- '(.~ sin l3ta + cos~). 1 1 12.3.3; u(t) = -tan- (t) E 13 exp( + 2;ih) (l sin( 2~a)- cos(~)) 1 1 1 El/322/3 exp(2 / (t-l)) + i ,_1/1 12.3.4; For all {3 there is a solution with u(x) ""'-1+ a boundary layer correction at x = 1. For {3 > 0.2753 (the real root of 3x3 + 12x2 + llx-4 = 0}, there is a solution with u(x) ""'2+ a boundary layer correction at x = 1. 111 1 ·12.3.5; The solutions are u(x) = ;h - tanh( ;[ ) + O(E) and u(x) = ~ - tanh( 111 2?) + O(E) with 1/J. and fb appropriately chosen. 12.3.6; The two solutions are u(x) = ::;:~ + 2e-z/e + O(E}, and u(x} = ::~ - 111 fe- /'- + O(E). 12.3.7; (a) u(x) = ;h-tanh(fe- tanh-1{j)). INDEX 597 Cauchy's equation, 234 LU, 32 Cauchy-Riemann conditions, 215, 355 QR, 33,36 causality, 149, 369, 414 singular value, 37 characteristic polynomial, 11 spectral, 15, 37, 52 Chebyshev polynomials, 50, 73, 168, degenerate kernel, 112 Index 281 delta distribution, 139, 340 chemostat, 503 delta sequences, 134 circulation, 235 dendrites, 409 coaxial cable, 365, 380, 406 difference equation, 170, 324 Abel's equation, 158 Bessel's inequality, 68, 111, 136, 163 commutator, 418 difference operator, 310, 332 Abel's integral equation, 103 beta function, 261, 281 compact differential-difference equations, 390 activation energy, 539 bifurcating solution branch, 479 disc, 300 diffraction pattern, 376, 407 adiabatic invariant, 511 bifurcation point, 479 operator, 111 diffusion coefficient, 380 adjoint matrix, 13 biharmonic equation, 474 support, 87, 137 diffusion-reaction equation, 380, 382, adjoint operator, 106, 152 bilinear transformation, 237 complementary error function, 442 383, 389 admissible functions, 177 binomial expansion, 70, 279, 443 complete set, 68, 113 digital recording, 300 age structure, 102 biorthogonality, 52 complete space, 61 digital signal, 463 airfoil, 241, 243 Biot-Savart law, 343 complex dilation equation, 80 Airy function, 467 bistable equation, 549 calculus, 214 Dilbert space, 65 algebraic multiplicity, 11, 43, 52, 476 Blasius theorem, 233 conjugate, 209 dimension, 3 amplifier, 523 blood flow, 367 differentiation, 214 dipole, 230 analytic continuation, 227 bound states, 317 functions, 211 dipole distribution, 139 analytic function, 214 boundary conditions integration, 217 dipole source, 342, 358 Argand diagram, 209 Dirichlet, 338 numbers, 209 Dirichlet boundary conditions, 338 Arrhenius rate function, 539 inhomogeneous, 154 velocity, 231 Dirichlet-Jordan convergence theo- associated Legendre equation, 268 mixed, 338 computer aided tomography, 105 rem, 164 associated Legendre functions, 167 natural, 185, 205 conformal map, 236 discrete Fourier transform, 76 asymptotic sequence, 438 Neumann, 338 conservation law, 380 discrete sine transform, 77, 170 asymptotic series, 437 periodic, 152 constitutive relationship, 380 dispersion, 395 averaging theorem, 512 separated, 147, 152 continuous linear operator, 105 dispersion relation, 463 boundary element method, 358 continuous spectrum, 284, 312 distributions, 138 B-splines, 81, 89, 304 boundarylayer,521 contraction mapping principle, 122, delta, 139, 340 Banach space, 61 boundary operator, 151 313,498 dipole, 139 band limited, 299 bounded linear functional, 108 convergence, 61,437 Heaviside, 139 basis, 3 bounded linear operator, 105 convolution, 172 higher dimensional, 339 Battle-Lemarie function, 305 branch cut, 211 integral, 78 regular, 139 Belousov-Zhabotinsky reaction, 557 branch point, 211 kernel, 296 singular, 139 Bernoulli's law, 234, 243 Bunyakowsky inequality, 6 theorem,294,295,304,356,384 divergence, 228, 234 Bernstein polynomials, 69 cosine integral transform, 290 divergence theorem, 338, 380 Bessel functions, 69, 169 Calm-Allen equation, 382, 549 Courant minimax principle, 19 domain of an operator, 106, 151 modified, 265 Calm-Hilliard equation, 383 cubic splines, 89 dot product, 4 of the first kind, 263 canonical variables, 188 curve fitting, 25 drag, 232 of the second kind, 264 CAT scan, 105 cycloid, 180, 185, 204 dual manifold, 153 of the third kind, 265 Cauchy inequality, 6 Duffing's equation, 194 spherical, 364 Cauchy integral formula, 220 Daubechies wavelets, 82, 307 Bessel's equation, 169, 262 .Cauchy sequence, 61, 68 decompositions eigenfunctions, 161, 283, 359, 362 596 598 INDEX INDEX 599 improper, 296 Fresnel integrals, 464, 466 heat equation, 338 Lebesgue, 63 nodal curves, 361 Frobenius norm, 41 Heaviside distribution, 139 line, 217 eigenpair, 11, 161 Fubini's theorem, 64 Heisenberg uncertainty principle, 334 Riemann, 63, 217 eigenvalue, 11, 161, 199, 283 function spaces, 60 Helmholtz equation, 360 interpolation, 78, 397 eigenvector, 11 functional, 108, 177 Hermite functions, 89 invariant manifold, 14 elastic membrane, 194 fundamental theorem of algebra, 276 Hermite polynomials, 50, 73, 169, inverse integral operator, 110 elastic rod, 189 fundamental theorem of calculus, 218 298 inviscid fluid, 228 elastica equation, 193 Hermitian matrix, 14 irrotational flow, 229 elementary reflector, 34 Gabor transform, 301 Hilbert space, 65, 283 isolated singularities, 226 elementary row operations, 30 Galerkin approximation, 92, 124, 131, Hilbert transform, 355 isoperimetric problem, 181 entire function, 226 176, 197 Hilbert-Schmidt kernel, 103, 112 isospectral flow, 417, 418 enzyme kinetics, 523 Galerkin method, 349 Hilbert-Schmidt operator, 112, 161 iterative solution technique, 121 equivalent matrices, 10 Galerkin projections, 399 hodograph transformation, 245 essential singularity, 227 gamma function, 259, 441 Hooke's constant, 189, 319 Jacobi iterates, 45 Euclidean inner product, 4 gauge function, 438 Hooke's law, 368 Jacobi polynomials, 73, 168 Euclidean norm, 5 Gauss-Seidel iterates, 46 Hopf bifurcation, 494, 496, 510 jet nozzle, 243, 246 Euler column, 193 Gaussian elimination, 30 Householder transformation, 34 Jordan's lemma, 253, 291, 292, 353, Euler identity, 209 Gegenbauer polynomials, 73, 168 413, 450, 461 Euler-Lagrange equation, 178 Gel 'fand-Levitan-Marchenko ( GLM) ill-conditioned matrices, 40 Josephson junction, 406 exponential integral, 440 equation, 413, 415, 433 implicit function theorem, 470, 475, Joukowski transformation, 242, 311 generalized Green's function, 159 513 jump condition, 147, 159 fast Fourier transform, 77 generalized Laguerre functions, 168, improper eigenfunctions, 296 Fermat's problem, 179 281 improper eigenvalues, 296 Kaplan matching principle, 530, 541 Fick's law, 380 generating function indicia! equation, 263 kinetic energy, 186, 188, 204 filter, 78, 296 for Bessel functions, 266 induced norm, 6 Kirchhoff's laws, 365, 485, 523 finite element space, 88 for Legendre polynomials, 269, inequality Korteweg-deVries {KdV) equation, finite elements, 88 280 Bessel, 68 418 FitzHugh-Nagumo equations, 503 geometric multiplicity, 11, 52, 476 Bunyakowsky, 6 Kronecker delta, 3 formally self-adjoint, 153 Gram-Schmidt procedure, 8, 34, 39, Cauchy, 6 Kutta condition, 242 Fourier coefficients, 68, 136, 162 65, 68, 114 Schwarz, 6 Fourier convergence theorem, 74 Green's formula, 338 Sobolev, 66 .L'Hopital's rule, 251, 264 Fourier cosine integral transform, 290 Green's functions, 146, 284, 339, 383 triangle, 5, 60 lady bugs, 366 Fourier integral transform, 290 generalized, 159 influence function (see also Green's Lagrange multiplier, 18, 180, 198 Fourier series, 68, 74, 164, 303 modified, 159, 175 function), 146 Laguerre polynomials, 50, 73, 168, Fourier sine integral transform, 289 Green's theorem, 217 inner product, 4, 65, 108 298 Fourier transform theorem, 290 inner product space, 4 Landau equation, 509, 593 Fourier's law of cooling, 380 Haar functions, 83 integral equations Langrangian, 187 Frechet derivative, 178 Hamilton's equations, 188 Abel, 103 Laplace transform, 307, 331 Fredholm alternative, 24, 110, 156, Hamilton's principle, 186 Fredholm, 101, 102 Laplace's equation, 223, 229, 337 476, 493, 507, 514, 516 Hamiltonian, 187 Hammerstein, 500 Laplace's method, 442 Fredholm integral equation Hammerstein integral equations, 500 singular, 103 Laplacian operator, 337 of the first kind, 101 Hankel functions Volterra, 101 lattice, 22, 310, 420 of the second kind, 102 of the first kind, 265, 407 integration Laurent series, 225, 267 free boundary problems, 243 of the second kind, 265 by parts, 217, 338 least squares problem, 26 Frenet frame, 489 Hankel transform, 309, 352 complex, 217 least squares pseudo-inverse, 28 Frenet-Serret equations, 489 Hardy-Weinberg proportions, 527 contour, 220, 291, 353 least squares solution, 26 600 INDEX INDEX 601 Lebesgue dominated convergence the moment of inertia, 53 Hermite, 50, 73, 169, 298 reflectionless potential, 323 orem, 64, 94, 134 Moore-Penrose least squares solution, Jacobi, 168 region of influence, 369 Lebesgue integration, 63, 94 28 Laguerre, 50, 73, 168, 298 Reisz representation theorem, 138 Legendre function Moore-Penrose pseudo-inverse, 28 Legendre, 9, 50, 72, 167 relaxation oscillations, 545 of the first kind, 269 mother wavelet, 80 orthogonality, 7 removable singularity, 226 of the second kind, 269 multigrid method, 4 7 renewal processes, 102 Legendre polynomials, 9, 50, 72, 167 multiplicity Parseval's equality, 69, 113, 271, 293 residue, 225 Legendre's equation, 262, 268 algebraic, 43, 52, 476 path independence, 219 residue theorem, 225, 285, 414 Lewis number, 540 geometric, 52, 476 pendulum, 148, 188, 510 resolvent kernel, 120, 128 lift, 232 multiscale method, 509 perihelion of Mercury, 183, 483 resolvent operator, 110, 118 linear combination, 2 periodic boundary conditions, 152 resolvent set, 283 linear functional, 108, 137 natural basis, 3 Perron Frobenius theorem, 44 resonant frequencies, 24 bounded, 108 natural modes, 23 Phragmen-Lindelof theorem, 222 Riemann mapping theorem, 237 linear manifold, 14, 107 natural norm, 6 pitchfork bifurcation, 482 Riesz representation theorejll, 108 linear operator, 105 near identity transformation, 512 Plancherel's equation, 293 Rodrigue's formula, 268 linear vector space, 2 Neumann boundary conditions, 338 Poisson equation, 337, 392 root cellar, 388 linearly dependent, 2 Neumann iterates, 122 Poisson kernel, 223 rotating top, 206 linearly independent, 2 Newton's law, 233 pole of order n, 226 rotational matrix, 12 Liouville's theorem, 221 Newton's method, 131 population dynamics, 101, 472 Lipschitz continuity, 136, 164 nodal curves, 361 populations genetics, 526 saddle point, 449 logistic equation, 471 normal equation, 27 positive definite saddle point method, 450 LU decomposition, 30, 32 normal modes, 23 matrices, 18 scalar product, 4 lung tissue, 409 normed vector space, 5 operator, 161, 338 scattering transform, 411 Lyapunov-Schmidt method, 482 norms,5,59 positive matrices, 44 SchrOdinger equation, 169, 313, 411 Euclidean, 5 potatoes, 386 nonlinear, 420 Maple, 48, 50, 453, 455, 492 Frobenius, 41 potential energy, 187-189, 194 SchrOdinger operator, 312 Mathematica, 48 induced, 6 potential function, 228 Schwarz inequality, 6 Mathieu's equation, 502, 552 £2,60 power method, 43 Schwarzschild metric, 182 Matlab, 48 LP,60 precompact, 111 secular terms, 506, 532 maximum modulus theorem, 222 IJ', 59 principal branch, 211 self-adjoint, 13, 153 maximum principle, 17, 115, 117 natural, 6 problem of Procrustes, 41 separable kernel, 112 mean value theorems, 222 Sobolev, 66 projection, 8, 98, 119, 124, 347, 397 separated boundary conditions, 147, Mellin transform, 308, 354 supremum, 5, 60 pseudo-inverse, 28 152 Melnikov function, 494 uniform, 60 pseudo-resolvent operator, 120 separation of variables, 345 method of averaging, 511 null space, 106 Pythagorean theorem, 7 sequence spaces, 59 method of images, 343, 370 sequentially compact, 111 method of lines, 391 operational amplifier (op-amp}, 521 QR decomposition, 33 set of measure zero, 63, 94, 134, 163 Meyer functions, 304 orthogonal complement, 15, 108 quadratic form, 17, 186 shallow water waves, 421 Michaelis-Menten law, 525 orthogonal functions quasi-steady state solution, 523, 525 Shannon sampling function, 78 microwave ovens, 24 associated Legendre, 167 similar matrices, 10 midline strain, 191 Haar, 83 range, 14 similarity solution, 384 minimax principle, 19, 199 Laguerre, 168 range of an operator, 106 similarity transformation, 10 minimum principle, 199 orthogonal matrices, 33 ranking of teams, 44 similarity variable, 384 modified Bessel function, 265, 266 orthogonal polynomials, 9, 69 Rayleigh-Ritz technique, 92, 197 sine function, 78, 270, 302 modified Green's function, 159 Chebyshev, 50, 73, 168, 281 recurrence formulas, 266 sine integral transform, 289 modulus of continuity, 71 Gegenbauer, 73, 168 reflection coefficient, 316 sine-Gordon equation, 419 602 INDEX INDEX 603 singular Sturm-Liouville operator, 153, 160, two component scattering, 419 integral equations, 103 163, 198, 500 two-timing method, 509 matrices, 11 successive over-relaxation, 46 value decomposition, 37 superposition principle, 200 uniform norm, 60 values, 38 support, 137 unitary matrices, 33 singularities, 292, 308 supremum norm, 60 van der Pol equation, 486, 494, 542 essential, 227 swimming protozoan, 473 Van Dyke matching principle, 530 isolated, 226 symbolic function, 140 pole of order n, 226 variational principle, 177 removable, 226 Taylor series, 224 vector space, 1 sink, 230 telegrapher's equation, 366, 373, 406, vectors, 1 smoke rings, 488 461 Volterra integral equation, 101 smoothing algorithm, 397 test function, 137, 339 vortex, 230 Snell's law, 132, 180 Toda lattice, 420 vortex filament, 488 Sobolev inequalities, 66 tomography, 103 vorticity, 342, 488 Sobolev inner product, 66 transcendentally small, 439, 540 Watson's lemma, 446 transfer function, 296 Sobolev norm, 66 wave equation, 189, 337 Sobolev space, 65, 66 transform theory, 9 wavelets, 69, 80 transformations solitary pulse, 548 Battle-Lemarie, 305 bilinear, 237 solitary wave, 421 Daubechies, 82, 307 SOR, 46 hodograph, 245 Haar, 83 source, 230 Householder, 34 hat, 83 span, 3 Joukowski, 242, 311 Meyer, 304 near identity, 512 spanning set, 3 mother, 80 special functions orthogonal, 33 weak convergence, 142 Airy function, 467 similarity, 10 weak formulation, 142, 340 · beta function, 261, 281 unitary, 33 Weierstrass approximation theorem, gamma function, 259, 441 transforms 69 spectral decomposition, 15, 37, 52 discrete Fourier, 76 Whittaker cardinal function, 78, 270, spectral representation, 11, 283, 297 discrete sine, 77, 170 300 spectroscopy, 24 Fourier, 290 Wiener-Paley theorem, 299 spectrum Fourier cosine transform, 290 winding number, 490 continuous, 284, 312 Fourier sine integral, 289 windowed Fourier transform, 300 discrete, 283 Gabor, 301 Wronskian, 147, 175, 263 point, 283 Hanke1,309,352,405 residual, 284 Hilbert, 355 X-ray tomography, 103 splines Laplace, 307, 331 B, 81 Mellin, 308, 354 Young's modulus, 192, 368 cubic, 89 scattering, 411 square well potential, 321 windowed Fourier, 300 z-transform, 311 stagnation points, 231 transmission coefficient, 316 Zeldovich number, 540 stationary phase, 456 transpose, 13 zero sequence, 138 Stirling's formula, 444, 453 trefoil knots, 488 strain matrix, 190 triangle inequality, 5 stream function, 474 trigonometric functions, 69, 73 streamlines,230 . tunnel diode, 485