Applied Mathematical Methods 1, Applied Mathematical Methods 2, Contents I

Preliminary Background

Applied Mathematical Methods Matrices and Linear Transformations

Operational Fundamentals of Linear Algebra Bhaskar Dasgupta Systems of Linear Equations Department of Mechanical Engineering Indian Institute of Technology Kanpur (INDIA) Gauss Elimination Family of Methods [email protected]

Special Systems and Special Methods (Pearson Education 2006, 2007)

Numerical Aspects in Linear Systems May 13, 2008

Applied Mathematical Methods 3, Applied Mathematical Methods 4, Contents II Contents III Eigenvalues and Eigenvectors Topics in Multivariate Calculus

Diagonalization and Similarity Transformations Vector Analysis: Curves and Surfaces

Jacobi and Givens Rotation Methods Scalar and Vector Fields

Householder Transformation and Tridiagonal Matrices Polynomial Equations

QR Decomposition Method Solution of Nonlinear Equations and Systems

Eigenvalue Problem of General Matrices Optimization: Introduction

Singular Value Decomposition Multivariate Optimization

Vector Spaces: Fundamental Concepts* Methods of Nonlinear Optimization* Applied Mathematical Methods 5, Applied Mathematical Methods 6, Contents IV Contents V Constrained Optimization First Order Ordinary Differential Equations

Linear and Quadratic Programming Problems* Second Order Linear Homogeneous ODE’s

Interpolation and Approximation Second Order Linear Non-Homogeneous ODE’s

Basic Methods of Numerical Integration Higher Order Linear ODE’s

Advanced Topics in Numerical Integration* Laplace Transforms

Numerical Solution of Ordinary Differential Equations ODE Systems

ODE Solutions: Advanced Issues Stability of Dynamic Systems

Existence and Uniqueness Theory Series Solutions and Special Functions

Applied Mathematical Methods 7, Applied Mathematical Methods 8, Contents VI Contents VII Variational Calculus* Sturm-Liouville Theory Epilogue Fourier Series and Integrals Selected References Fourier Transforms

Minimax Approximation*

Partial Differential Equations

Analytic Functions

Integrals in the Complex Plane

Singularities of Complex Functions Applied Mathematical Methods Preliminary Background 9, Applied Mathematical Methods Preliminary Background 10, Theme of the Course Theme of the Course Outline Course Contents Theme of the Course Course Contents Sources for More Detailed Study Sources for More Detailed Study Logistic Strategy Logistic Strategy Expected Background To develop a firm mathematical backgroundExpnecessaected Backgroundry for graduate studies and research I a fast-paced recapitulation of UG mathematics Preliminary Background I extension with supplementary advanced ideas for a mature Theme of the Course and forward orientation Course Contents I exposure and highlighting of interconnections Sources for More Detailed Study Logistic Strategy Expected Background To pre-empt needs of the future challenges I trade-off between sufficient and reasonable I target mid-spectrum majority of students

Notable beneficiaries (at two ends) I would-be researchers in analytical/computational areas I students who are till now somewhat afraid of mathematics

Applied Mathematical Methods Preliminary Background 11, Applied Mathematical Methods Preliminary Background 12, Theme of the Course Theme of the Course Course Contents Course Contents Sources for More Detailed Study Course Contents Sources for More Detailed Study Sources for More Detailed Study Logistic Strategy Logistic Strategy Expected Background Expected Background

If you have the time, need and interest, then you may consult I Applied linear algebra I individual books on individual topics; I Multivariate calculus and vector calculus I another “umbrella” volume, like Kreyszig, McQuarrie, O’Neil I Numerical methods or Wylie and Barrett; I Differential equations + + I a good book of numerical analysis or scientific computing, like Acton, Heath, Hildebrand, Krishnamurthy and Sen, Press et I Complex analysis al, Stoer and Bulirsch; I friends, in joint-study groups. Applied Mathematical Methods Preliminary Background 13, Applied Mathematical Methods Preliminary Background 14, Theme of the Course Theme of the Course Logistic Strategy Course Contents Logistic Strategy Course Contents Sources for More Detailed Study Sources for More Detailed Study Logistic Strategy Logistic Strategy Expected Background Expected Background I Study in the given sequence, to the extent possible. I Do not read mathematics. Tutorial Plan I Chapter Selection Tutorial Chapter Selection Tutorial Use lots of pen and paper. 2 2,3 3 26 1,2,4,6 4 Read “mathematics books” and do mathematics. 3 2,4,5,6 4,5 27 1,2,3,4 3,4 4 1,2,4,5,7 4,5 28 2,5,6 6 I Exercises are must. 5 1,4,5 4 29 1,2,5,6 6 6 1,2,4,7 4 30 1,2,3,4,5 4 I Use as many methods as you can think of, certainly including 7 1,2,3,4 2 31 1,2 1(d) 8 1,2,3,4,6 4 32 1,3,5,7 7 the one which is recommended. 9 1,2,4 4 33 1,2,3,7,8 8 I Consult the Appendix after you work out the solution. Follow 10 2,3,4 4 34 1,3,5,6 5 11 2,4,5 5 35 1,3,4 3 the comments, interpretations and suggested extensions. 12 1,3 3 36 1,2,4 4 I Think. Get excited. Discuss. Bore everybody in your known 13 1,2 1 37 1 1(c) 14 2,4,5,6,7 4 38 1,2,3,4,5 5 circles. 15 6,7 7 39 2,3,4,5 4 I Not enough time to attempt all? Want a selection ? 16 2,3,4,8 8 40 1,2,4,5 4 17 1,2,3,6 6 41 1,3,6,8 8 I Program implementation is needed in algorithmic exercises. 18 1,2,3,6,7 3 42 1,3,6 6 19 1,3,4,6 6 43 2,3,4 3 I Master a programming environment. 20 1,2,3 2 44 1,2,4,7,9,10 7,10 I Use mathematical/numerical library/software. 21 1,2,5,7,8 7 45 1,2,3,4,7,9 4,9 22 1,2,3,4,5,6 3,4 46 1,2,5,7 7 Take a MATLAB tutorial session? 23 1,2,3 3 47 1,2,3,5,8,9,10 9,10 24 1,2,3,4,5,6 1 48 1,2,4,5 5 25 1,2,3,4,5 5

Applied Mathematical Methods Preliminary Background 15, Applied Mathematical Methods Preliminary Background 16, Theme of the Course Theme of the Course Expected Background Course Contents Points to note Course Contents Sources for More Detailed Study Sources for More Detailed Study Logistic Strategy Logistic Strategy Expected Background Expected Background

I moderate background of undergraduate mathematics I Put in effort, keep pace. I firm understanding of school mathematics and undergraduate I Stress concept as well as problem-solving. calculus I Follow methods diligently. I Ensure background skills. Take the preliminary test.

Necessary Exercises: Prerequisite problem sets ?? Grade yourself sincerely.

Prerequisite Problem Sets* Applied Mathematical Methods Matrices and Linear Transformations 17, Applied Mathematical Methods Matrices and Linear Transformations 18, Matrices Matrices Outline Geometry and Algebra Matrices Geometry and Algebra Linear Transformations Linear Transformations Matrix Terminology Matrix Terminology Question: What is a “matrix”? Answers: I a rectangular array of numbers/elements ? Matrices and Linear Transformations I a mapping f : M N F , where M = 1, 2, 3, , m , × → { · · · } Matrices N = 1, 2, 3, , n and F is the set of real numbers or { · · · } Geometry and Algebra complex numbers ? Linear Transformations Matrix Terminology Question: What does a matrix do? Explore: With an m n matrix A, ×

y = a x + a x + + a nxn 1 11 1 12 2 · · · 1 y2 = a21x1 + a22x2 + + a2nxn . . . . · · · .  or Ax = y . . . . .   ym = am x + am x + + amnxn 1 1 2 2 · · ·  

Applied Mathematical Methods Matrices and Linear Transformations 19, Applied Mathematical Methods Matrices and Linear Transformations 20, Matrices Matrices Matrices Geometry and Algebra Geometry and Algebra Geometry and Algebra Linear Transformations Linear Transformations Matrix Terminology Matrix Terminology Consider these definitions: T Let vector x = [x1 x2 x3] denote a point (x1, x2, x3) in I y = f (x) 3-dimensional space in frame of reference OX1X2X3. I y = f (x) = f (x , x , , x ) 1 2 n Example: With m = 2 and n = 3, I · · · yk = fk (x) = fk (x1, x2, , xn), k = 1, 2, , m I · · · · · · y = f(x) y1 = a11x1 + a12x2 + a13x3 I . y = Ax y2 = a21x1 + a22x2 + a23x3 Further Answer:  A matrix is the definition of a linear vector function of a Plot y1 and y2 in the OY1Y2 plane. ¤ ¤ vector variable. A: R 3 R 2

Anything deeper? X3 ¢ ¥ ¥ ¥ ¥ ¥ ¥ Y

¡ 2 ¢£ ¥ ¥ ¥ ¥ ¥ ¥ ¦ ¦ ¦ ¦ ¦ ¦

x ¡ y ¥ ¥ ¥ ¥ ¥ ¥ ¦ ¦ ¦ ¦ ¦ ¦ Y1 ¥ ¥ ¥ ¥ ¥ ¥ ¦ ¦ ¦ ¦ ¦ ¦ ¥ ¥ ¥ ¥ ¥ ¥ ¦ ¦ ¦ ¦ ¦ ¦ ¥ ¥ ¥ ¥ ¥ ¥ ¦ ¦ ¦ ¦ ¦ ¦ ¥ ¥ ¥ ¥ ¥ ¥ ¦ ¦ ¦ ¦ ¦ ¦ ¥ ¥ ¥ ¥ ¥ ¥ ¦ ¦ ¦ ¦ ¦ ¦ O X2 O

X1 Caution: Matrices do not define vector functions whose components are Domain Co−domain Figure: Linear transformation: schematic illustration of the form What is matrix A doing? yk = ak + ak x + ak x + + aknxn. 0 1 1 2 2 · · · Applied Mathematical Methods Matrices and Linear Transformations 21, Applied Mathematical Methods Matrices and Linear Transformations 22, Matrices Matrices Geometry and Algebra Geometry and Algebra Linear Transformations Geometry and Algebra Linear Transformations Linear Transformations Matrix Terminology Matrix Terminology 3 Operate A on a large number of points xi R . 3 2 2 ∈ Operating on point x in R , matrix A transforms it to y in R . Obtain corresponding images yi R . ∈ The linear transformation represented by A implies the totality of Point y is the image of point x under the mapping defined by these correspondences. matrix A. 3 We decide to use a different frame of reference OX10 X20X30 for R . [And, possibly OY Y for R2 at the same time.] Note domain R 3, co-domain R 2 with reference to the figure and 10 20 verify that A : R 3 R2 fulfils the requirements of a mapping, by → Coordinates change, i.e. xi changes to xi0 (and possibly yi to yi0). definition. Now, we need a different matrix, say A0, to get back the correspondence as y0 = A0x0. A matrix gives a definition of a linear transformation A matrix: just one description. from one vector space to another.

Question: How to get the new matrix A0?

Applied Mathematical Methods Matrices and Linear Transformations 23, Applied Mathematical Methods Matrices and Linear Transformations 24, Matrices Matrices Matrix Terminology Geometry and Algebra Points to note Geometry and Algebra Linear Transformations Linear Transformations Matrix Terminology Matrix Terminology

I I A matrix defines a linear transformation from one vector space · · · · · · I Matrix product to another. I Transpose I Matrix representation of a linear transformation depends on the selected bases (or frames of reference) of the source and I Conjugate transpose target spaces. I Symmetric and skew-symmetric matrices Important: Revise matrix algebra basics as necessary tools. I Hermitian and skew-Hermitian matrices I Determinant of a square matrix I Inverse of a square matrix Necessary Exercises: 2,3 I Adjoint of a square matrix I · · · · · · Applied Mathematical Methods Operational Fundamentals of Linear Algebra 25, Applied Mathematical Methods Operational Fundamentals of Linear Algebra 26, Range and Null Space: Rank and Nullity Range and Null Space: Rank and Nullity Outline Basis Range and Null Space: Rank and NullitBasisy Change of Basis Change of Basis Elementary Transformations Elementary Transformations Consider A R m n as a mapping ∈ × A : Rn Rm, Ax = y, x R n, y Rm. → ∈ ∈ Operational Fundamentals of Linear Algebra Range and Null Space: Rank and Nullity Observations Basis 1. Every x R n has an image y R m, but every y R m need ∈ ∈ ∈ Change of Basis not have a pre-image in R n. Elementary Transformations Range (or range space) as subset/subspace of co-domain: containing images of all x R n. ∈ 2. Image of x R n in Rm is unique, but pre-image of y R m ∈ ∈ need not be. It may be non-existent, unique or infinitely many. Null space as subset/subspace of domain: containing pre-images of only 0 R m. ∈

Applied Mathematical Methods Operational Fundamentals of Linear Algebra 27, Applied Mathematical Methods Operational Fundamentals of Linear Algebra 28, Range and Null Space: Rank and Nullity Range and Null Space: Rank and Nullity Range and Null Space: Rank and NullitBasisy Basis Basis Change of Basis Change of Basis Elementary Transformations Elementary Transformations n m R R 0 Take a set of vectors v1, v2, , vr in a vector space. O A · · · Range ( A ) Question: Given a vector v in the vector space, can we describe it

Null ( A ) as

Domain Co−domain v = k v + k v + + kr vr = Vk, 1 1 2 2 · · · T where V = [v v vr ] and k = [k k kr ] ? Figure: Range and null space: schematic representation 1 2 · · · 1 2 · · · Answer: Not necessarily. Question: What is the dimension of a vector space? Span, denoted as < v , v , , v >: the subspace Linear dependence and independence: Vectors x , x , , xr 1 2 r 1 2 · · · · · · in a vector space are called linearly independent if described/generated by a set of vectors.

k x + k x + + kr xr = 0 k = k = = kr = 0. 1 1 2 2 · · · ⇒ 1 2 · · · Basis: A basis of a vector space is composed of an ordered minimal set of vectors spanning the entire space. Range(A) = y : y = Ax, x R n { ∈ } Null(A) = x : x R n, Ax = 0 The basis for an n-dimensional space will have exactly n { ∈ } Rank(A) = dim Range(A) members, all linearly independent. Nullity(A) = dim Null(A) Applied Mathematical Methods Operational Fundamentals of Linear Algebra 29, Applied Mathematical Methods Operational Fundamentals of Linear Algebra 30, Range and Null Space: Rank and Nullity Range and Null Space: Rank and Nullity Basis Basis Change of Basis Basis Change of Basis Change of Basis Orthogonal basis: v , v , , vn with Elementary Transformations nElementary Transformations { 1 2 · · · } Suppose x represents a vector (point) in R in some basis. T v v = 0 j = k. Question: If we change over to a new basis c1, c2, , cn , how j k { · · · } ∀ 6 does the representation of a vector change? Orthonormal basis:

T 0 if j = k vj vk = δjk = 6 1 if j = k x = x¯ c + x¯ c + + x¯ncn  1 1 2 2 · · · Members of an orthonormal basis form an orthogonal matrix. x¯1 x¯ Properties of an orthogonal matrix:  2  = [c1 c2 cn] . . V 1 = VT or VVT = I, and · · · . −    x¯n  det V = +1 or 1,   −   Natural basis: With C = [c c cn], 1 2 · · · 1 0 0 new to old coordinates: Cx¯ = x and 0 1 0 old to new coordinates: x¯ = C 1x.       − e1 = 0 , e2 = 0 , , en = 0 .  .   .  · · ·  .  Note: Matrix C is invertible. How?  .   .   .        Special case with C orthogonal:  0   0   1        orthogonal coordinate transformation.      

Applied Mathematical Methods Operational Fundamentals of Linear Algebra 31, Applied Mathematical Methods Operational Fundamentals of Linear Algebra 32, Range and Null Space: Rank and Nullity Range and Null Space: Rank and Nullity Change of Basis Basis Elementary Transformations Basis Change of Basis Change of Basis Elementary Transformations Elementary Transformations Question: And, how does basis change affect the representation of a linear transformation? Observation: Certain reorganizations of equations in a system Consider the mapping A : R n Rm, Ax = y. have no effect on the solution(s). → Change the basis of the domain through P R n n and that of the ∈ × co-domain through Q R m m. Elementary Row Transformations: ∈ × New and old vector representations are related as 1. interchange of two rows, 2. scaling of a row, and Px¯ = x and Qy¯ = y. 3. addition of a scalar multiple of a row to another. Then, Ax = y A¯ x¯ = y¯, with ⇒ ¯ 1 A = Q− AP Elementary Column Transformations: Similar operations with Special case: m = n and P = Q gives a similarity transformation columns, equivalent to a corresponding shuffling of the variables (unknowns). 1 A¯ = P− AP Applied Mathematical Methods Operational Fundamentals of Linear Algebra 33, Applied Mathematical Methods Operational Fundamentals of Linear Algebra 34, Range and Null Space: Rank and Nullity Range and Null Space: Rank and Nullity Elementary Transformations Basis Points to note Basis Change of Basis Change of Basis Elementary Transformations Elementary Transformations Equivalence of matrices: An elementary transformation defines an equivalence relation between two matrices. Reduction to normal form: I Concepts of range and null space of a linear transformation. I I 0 Effects of change of basis on representations of vectors and A = r N 0 0 linear transformations.   I Elementary transformations as tools to modify (simplify) Rank invariance: Elementary transformations do not alter the systems of (simultaneous) linear equations. rank of a matrix. Elementary transformation as matrix multiplication: an elementary row transformation on a matrix is Necessary Exercises: 2,4,5,6 equivalent to a pre-multiplication with an elementary matrix, obtained through the same row transformation on the identity matrix (of appropriate size).

Similarly, an elementary column transformation is equivalent to post-multiplication with the corresponding elementary matrix.

Applied Mathematical Methods Systems of Linear Equations 35, Applied Mathematical Methods Systems of Linear Equations 36, Nature of Solutions Nature of Solutions Outline Basic Idea of Solution Methodology Nature of Solutions Basic Idea of Solution Methodology Homogeneous Systems Homogeneous Systems Pivoting Pivoting Partitioning and Block Operations Ax = b Partitioning and Block Operations Coefficient matrix: A, augmented matrix: [A b]. | Existence of solutions or consistency: Systems of Linear Equations Ax = b has a solution Nature of Solutions b Range(A) Basic Idea of Solution Methodology ⇔ ∈ Homogeneous Systems Rank(A) = Rank([A b]) ⇔ | Pivoting Partitioning and Block Operations Uniqueness of solutions: Rank(A) = Rank([A b]) = n | Solution of Ax = b is unique. ⇔ Ax = 0 has only the trivial (zero) solution. ⇔ Infinite solutions: For Rank(A) = Rank([A b]) = k < n, solution |

x = x¯ + xN , with Ax¯ = b and xN Null(A) ∈ Applied Mathematical Methods Systems of Linear Equations 37, Applied Mathematical Methods Systems of Linear Equations 38, Nature of Solutions Nature of Solutions Basic Idea of Solution Methodology Basic Idea of Solution Methodology Homogeneous Systems Basic Idea of Solution Methodology Homogeneous Systems Homogeneous Systems Pivoting Pivoting Partitioning and Block Operations To solve Ax = 0 or to describe Null(A), Partitioning and Block Operations To diagnose the non-existence of a solution, apply a series of elementary row transformations on A to reduce it To determine the unique solution, or to the A∼, To describe infinite solutions; the row-reduced echelon form or RREF. decouple the equations using elementary transformations. Features of RREF: 1. The first non-zero entry in any row is a ‘1’, the leading ‘1’. For solving Ax = b, apply suitable elementary row transformations 2. In the same column as the leading ‘1’, other entries are zero. on both sides, leading to 3. Non-zero entries in a lower row appear later. RqRq 1 R2R1Ax = RqRq 1 R2R1b, − · · · − · · · Variables corresponding to columns having leading ‘1’s or, [RA]x = Rb; are expressed in terms of the remaining variables.

such that matrix [RA] is greatly simplified. u1 In the best case, with complete reduction, RA = In, and u2 Solution of Ax = 0: x = z1 z2 zn k   components of x can be read off from Rb. · · · − · · · 1    un k  For inverting matrix A, treat AA− = In similarly.  −  Basis of Null(A): z1, z2, , zn k   { · · · − }

Applied Mathematical Methods Systems of Linear Equations 39, Applied Mathematical Methods Systems of Linear Equations 40, Nature of Solutions Nature of Solutions Pivoting Basic Idea of Solution Methodology Partitioning and Block Operations Basic Idea of Solution Methodology Homogeneous Systems Homogeneous Systems Pivoting Pivoting Attempt: Partitioning and Block Operations Partitioning and Block Operations To get ‘1’ at diagonal (or leading) position, with ‘0’ elsewhere. Equation Ax = y can be written as Key step: division by the diagonal (or leading) entry.

Consider x1 I . . . . . A11 A12 A13 y1 k x2 = , . δ . . . . A21 A22 A23   y2   x3    . . . . BIG .  A¯ = .    . big . . . .  with x , x etc being themselves vectors (or matrices).   1 2  ......    I For a valid partitioning, block sizes should be consistent.  ......      I Elementary transformations can be applied over blocks. Cannot divide by zero. Should not divide by δ. I Block operations can be computationally economical at times. I partial pivoting: row interchange to get ‘big’ in place of δ I Conceptually, different blocks of contributions/equations can I complete pivoting: row and column interchanges to get be assembled for mathematical modelling of complicated ‘BIG’ in place of δ coupled systems.

Complete pivoting does not give a huge advantage over partial pivoting, but requires maintaining of variable permutation for later unscrambling. Applied Mathematical Methods Systems of Linear Equations 41, Applied Mathematical Methods Gauss Elimination Family of Methods 42, Nature of Solutions Gauss-Jordan Elimination Points to note Basic Idea of Solution Methodology Outline Gaussian Elimination with Back-Substitution Homogeneous Systems LU Decomposition Pivoting Partitioning and Block Operations

I Solution(s) of Ax = b may be non-existent, unique or infinitely many. I Complete solution can be described by composing a particular Gauss Elimination Family of Methods solution with the null space of A. Gauss-Jordan Elimination I Null space basis can be obtained conveniently from the Gaussian Elimination with Back-Substitution row-reduced echelon form of A. LU Decomposition I For a strategy of solution, pivoting is an important step.

Necessary Exercises: 1,2,4,5,7

Applied Mathematical Methods Gauss Elimination Family of Methods 43, Applied Mathematical Methods Gauss Elimination Family of Methods 44, Gauss-Jordan Elimination Gauss-Jordan Elimination Gauss-Jordan Elimination Gaussian Elimination with Back-Substitution Gauss-Jordan Elimination Gaussian Elimination with Back-Substitution LU Decomposition LU Decomposition 1 Task: Solve Ax = b1, Ax = b2 and Ax = b3; find A− and evaluate A 1B, where A R n n and B Rn p. Gauss-Jordan Algorithm − ∈ × ∈ × n (2n+3+p) I Assemble C = [A b b b In B] R ∆ = 1 1 2 3 ∈ × and follow the algorithm . I For k = 1, 2, 3, , (n 1) · · · − 1. Pivot : identify l such that clk = max cjk for k j n. Collect solutions from the result | | | | ≤ ≤ If clk = 0, then ∆ = 0 and exit. ∼ 1 1 1 1 1 Else, interchange row k and row l. C C = [In A− b A− b A− b A− A− B]. 1 2 3 2. ∆ ckk ∆, −→ ←− Divide row k by ckk . Remarks: 3. Subtract cjk times row k from row j, j = k. ∀ 6 I Premature termination: matrix A singular — decision? I ∆ cnn∆ ←− I If you use complete pivoting, unscramble permutation. If cnn = 0, then exit. Else, divide row n by cnn. I ∼ 1 Identity matrix in both C and C? Store A− ‘in place’. I For evaluating A 1b, do not develop A 1. − − In case of non-singular A, default termination . I Gauss-Jordan elimination an overkill? Want something cheaper ? This outline is for partial pivoting. Applied Mathematical Methods Gauss Elimination Family of Methods 45, Applied Mathematical Methods Gauss Elimination Family of Methods 46, Gauss-Jordan Elimination Gauss-Jordan Elimination Gaussian Elimination with Back-SubstitutionGaussian Elimination with Back-Substitution Gaussian Elimination with Back-SubstitutionGaussian Elimination with Back-Substitution LU Decomposition LU Decomposition Gaussian elimination: Anatomy of the Gaussian elimination: The process of Gaussian elimination (with no pivoting) leads to Ax = b ∼ ∼ U = RqRq 1 R2R1A = RA. Ax = b − · · · −→ a a a x b The steps given by 110 120 · · · 10 n 1 10 a0 a0 n x2 b0 for k = 1, 2, 3, , (n 1)  22 · · · 2     2  or, . . . = . · · · − a .. . . . j-th row j-th row jk k-th row for       ←− − akk ×  ann0   xn   bn0  j = k + 1, k + 2, , n       · · · Back-substitutions:      involve elementary matrices xn = b0 /a0 , n nn 1 0 0 0 n a21 · · · 1 a 1 0 0 xi = b0 a0 xj for i = n 1, n 2, , 2, 1  − a11 · · ·  a  i − ij  − − · · · 31 0 1 0 ii0 j=i+1 Rk k=1 = − a11 · · · etc. X |  . . . . .     ......  Remarks    an1 0 0 1  I Computational cost half compared to G-J elimination.  − a11 · · ·  I 1   Like G-J elimination, prior knowledge of RHS needed. With L = R− , A = LU.

Applied Mathematical Methods Gauss Elimination Family of Methods 47, Applied Mathematical Methods Gauss Elimination Family of Methods 48, Gauss-Jordan Elimination Gauss-Jordan Elimination LU Decomposition Gaussian Elimination with Back-Substitution LU Decomposition Gaussian Elimination with Back-Substitution LU Decomposition LU Decomposition A square matrix with non-zero leading minors is LU-decomposable. Question: How to LU-decompose a given matrix? No reference to a right-hand-side (RHS) vector! To solve Ax = b, denote y = Ux and split as l 0 0 0 u u u u n 11 · · · 11 12 13 · · · 1 l21 l22 0 0 0 u22 u23 u2n  · · ·   · · ·  Ax = b LUx = b l l l 0 0 0 u u n ⇒ L = 31 32 33 · · · and U = 33 · · · 3 Ly = b and Ux = y.  . . . . .   . . . . .   ......   ......  ⇒      ln ln ln lnn   0 0 0 unn   1 2 3 · · ·   · · ·  Forward substitutions:     i 1 Elements of the product give 1 − yi = bi lij yj for i = 1, 2, 3, , n; lii  −  · · · i j=1 X lik ukj = aij for i j,   ≤ k Back-substitutions: X=1 j n 1 and lik ukj = aij for i > j. xi = yi uij xj for i = n, n 1, n 2, , 1. k=1 uii  −  − − · · · X j=i+1 X 2 2   n equations in n + n unknowns: choice of n unknowns Applied Mathematical Methods Gauss Elimination Family of Methods 49, Applied Mathematical Methods Gauss Elimination Family of Methods 50, Gauss-Jordan Elimination Gauss-Jordan Elimination LU Decomposition Gaussian Elimination with Back-Substitution LU Decomposition Gaussian Elimination with Back-Substitution LU Decomposition LU Decomposition Question: What about matrices which are not LU-decomposable? Doolittle’s algorithm Question: What about pivoting?

I Choose lii = 1 Consider the non-singular matrix I For j = 1, 2, 3, , n · · ·i 1 0 1 2 1 0 0 u11 = 0 u12 u13 1. uij = aij k−=1 lik ukj for 1 i j 1 − j 1 ≤ ≤ 3 1 2 = l21 =? 1 0 0 u22 u23 . 2. lij = (aij − lik ukj ) for i > j ujj k=1       P− 2 1 3 l31 l32 1 0 0 u33 P       Evaluation proceeds in column order of the matrix (for storage) LU-decompose a permutation of its rows

u11 u12 u13 u1n 0 1 2 0 1 0 3 1 2 · · · 3 1 2 = 1 0 0 0 1 2 l21 u22 u23 u2n  · · ·   2 1 3   0 0 1   2 1 3  l31 l32 u33 u3n A∗ = · · ·        ......  0 1 0 1 0 0 3 1 2  . . . . .  = 1 0 0 0 1 0 0 1 2 .          ln1 ln2 ln3 unn  0 0 1 2 1 1 0 0 1  · · ·  3 3         In this PLU decomposition, permutation P is recorded in a vector.

Applied Mathematical Methods Gauss Elimination Family of Methods 51, Applied Mathematical Methods Special Systems and Special Methods 52, Gauss-Jordan Elimination Quadratic Forms, Symmetry and Positive Definiteness Points to note Gaussian Elimination with Back-Substitution Outline Cholesky Decomposition LU Decomposition Sparse Systems*

For invertible coefficient matrices, use I Gauss-Jordan elimination for large number of RHS vectors available all together and also for matrix inversion, Special Systems and Special Methods I Gaussian elimination with back-substitution for small number Quadratic Forms, Symmetry and Positive Definiteness of RHS vectors available together, Cholesky Decomposition I LU decomposition method to develop and maintain factors to Sparse Systems* be used as and when RHS vectors are available. Pivoting is almost necessary (without further special structure).

Necessary Exercises: 1,4,5 Applied Mathematical Methods Special Systems and Special Methods 53, Applied Mathematical Methods Special Systems and Special Methods 54, Quadratic Forms, Symmetry and Positive Definiteness Quadratic Forms, Symmetry and Positive Definiteness Quadratic Forms, Symmetry and PositiveCholesky DecompDefinitenessosition Cholesky Decomposition Cholesky Decomposition Sparse Systems* Sparse Systems* n n Quadratic form If A R × is symmetric and positive definite, then there exists a ∈ n n n n non-singular lower triangular matrix L R × such that T ∈ q(x) = x Ax = aij xi xj T i j A = LL . X=1 X=1 defined with respect to a symmetric matrix. Algorithm For i = 1, 2, 3, , n · · · i 1 2 Quadratic form q(x), equivalently matrix A, is called positive I Lii = aii − L − k=1 ik definite (p.d.) when q1 i 1 I Lji = aji P − L L for i < j n Lii k=1 jk ik xT Ax > 0 x = 0 − ≤ ∀ 6 For solving Ax = b,P  and positive semi-definite (p.s.d.) when Forward substitutions: Ly = b xT Ax 0 x = 0. Back-substitutions: LT x = y ≥ ∀ 6 Sylvester’s criteria: Remarks I Test of positive definiteness. a11 a12 a11 0, 0, , det A 0; I ≥ a21 a22 ≥ · · · ≥ Stable algorithm: no pivoting necessary!

I i.e. all leading minors non-negative, for p.s.d. Economy of space and time.

Applied Mathematical Methods Special Systems and Special Methods 55, Applied Mathematical Methods Special Systems and Special Methods 56, Quadratic Forms, Symmetry and Positive Definiteness Quadratic Forms, Symmetry and Positive Definiteness Sparse Systems* Cholesky Decomposition Points to note Cholesky Decomposition Sparse Systems* Sparse Systems*

I What is a sparse matrix? I Concepts and criteria of positive definiteness and positive I Bandedness and bandwidth semi-definiteness I Efficient storage and processing I Cholesky decomposition method in symmetric positive definite I Updates systems I Sherman-Morrison formula I Nature of sparsity and its exploitation

1 T 1 T 1 1 (A− u)(v A− ) (A + uv )− = A− T 1 − 1 + v A− u

I Woodbury formula Necessary Exercises: 1,2,4,7 I Conjugate gradient method I efficiently implemented matrix-vector products Applied Mathematical Methods Numerical Aspects in Linear Systems 57, Applied Mathematical Methods Numerical Aspects in Linear Systems 58, Norms and Condition Numbers Norms and Condition Numbers Outline Ill-conditioning and Sensitivity Norms and Condition Numbers Ill-conditioning and Sensitivity Rectangular Systems Rectangular Systems Singularity-Robust Solutions Singularity-Robust Solutions Iterative Methods Norm of a vector: a measure of size Iterative Methods I Euclidean norm or 2-norm 1 x = x = x2 + x2 + + x2 2 = xT x k k k k2 1 2 · · · n Numerical Aspects in Linear Systems   q Norms and Condition Numbers I The p-norm Ill-conditioning and Sensitivity p p p 1 x p = [ x + x + + xn ] p Rectangular Systems k k | 1| | 2| · · · | | Singularity-Robust Solutions

Iterative Methods I The 1-norm: x = x + x + + xn k k1 | 1| | 2| · · · | | I The -norm: ∞ p p p 1 x = lim [ x1 + x2 + + xn ] p = max xj ∞ p j k k →∞ | | | | · · · | | | |

I Weighted norm x = xT Wx k kw where weight matrix W is symmetricq and positive definite.

Applied Mathematical Methods Numerical Aspects in Linear Systems 59, Applied Mathematical Methods Numerical Aspects in Linear Systems 60, Norms and Condition Numbers Norms and Condition Numbers Norms and Condition Numbers Ill-conditioning and Sensitivity Ill-conditioning and Sensitivity Ill-conditioning and Sensitivity Rectangular Systems Rectangular Systems Singularity-Robust Solutions Singularity-Robust Solutions Iterative Methods Iterative Methods Norm of a matrix: magnitude or scale of the transformation 0.9999x 1.0001x = 1 1 − 2 x1 x2 = 1 +  Matrix norm (induced by a vector norm) is given by the largest − Solution: x = 10001+1 , x = 9999 1 magnification it can produce on a vector 1 2 2 2 − I sensitive to small changes in the RHS Ax I A = max k k = max Ax insensitive to error in a guess See illustration k k x x x =1 k k 1 k k k k For the system Ax = b, solution is x = A− b and 1 1 δx = A− δb A− δA x Direct consequence: Ax A x − k k ≤ k k k k If the matrix A is exactly known, then Index of closeness to singularity: Condition number δx 1 δb δb k k A A− k k = κ(A)k k 1 x ≤ k k k k b b κ(A) = A A− , 1 κ(A) k k k k ≤ ≤ ∞ k k k k k k If the RHS is known exactly, then

δx 1 δA δA ** Isotropic, well-conditioned, ill-conditioned and singular matrices k k A A− k k = κ(A)k k x ≤ k k k k A A k k k k k k Applied Mathematical Methods Numerical Aspects in Linear Systems 61, Applied Mathematical Methods Numerical Aspects in Linear Systems 62, Norms and Condition Numbers Norms and Condition Numbers Ill-conditioning and Sensitivity Ill-conditioning and Sensitivity Rectangular Systems Ill-conditioning and Sensitivity Rectangular Systems Rectangular Systems

X Singularity-Robust Solutions Singularity-Robust Solutions 2 X2 m n (2) Iterative Methods Iterative Methods (2) (2b) Consider Ax = b with A R × and Rank(A) = n < m. (1) (1) ∈

(b) T T T 1 T X A Ax = A b x = (A A)− A b ⇒ o X 1 o X1 (a) (a) X X Square of error norm

1 1 T (a) Reference system (b) Parallel shift U(x) = Ax b 2 = (Ax b) (Ax b) 2k − k 2 − − X 2 X2 (2) 1 T T T T 1 T (2) (2d) = x A Ax x A b + b b (1) (1) 2 − 2 Least square error solution: (c) o X X1 o X1 ∂U = AT Ax AT b = 0 ∂x − (c) Guess validation (d) Singularity Pseudoinverse or Moore-Penrose inverse or left-inverse Figure: Ill-conditioning: a geometric perspective # T 1 T A = (A A)− A

Applied Mathematical Methods Numerical Aspects in Linear Systems 63, Applied Mathematical Methods Numerical Aspects in Linear Systems 64, Norms and Condition Numbers Norms and Condition Numbers Rectangular Systems Ill-conditioning and Sensitivity Singularity-Robust Solutions Ill-conditioning and Sensitivity Rectangular Systems Rectangular Systems m n Singularity-Robust Solutions Singularity-Robust Solutions Consider Ax = b with A R and RankIterative(A) =Methomds< n. Iterative Methods ∈ × Ill-posed problems: Tikhonov regularization Look for λ R m that satisfies AT λ = x and ∈ I recipe for any linear system (m > n, m = n or m < n), with T AA λ = b any condition! Solution Ax = b may have conflict: form AT Ax = AT b. T T T 1 x = A λ = A (AA )− b AT A may be ill-conditioned: rig the system as Consider the problem T 2 T 1 T (A A + ν In)x = A b minimize U(x) = 2 x x subject to Ax = b. Extremum of the Lagrangian (x, λ) = 1 xT x λT (Ax b) is Coefficient matrix: symmetric and positive definite! L 2 − − given by The idea: Immunize the system, paying a small price. ∂ ∂ Issues: L = 0, L = 0 x = AT λ, Ax = b. ∂x ∂λ ⇒ I The choice of ν? T T 1 Solution x = A (AA )− b gives foot of the perpendicular on the I When m < n, computational advantage by solution ‘plane’ and the pseudoinverse T 2 T # T T 1 (AA + ν Im)λ = b, x = A λ A = A (AA )− here is a right-inverse! Applied Mathematical Methods Numerical Aspects in Linear Systems 65, Applied Mathematical Methods Numerical Aspects in Linear Systems 66, Norms and Condition Numbers Norms and Condition Numbers Iterative Methods Ill-conditioning and Sensitivity Points to note Ill-conditioning and Sensitivity Rectangular Systems Rectangular Systems Singularity-Robust Solutions Singularity-Robust Solutions Iterative Methods Iterative Methods Jacobi’s iteration method:

n I (k+1) 1 (k) Solutions are unreliable when the coefficient matrix is xi = bi aij xj for i = 1, 2, 3, , n. aii  −  · · · ill-conditioned. j=1, j=i X6 I Finding pseudoinverse of a full-rank matrix is ‘easy’.   I Tikhonov regularization provides singularity-robust solutions. Gauss-Seidel method: I Iterative methods may have an edge in certain situations! i 1 n (k+1) 1 − (k+1) (k) xi = bi aij xj aij xj for i = 1, 2, 3, , n. aii  − −  · · · j j i X=1 =X+1   Necessary Exercises: 1,2,3,4 The category of relaxation methods: diagonal dominance and availability of good initial approximations

Applied Mathematical Methods Eigenvalues and Eigenvectors 67, Applied Mathematical Methods Eigenvalues and Eigenvectors 68, Eigenvalue Problem Eigenvalue Problem Outline Generalized Eigenvalue Problem Eigenvalue Problem Generalized Eigenvalue Problem Some Basic Theoretical Results Some Basic Theoretical Results Power Method In mapping A : R n Rn, special vectors ofPomatrixwer MethodA R n n → ∈ × I mapped to scalar multiples, i.e. undergo pure scaling

Av = λv Eigenvalues and Eigenvectors Eigenvalue Problem Eigenvector (v) and eigenvalue (λ): eigenpair (λ, v) Generalized Eigenvalue Problem algebraic eigenvalue problem Some Basic Theoretical Results Power Method (λI A)v = 0 − For non-trivial (non-zero) solution v,

det(λI A) = 0 − Characteristic equation: characteristic polynomial: n roots I n eigenvalues — for each, find eignevector(s) Multiplicity of an eigenvalue: algebraic and geometric Multiplicity mismatch: diagonalizable and defective matrices Applied Mathematical Methods Eigenvalues and Eigenvectors 69, Applied Mathematical Methods Eigenvalues and Eigenvectors 70, Eigenvalue Problem Eigenvalue Problem Generalized Eigenvalue Problem Generalized Eigenvalue Problem Some Basic Theoretical Results Generalized Eigenvalue Problem Some Basic Theoretical Results Some Basic Theoretical Results 1-dof mass-spring system: mx¨ + kx = 0 Power Method Power Method Eigenvalues of transpose k T Natural frequency of vibration: ωn = m Eigenvalues of A are the same as those of A. q Free vibration of n-dof system: Caution: Eigenvectors of A and AT need not be same.

Mx¨ + Kx = 0, Diagonal and block diagonal matrices Natural frequencies and corresponding modes? Eigenvalues of a diagonal matrix are its diagonal entries. Assuming a vibration mode x = Φ sin(ωt + α), Corresponding eigenvectors: natural basis members (e1, e2 etc).

( ω2MΦ + KΦ) sin(ωt + α) = 0 KΦ = ω2MΦ Eigenvalues of a block diagonal matrix: those of diagonal blocks. − ⇒ Eigenvectors: coordinate extensions of individual eigenvectors. 1 2 Reduce as M− K Φ = ω Φ? Why is it not a good idea? With (λ2, v2) as eigenpair of block A2,
K symmetric, M symmetric and positive definite!! A1 0 0 0 0 0 Av∼2 = 0 A2 0 v2 = A2v2 = λ2 v2 With M = LLT , Φ∼ = LT Φ and K∼ = L 1KL T ,         − − 0 0 A3 0 0 0         K∼ Φ∼ = ω2Φ∼

Applied Mathematical Methods Eigenvalues and Eigenvectors 71, Applied Mathematical Methods Eigenvalues and Eigenvectors 72, Eigenvalue Problem Eigenvalue Problem Some Basic Theoretical Results Generalized Eigenvalue Problem Some Basic Theoretical Results Generalized Eigenvalue Problem Some Basic Theoretical Results Some Basic Theoretical Results Power Method Power Method Triangular and block triangular matrices Eigenvalues of a triangular matrix are its diagonal entries. Shift theorem Eigenvalues of a block triangular matrix are the collection of Eigenvectors of A + µI are the same as those of A. eigenvalues of its diagonal blocks. Eigenvalues: shifted by µ. Take A B r r s s H = , A R × and C R × Deflation 0 C ∈ ∈   For a symmetric matrix A, with mutually orthogonal eigenvectors, If Av = λv, then having (λj , vj ) as an eigenpair,

v A B v Av λv v T H = = = = λ vj vj 0 0 C 0 0 0 0 B = A λj T             − vj vj If µ is an eigenvalue of C, then it is also an eigenvalue of CT and has the same eigenstructure as A, except that the eigenvalue 0 AT 0 0 0 corresponding to vj is zero. CT w = µw HT = = µ ⇒ w BT CT w w         Applied Mathematical Methods Eigenvalues and Eigenvectors 73, Applied Mathematical Methods Eigenvalues and Eigenvectors 74, Eigenvalue Problem Eigenvalue Problem Some Basic Theoretical Results Generalized Eigenvalue Problem Power Method Generalized Eigenvalue Problem Some Basic Theoretical Results Some Basic Theoretical Results Power Method Power Method Consider matrix A with Eigenspace If v1, v2, , vk are eigenvectors of A corresponding to the same · · · λ1 > λ2 λ3 λn 1 > λn eigenvalue λ, then | | | | ≥ | | ≥ · · · ≥ | − | | |

eigenspace: < v , v , , vk > and a full set of n eigenvectors v1, v2, , vn. 1 2 · · · · · · For vector x = α v + α v + + αnvn, 1 1 2 2 · · · Similarity transformation p p p 1 p p λ2 λ3 λn B = S− AS: same transformation expressed in new basis. A x = λ1 α1v1 + α2v2 + α3v3 + + αnvn λ1 λ1 · · · λ1 1         det(λI A) = det S− det(λI A) det S = det(λI B) − − − p p As p , A x λ1α1v1, and Same characteristic polynomial! → ∞ → p (A x)r Eigenvalues are the property of a linear transformation, λ1 = lim , r = 1, 2, 3, , n. p p 1 not of the basis. →∞ (A − x)r · · · 1 An eigenvector v of A transforms to S− v, as the corresponding At convergence, n ratios will be the same. eigenvector of B. Question: How to find the least magnitude eigenvalue?

Applied Mathematical Methods Eigenvalues and Eigenvectors 75, Applied Mathematical Methods Diagonalization and Similarity Transformations 76, Eigenvalue Problem Diagonalizability Points to note Generalized Eigenvalue Problem Outline Canonical Forms Some Basic Theoretical Results Symmetric Matrices Power Method Similarity Transformations

I Meaning and context of the algebraic eigenvalue problem I Fundamental deductions and vital relationships Diagonalization and Similarity Transformations I Power method as an inexpensive procedure to determine Diagonalizability extremal magnitude eigenvalues Canonical Forms Symmetric Matrices Similarity Transformations

Necessary Exercises: 1,2,3,4,6 Applied Mathematical Methods Diagonalization and Similarity Transformations 77, Applied Mathematical Methods Diagonalization and Similarity Transformations 78, Diagonalizability Diagonalizability Diagonalizability Canonical Forms Diagonalizability Canonical Forms Symmetric Matrices Symmetric Matrices Similarity Transformations Similarity Transformations n n Consider A R , having n eigenvectors v , v , , vn; ∈ × 1 2 · · · with corresponding eigenvalues λ , λ , , λn. 1 2 · · · Diagonalizability: A matrix having a complete set of n linearly independent eigenvectors is diagonalizable. AS = A[v1 v2 vn] = [λ1v1 λ2v2 λnvn] · · · · · · Existence of a complete set of eigenvectors: λ1 0 0 · · · A diagonalizable matrix possesses a complete set of n 0 λ2 0 = [v1 v2 vn]  . . ·.· · .  = SΛ linearly independent eigenvectors. · · · . . .. .    0 0 λn   · · ·  I All distinct eigenvalues implies diagonalizability.   I 1 1 But, diagonalizability does not imply distinct eigenvalues! A = SΛS− and S− AS = Λ ⇒ I However, a lack of diagonalizability certainly implies a multiplicity mismatch. Diagonalization: The process of changing the basis of a linear transformation so that its new matrix representation is diagonal, i.e. so that it is decoupled among its coordinates.

Applied Mathematical Methods Diagonalization and Similarity Transformations 79, Applied Mathematical Methods Diagonalization and Similarity Transformations 80, Diagonalizability Diagonalizability Canonical Forms Canonical Forms Canonical Forms Canonical Forms Symmetric Matrices Symmetric Matrices Similarity Transformations Jordan canonical form (JCF): composed ofSimilaJorityrdanTransfoblormationscks λ 1 J1 λ 1 Jordan canonical form (JCF)   J2 . J =   , J = λ .. Diagonal (canonical) form .. r   .  .     .. 1   Jk    Triangular (canonical) form    λ      The key equation AS = SJ in extendedform gives  . .. Other convenient forms A[ Sr ] = [ Sr ]  Jr  , Tridiagonal form · · · · · · · · · · · · .  ..  Hessenberg form     where Jordan block Jr is associated with the subspace of

Sr = [v w w ] 2 3 · · · Applied Mathematical Methods Diagonalization and Similarity Transformations 81, Applied Mathematical Methods Diagonalization and Similarity Transformations 82, Diagonalizability Diagonalizability Canonical Forms Canonical Forms Canonical Forms Canonical Forms Symmetric Matrices Symmetric Matrices Equating blocks as ASr = Sr Jr gives Similarity Transformations Similarity Transformations

λ 1 λ 1 Diagonal form   [Av Aw2 Aw3 ] = [v w2 w3 ] . I Special case of Jordan form, with each Jordan block of 1 1 · · · · · · λ ..   ×  .  size  ..    I Matrix is diagonalizable   Columnwise equality leads to I Similarity transformation matrix S is composed of n linearly independent eigenvectors as columns Av = λv, Aw = v + λw , Aw = w + λw , 2 2 3 2 3 · · · I None of the eigenvectors admits any generalized eigenvector I Generalized eigenvectors w2, w3 etc: Equal geometric and algebraic multiplicities for every eigenvalue (A λI)v = 0, − (A λI)w = v and (A λI)2w = 0, − 2 − 2 (A λI)w = w and (A λI)3w = 0, − 3 2 − 3 · · ·

Applied Mathematical Methods Diagonalization and Similarity Transformations 83, Applied Mathematical Methods Diagonalization and Similarity Transformations 84, Diagonalizability Diagonalizability Canonical Forms Canonical Forms Canonical Forms Canonical Forms Symmetric Matrices Symmetric Matrices Similarity Transformations Similarity Transformations Triangular form Forms that can be obtained with pre-determined number of Triangularization: Change of basis of a linear tranformation so as arithmetic operations (without iteration): to get its matrix in the triangular form Tridiagonal form: non-zero entries only in the (leading) diagonal, sub-diagonal and super-diagonal I For real eigenvalues, always possible to accomplish with I useful for symmetric matrices orthogonal similarity transformation Hessenberg form: A slight generalization of a triangular matrix I Always possible to accomplish with unitary similarity transformation, with complex arithmetic ∗ ∗ ∗ · · · ∗ ∗ I Determination of eigenvalues  ∗ ∗ ∗ · · · ∗ ∗  ∗ ∗ · · · ∗ ∗ Hu =  . . . .   ......  Note: The case of complex eigenvalues: 2 2 real diagonal block   ×  . . .   .. .. .  α β α + iβ 0   −   β α ∼ 0 α iβ  ∗ ∗     −    Note: Tridiagonal and Hessenberg forms do not fall in the category of canonical forms. Applied Mathematical Methods Diagonalization and Similarity Transformations 85, Applied Mathematical Methods Diagonalization and Similarity Transformations 86, Diagonalizability Diagonalizability Symmetric Matrices Canonical Forms Symmetric Matrices Canonical Forms Symmetric Matrices Symmetric Matrices Similarity Transformations Similarity Transformations A real symmetric matrix has all real eigenvalues and Proposition: Eigenvalues of a real symmetric matrix must be real. is diagonalizable through an orthogonal similarity n n T transformation. Take A R × such that A = A , with eigenvalue λ = h + ik. ∈ Since λI A is singular, so is − Eigenvalues must be real. B = (λI A) (λ¯I A) = (hI A + ikI)(hI A ikI) A complete set of eigenvectors exists. − − − − − = (hI A)2 + k2I Eigenvectors corresponding to distinct eigenvalues are − necessarily orthogonal. For some x = 0, Bx = 0, and Corresponding to repeated eigenvalues, orthogonal eigenvectors 6 are available. xT Bx = 0 xT (hI A)T (hI A)x + k 2xT x = 0 ⇒ − − In all cases of a symmetric matrix, we can form an T Thus, (hI A)x 2 + kx 2 = 0 orthogonal matrix V, such that V AV = Λ is a real k − k k k diagonal matrix. k = 0 and λ = h Further, A = VΛVT . Similar results for complex Hermitian matrices.

Applied Mathematical Methods Diagonalization and Similarity Transformations 87, Applied Mathematical Methods Diagonalization and Similarity Transformations 88, Diagonalizability Diagonalizability Symmetric Matrices Canonical Forms Symmetric Matrices Canonical Forms Symmetric Matrices Symmetric Matrices Similarity Transformations Similarity Transformations Proposition: A symmetric matrix possesses a complete set of Proposition: Eigenvectors of a symmetric matrix corresponding to eigenvectors. distinct eigenvalues are necessarily orthogonal. Consider a repeated real eigenvalue λ of A and examine its Jordan Take two eigenpairs (λ1, v1) and (λ2, v2), with λ1 = λ2. 6 block(s). T T T v1 Av2 = v1 (λ2v2) = λ2v1 v2 Suppose Av = λv. vT Av = vT AT v = (Av )T v = (λ v )T v = λ vT v The first generalized eigenvector w satisfies (A λI)w = v, giving 1 2 1 2 1 2 1 1 2 1 1 2 − From the two expressions, (λ λ )vT v = 0 vT (A λI)w = vT v vT AT w λvT w = vT v 1 − 2 1 2 − ⇒ − vT v = 0 (Av)T w λvT w = v 2 1 2 ⇒ − k k v 2 = 0 ⇒ k k Proposition: Corresponding to a repeated eigenvalue of a symmetric matrix, an appropriate number of orthogonal which is absurd. eigenvectors can be selected. An eigenvector will not admit a generalized eigenvector. If λ1 = λ2, then the entire subspace < v1, v2 > is an eigenspace. All Jordan blocks will be of 1 1 size. Select any two mutually orthogonal eigenvectors for the basis. × Applied Mathematical Methods Diagonalization and Similarity Transformations 89, Applied Mathematical Methods Diagonalization and Similarity Transformations 90, Diagonalizability Diagonalizability Symmetric Matrices Canonical Forms Similarity Transformations Canonical Forms Symmetric Matrices Symmetric Matrices Facilities with the ‘omnipresent’ symmetric matrices:Similarity Transformations Similarity Transformations I Expression General Hessenberg A = VΛVT T Symmetric Tridiagonal Triangular λ1 v1 T λ2 v2 = [v1 v2 vn]  .   .  Symmetric Tridiagonal · · · .. .    T  Diagonal  λn   v     n   n   T T T T Figure: Eigenvalue problem: forms and steps = λ v v + λ v v + + λnvnv = λi vi v 1 1 1 2 2 2 · · · n i i X=1 I Reconstruction from a sum of rank-one components How to find suitable similarity transformations? I Efficient storage with only large eigenvalues and corresponding 1. rotation eigenvectors 2. reflection I Deflation technique 3. matrix decomposition or factorization I Stable and effective methods: easier to solve the eigenvalue problem 4. elementary transformation

Applied Mathematical Methods Diagonalization and Similarity Transformations 91, Applied Mathematical Methods Jacobi and Givens Rotation Methods 92, Diagonalizability Plane Rotations Points to note Canonical Forms Outline Jacobi Rotation Method Symmetric Matrices Givens Rotation Method Similarity Transformations

I Generally possible reduction: Jordan canonical form I Condition of diagonalizability and the diagonal form I Possible with orthogonal similarity transformations: triangular Jacobi and Givens Rotation Methods form (for symmetric matrices) I Useful non-canonical forms: tridiagonal and Hessenberg Plane Rotations Jacobi Rotation Method I Orthogonal diagonalization of symmetric matrices Givens Rotation Method Caution: Each step in this context to be effected through similarity transformations

Necessary Exercises: 1,2,4 Applied Mathematical Methods Jacobi and Givens Rotation Methods 93, Applied Mathematical Methods Jacobi and Givens Rotation Methods 94, Plane Rotations Plane Rotations Plane Rotations Jacobi Rotation Method Plane Rotations Jacobi Rotation Method Givens Rotation Method Givens Rotation Method

Y / Y Orthogonal change of basis: P (x, y) x cos φ sin φ x0 r = = = r0 φ y sin φ cos φ y <      0  y − y / Mapping of position vectors with L x M O φ X x / 1 T cos φ sin φ K N − = = − < < sin φ cos φ X /  

Figure: Rotation of axes and change of basis In three-dimensional (ambient) space,

cos φ sin φ 0 cos φ 0 sin φ xy = sin φ cos φ 0 , xz = 0 1 0 etc. <  −  <   x = OL + LM = OL + KN = x cos φ + y sin φ 0 0 1 sin φ 0 cos φ 0 0 −     y = PN MN = PN LK = y 0 cos φ x0 sin φ − − −

Applied Mathematical Methods Jacobi and Givens Rotation Methods 95, Applied Mathematical Methods Jacobi and Givens Rotation Methods 96, Plane Rotations Plane Rotations Plane Rotations Jacobi Rotation Method Jacobi Rotation Method Jacobi Rotation Method Givens Rotation Method Givens Rotation Method Generalizing to n-dimensional Euclidean space (R n), 1 0 0 a0 = a0 = carp sarq for p = r = q, 1 0 0 pr rp − 6 6  . . .  aqr0 = arq0 = carq + sarp for p = r = q, .. . . 6 6 . . 2 2   a0 = c app + s aqq 2scapq ,  1 0 0  pp −   2 2  0 0 0 c 0 0 s 0  a0 = s app + c aqq + 2scapq , and  · · · · · · · · ·  qq  0 1 0  2 2 Ppq =   ap0 q = aqp0 = (c s )apq + sc(app aqq)  . . .  − −  . .. .     0 1 0  In a Jacobi rotation,    0 0 0 s 0 0 c 0    c2 s2 a a  · · · −. · · · . ·.· ·  qq pp  . . ..  ap0 q = 0 − = − = k (say).   ⇒ 2sc 2apq  0 0 1      Left side is cot 2φ: solve this equation for φ. Matrix A is transformed as Jacobi rotation transformations P12, P13, , P1n; P23, , P2n; 1 T · · · · · · A0 = Pp−q APpq = PpqAPpq, ; Pn 1,n complete a full sweep. · · · − only the p-th and q-th rows and columns being affected. Note: The resulting matrix is far from diagonal! Applied Mathematical Methods Jacobi and Givens Rotation Methods 97, Applied Mathematical Methods Jacobi and Givens Rotation Methods 98, Plane Rotations Plane Rotations Jacobi Rotation Method Jacobi Rotation Method Givens Rotation Method Jacobi Rotation Method Givens Rotation Method Givens Rotation Method Sum of squares of off-diagonal terms before the transformation arq While applying the rotation Ppq, demand a0 = 0: tan φ = rq − arp 2 2 2 r = p 1: Givens rotation S = ars = 2 arp + arq − | |   I r=s r=p p=r=q Once ap 1,q is annihilated, it is never updated again! X6 X6 6X6 −   Sweep P23, P24, , P2n; P34, , P3n; ; Pn 1,n to = 2 (a2 + a2 ) + a2 · · · · · · · · · −  rp rq pq annihilate a13, a14, , a1n; a24, , a2n; ; an 2,n. p=r=q · · · · · · · · · − 6X6 Symmetric tridiagonal matrix and that afterwards  

2 2 2 S0 = 2 (a0 + a0 ) + a0 How do eigenvectors transform through Jacobi/Givens rotation  rp rq pq p=r=q steps? 6 6 X T T = 2  (a2 + a2 )  A∼ = P(2) P(1) AP(1)P(2) rp rq · · · · · · p=r=q (1) (2) 6X6 Product matrix P P gives the basis. differ by · · · To record it, initialize V by identity and keep multiplying new 2 ∆S = S0 S = 2a 0; and S 0. rotation matrices on the right side. − − pq ≤ →

Applied Mathematical Methods Jacobi and Givens Rotation Methods 99, Applied Mathematical Methods Jacobi and Givens Rotation Methods 100, Plane Rotations Plane Rotations Givens Rotation Method Jacobi Rotation Method Points to note Jacobi Rotation Method Givens Rotation Method Givens Rotation Method

Contrast between Jacobi and Givens rotation methods Rotation transformation on symmetric matrices I What happens to intermediate zeros? I Plane rotations provide orthogonal change of basis that can I What do we get after a complete sweep? be used for diagonalization of matrices. I How many sweeps are to be applied? I For small matrices (say 4 n 8), Jacobi rotation sweeps ≤ ≤ I What is the intended final form of the matrix? are competitive enough for diagonalization upto a reasonable tolerance. I How is size of the matrix relevant in the choice of the method? I For large matrices, one sweep of Givens rotations can be applied to get a symmetric tridiagonal matrix, for efficient Fast forward ... further processing. I Housholder method accomplishes ‘tridiagonalization’ more efficiently than Givens rotation method. Necessary Exercises: 2,3,4 I But, with a half-processed matrix, there come situations in which Givens rotation method turns out to be more efficient! Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 101, Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 102, Householder Reflection Transformation Householder Reflection Transformation Outline Householder Method Householder Reflection TransformationHouseholder Method Eigenvalues of Symmetric Tridiagonal Matrices Eigenvalues of Symmetric Tridiagonal Matrices

u u − v w Plane of O Reflection

v Householder Transformation and Tridiagonal Matrices Figure: Vectors in Householder reflection Householder Reflection Transformation Householder Method k u v Eigenvalues of Symmetric Tridiagonal Matrices Consider u, v R , u = v and w = u−v . ∈ k k k k k − k Householder reflection matrix

T Hk = Ik 2ww − is symmetric and orthogonal. For any vector x orthogonal to w,

T T Hk x = (Ik 2ww )x = x and Hk w = (Ik 2ww )w = w. − − −

Hence, Hk y = Hk (yw + y ) = yw + y , Hk u = v and Hk v = u. ⊥ − ⊥

Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 103, Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 104, Householder Reflection Transformation Householder Reflection Transformation Householder Method Householder Method Householder Method Householder Method Eigenvalues of Symmetric Tridiagonal Matrices Eigenvalues of Symmetric Tridiagonal Matrices Next, with v = u e , we form Consider n n symmetric matrix A. 2 k 2k 1 × T n 1 n 1 Let u = [a21 a31 an1] R − and v = u e1 R − . I2 0 · · · ∈ k k ∈ P2 = 0 Hn 2 1 0  −  Construct P1 = and operate as 0 Hn 1 (2) (1)  −  and operate as A = P2A P2. T After j steps, (1) 1 0 a11 u 1 0 A = P1AP1 = 0 Hn 1 u A1 0 Hn 1 d e  −     −  1 2 a vT .. = 11 .  e2 d2 .  v Hn 1A1Hn 1 A(j) = .. ..  − −   . . ej+1   T   ej+1 dj+1 u  Reorganizing and re-naming,  j+1   uj Aj   +1 +1    d1 e2 0 By n 2 steps, with P = P1P2P3 Pn 2, (1) T − A = e2 d2 u2 . − · · ·   (n 2) T 0 u2 A2 A − = P AP   is symmetric tridiagonal. Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 105, Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 106, Householder Reflection Transformation Householder Reflection Transformation Eigenvalues of Symmetric TridiagonalHouseholderMatricesMethod Eigenvalues of Symmetric TridiagonalHouseholderMatricesMethod Eigenvalues of Symmetric Tridiagonal Matrices Eigenvalues of Symmetric Tridiagonal Matrices

Characteristic polynomial of the leading k k sub-matrix: pk (λ) d1 e2 × ..  e2 d2 .  T = .. .. p0(λ) = 1,  . . en 1   −  p1(λ) = λ d1,  en 1 dn 1 en  −  − −  2  en dn  p2(λ) = (λ d2)(λ d1) e2 ,   − − −   , · · · · · · · · · Characteristic polynomial 2 pk+1(λ) = (λ dk+1)pk (λ) ek+1pk 1(λ). − − − λ d e − 1 − 2 .. P(λ) = p (λ), p (λ), , p (λ) e2 λ d2 . 0 1 n − − { · · · } p(λ) = .. .. . I a Sturmian sequence if ej = 0 j . . en 1 6 ∀ − − en 1 λ dn 1 en − − − − − Question: What if ej = 0 for some j?! en λ dn − − Answer: That is good news. Split the matrix.

Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 107, Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 108, Householder Reflection Transformation Householder Reflection Transformation Eigenvalues of Symmetric TridiagonalHouseholderMatricesMethod Eigenvalues of Symmetric TridiagonalHouseholderMatricesMethod Eigenvalues of Symmetric Tridiagonal Matrices Eigenvalues of Symmetric Tridiagonal Matrices Sturmian sequence property of P(λ) with ej = 0: 6 Next, we assume that the statement is true for k = i. Interlacing property: Roots of pk+1(λ) interlace the Roots of pi (λ): α > α > > αi 1 2 · · · roots of pk (λ). That is, if the roots of pk+1(λ) are Roots of pi (λ): β > β > > βi > βi +1 1 2 · · · +1 λ1 > λ2 > > λk+1 and those of pk (λ) are Roots of pi (λ): γ > γ > > γi > γi > γi · · · +2 1 2 · · · +1 +2 µ1 > µ2 > > µk ; then · · · Assumption: β > α > β > α > > βi > αi > βi 1 1 2 2 · · · · · · +1 γ α α α α γ λ1 > µ1 > λ2 > µ2 > > λk > µk > λk+1. i+2 i i−1 2 1 1 · · · · · · 8 Ο Ο 8 β β β β i+1 i 2 1

procedure (a) Roots of p ( λ ) and p ( λ ) This property leads to a convenient . i i+1 Proof α α α p (λ) has a single root, d . j+1 j j−1 1 1 β β j+1 j 2 ve ve p2(d1) = e2 < 0, (b) Sign of p p − i i+2 Since p ( ) = > 0, roots t and t of p (λ) are separated as 2 ∞ ∞ 1 2 2 > t > d > t > . Figure: Interlacing of roots of characteristic polynomials ∞ 1 1 2 −∞ The statement is true for k = 1. To show: γ > β > γ > β > > γi > βi > γi 1 1 2 2 · · · · · · +1 +1 +2 Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 109, Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 110, Householder Reflection Transformation Householder Reflection Transformation Eigenvalues of Symmetric TridiagonalHouseholderMatricesMethod Eigenvalues of Symmetric TridiagonalHouseholderMatricesMethod Eigenvalues of Symmetric Tridiagonal Matrices Eigenvalues of Symmetric Tridiagonal Matrices

Since β > α , pi (β ) is of the same sign as pi ( ), i.e. positive. 1 1 1 ∞ Examine sequence P(w) = p0(w), p1(w), p2(w), , pn(w) . 2 { · · · } Therefore, pi (β ) = e pi (β ) is negative. If pk (w) and pk+1(w) have opposite signs then pk+1(λ) has one +2 1 − i+2 1 But, pi ( ) is clearly positive. root more than pk (λ) in the interval (w, ). +2 ∞ ∞ Hence, γ (β , ). Number of roots of p (λ) above w = number of sign 1 ∈ 1 ∞ n Similarly, γi ( , βi ). changes in the sequence P(w). +2 ∈ −∞ +1 Question: Where are the rest of the i roots of pi+2(λ)? Consequence: Number of roots of pn(λ) in (a, b) = difference 2 2 between numbers of sign changes in P(a) and P(b). pi+2(βj ) = (βj di+2)pi+1(βj ) ei+2pi (βj ) = ei+2pi (βj ) 2− − − a+b pi (βj ) = e pi (βj ) Bisection method: Examine the sequence at 2 . +2 +1 − i+2 +1 Separate roots, bracket each of them and then squeeze That is, pi and pi+2 are of opposite signs at each β. the interval! Refer figure. Over [βi+1, β1], pi+2(λ) changes sign over each sub-interval Any way to start with an interval to include all eigenvalues? [βj+1, βj ], along with pi (λ), to maintain opposite signs at each β. Conclusion: p (λ) has exactly one root in (β , β ). λi λbnd = max ej + dj + ej+1 i+2 j+1 j | | ≤ 1 j n{| | | | | |} ≤ ≤

Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 111, Applied Mathematical Methods Householder Transformation and Tridiagonal Matrices 112, Householder Reflection Transformation Householder Reflection Transformation Eigenvalues of Symmetric TridiagonalHouseholderMatricesMethod Points to note Householder Method Eigenvalues of Symmetric Tridiagonal Matrices Eigenvalues of Symmetric Tridiagonal Matrices

I A Householder matrix is symmetric and orthogonal. It effects Algorithm a reflection transformation. I Identify the interval [a, b] of interest. I A sequence of Householder transformations can be used to I For a degenerate case (some ej = 0), split the given matrix. convert a symmetric matrix into a symmetric tridiagonal form. I For each of the non-degenerate matrices, I Eigenvalues of the leading square sub-matrices of a symmetric I by repeated use of bisection and study of the sequence P(λ), tridiagonal matrix exhibit a useful interlacing structure. bracket individual eigenvalues within small sub-intervals, and I I by further use of the bisection method (or a substitute) within This property can be used to separate and bracket eigenvalues. each such sub-interval, determine the individual eigenvalues to I Method of bisection is useful in the separation as well as the desired accuracy. subsequent determination of the eigenvalues.

Note: The algorithm is based on Sturmian sequence property . Necessary Exercises: 2,4,5 Applied Mathematical Methods QR Decomposition Method 113, Applied Mathematical Methods QR Decomposition Method 114, QR Decomposition QR Decomposition Outline QR Iterations QR Decomposition QR Iterations Conceptual Basis of QR Method* Conceptual Basis of QR Method* QR Algorithm with Shift* Decomposition (or factorization) A = QR intoQR Algotwrithmo factowith Shift*rs, orthogonal Q and upper-triangular R: (a) It always exists. (b) Performing this decomposition is pretty straightforward. QR Decomposition Method (c) It has a number of properties useful in the solution of the QR Decomposition eigenvalue problem. QR Iterations r11 r1n ·.· · . Conceptual Basis of QR Method* [a1 an] = [q1 qn] .. . · · · · · ·  .  QR Algorithm with Shift* r  nn  A simple method based on Gram-Schmidt orthogonalization: j Considering columnwise equality aj = i=1 rij qi , for j = 1, 2, 3, , n; · · · P j 1 T − rij = q aj i < j, a0 = aj rij qi , rjj = a0 ; i ∀ j − k j k i X=1 aj0 /rjj , if rjj = 0; qj = 6 T any vector satisfying q qj = δij for 1 i j, if rjj = 0.  i ≤ ≤

Applied Mathematical Methods QR Decomposition Method 115, Applied Mathematical Methods QR Decomposition Method 116, QR Decomposition QR Decomposition QR Decomposition QR Iterations QR Decomposition QR Iterations Conceptual Basis of QR Method* Conceptual Basis of QR Method* Practical method: one-sided Householder QRtransfoAlgorithmrmations,with Shift* QR Algorithm with Shift* starting with Alternative method useful for tridiagonal and Hessenberg n u0 v0 u0 = a1, v0 = u0 e1 R and w0 = − matrices: One-sided plane rotations k k ∈ u0 v0 I k − k rotations P12, P23 etc to annihilate a21, a32 etc in that T and P = Hn = In 2w w . sequence 0 − 0 0 Givens rotation matrices!

a1 Pn 2Pn 3 P2P1P0A = Pn 2Pn 3 P2P1 k k ∗∗ − − · · · − − · · · 0 A0   Application in solution of a linear system: Q and R factors of r 11 ∗ ∗∗ a matrix A come handy in the solution of Ax = b = Pn 2Pn 3 P2 r22 = = R − − · · ·  ∗∗  · · · · · · A1 QRx = b Rx = QT b   ⇒ With needs only a sequence of back-substitutions. T Q = (Pn 2Pn 3 P2P1P0) = P0P1P2 Pn 3Pn 2, − − · · · · · · − − we have QT A = R A = QR. ⇒ Applied Mathematical Methods QR Decomposition Method 117, Applied Mathematical Methods QR Decomposition Method 118, QR Decomposition QR Decomposition QR Iterations QR Iterations QR Iterations QR Iterations Conceptual Basis of QR Method* Conceptual Basis of QR Method* QR Algorithm with Shift* QR Algorithm with Shift* Multiplying Q and R factors in reverse, Quasi-upper-triangular form: λ1 ?? T ∗ · · · ∗ · · · ∗ ∗ A0 = RQ = Q AQ, λ2 ??  ·.· · ∗ · · · ∗ ∗  .. ?? an orthogonal similarity transformation.  ∗ · · · ∗ ∗   λr ??   · · · ∗ ∗  1. If A is symmetric, then so is A0.   ,  B  2. If A is in upper Hessenberg form, then so is A0.  k · · · ∗ ∗   . . .   .. . .  3. If A is symmetric tridiagonal, then so is A0.    α ω  Complexity of QR iteration: (n) for a symmetric tridiagonal  −  O  ω β  matrix, (n2) operation for an upper Hessenberg matrix and     O   (n3) for the general case. with λ1 > λ2 > . I | | | | · · · O Diagonal blocks Bk correspond to eigenspaces of equal/close Algorithm: Set A = A and for k = 1, 2, 3, , 1 · · · (magnitude) eigenvalues. I decompose Ak = Qk Rk , I 2 2 diagonal blocks often correspond to pairs of complex × I reassemble Ak+1 = Rk Qk . eigenvalues (for non-symmetric matrices). I For symmetric matrices, the quasi-upper-triangular form As k , Ak approaches the quasi-upper-triangular form. → ∞ reduces to quasi-diagonal form.

Applied Mathematical Methods QR Decomposition Method 119, Applied Mathematical Methods QR Decomposition Method 120, QR Decomposition QR Decomposition Conceptual Basis of QR Method* QR Iterations QR Algorithm with Shift* QR Iterations Conceptual Basis of QR Method* Conceptual Basis of QR Method* QR Algorithm with Shift* QR Algorithm with Shift*k λi For λi < λj , entry aij decays through iterations as . QR decomposition algorithm operates on the basis of the relative λj magnitudes of eigenvalues and segregates subspaces. With shift,  

With k , A¯ k = Ak µk I; → ∞ − ¯ ¯ k Ak = Qk Rk , Ak+1 = Rk Qk ; A Range e1 = Range q1 Range v1 { } { } → { } Ak+1 = A¯ k+1 + µk I. T T and (a1)k k Aq1 = λ1 k q1 = λ1e1. → Q Q Resulting transformation is Further, T ¯ Ak+1 = Rk Qk + µk I = Qk Ak Qk + µk I k A Range e1, e2 = Range q1, q2 Range v1, v2 . T T = Q (Ak µk I)Qk + µk I = Q Ak Qk . { } { } → { } k − k (λ1 λ2)α1 T − For the iteration, and (a2)k k Aq2 = λ2 . λi µ → Q   convergence ratio = − k . 0 λj µ − k And, so on ...   Question: How to find a suitable value for µk ? Applied Mathematical Methods QR Decomposition Method 121, Applied Mathematical Methods Eigenvalue Problem of General Matrices 122, QR Decomposition Introductory Remarks Points to note QR Iterations Outline Reduction to Hessenberg Form* Conceptual Basis of QR Method* QR Algorithm on Hessenberg Matrices* QR Algorithm with Shift* Inverse Iteration Recommendation I QR decomposition can be effected on any square matrix. I Practical methods of QR decomposition use Householder transformations or Givens rotations. I A QR iteration effects a similarity transformation on a matrix, Eigenvalue Problem of General Matrices preserving symmetry, Hessenberg structure and also a Introductory Remarks symmetric tridiagonal form. Reduction to Hessenberg Form* QR Algorithm on Hessenberg Matrices* I A sequence of QR iterations converge to an almost Inverse Iteration upper-triangular form. Recommendation I Operations on symmetric tridiagonal and Hessenberg forms are computationally efficient. I QR iterations tend to order subspaces according to the relative magnitudes of eigenvalues. I Eigenvalue shifting is useful as an expediting strategy.

Necessary Exercises: 1,3

Applied Mathematical Methods Eigenvalue Problem of General Matrices 123, Applied Mathematical Methods Eigenvalue Problem of General Matrices 124, Introductory Remarks Introductory Remarks Introductory Remarks Reduction to Hessenberg Form* Reduction to Hessenberg Form* Reduction to Hessenberg Form* QR Algorithm on Hessenberg Matrices* QR Algorithm on Hessenberg Matrices* Inverse Iteration Inverse Iteration Recommendation Recommendation I A general (non-symmetric) matrix may not be diagonalizable. Methods to find appropriate similarity transformations We attempt to triangularize it. 1. a full sweep of Givens rotations, I With real arithmetic, 2 2 diagonal blocks are inevitable — 2. a sequence of n 2 steps of Householder transformations, and × − signifying complex pair of eigenvalues. 3. a cycle of coordinated Gaussian elimination. I Higher computational complexity, slow convergence and lack of numerical stability. Method based on Gaussian elimination or elementary transformations: A non-symmetric matrix is usually unbalanced and is prone to The pre-multiplying matrix corresponding to the higher round-off errors. elementary row transformation and the post-multiplying matrix corresponding to the matching column Balancing as a pre-processing step: multiplication of a row and transformation must be inverses of each other. division of the corresponding column with the same number, ensuring similarity. Two kinds of steps I Note: A balanced matrix may get unbalanced again through Pivoting similarity transformations that are not orthogonal! I Elimination Applied Mathematical Methods Eigenvalue Problem of General Matrices 125, Applied Mathematical Methods Eigenvalue Problem of General Matrices 126, Introductory Remarks Introductory Remarks Reduction to Hessenberg Form* Reduction to Hessenberg Form* QR Algorithm on Hessenberg Matrices*Reduction to Hessenberg Form* QR Algorithm on Hessenberg Matrices* QR Algorithm on Hessenberg Matrices* Inverse Iteration Inverse Iteration ¯ 1 Recommendation Recommendation Pivoting step: A = Prs APrs = Prs− APrs . QR iterations: (n2) operations for upper Hessenberg form. I O Permutation Prs : interchange of r-th and s-th columns. Whenever a sub-diagonal zero appears, the matrix is split I 1 Prs− = Prs : interchange of r-th and s-th rows. into two smaller upper Hessenberg blocks, and they are I Pivot locations: a21, a32, , an 1,n 2. processed separately, thereby reducing the cost drastically. · · · − − ¯ 1 Elimination step: A = Gr− AGr with elimination matrix Particular cases: I Ir 0 0 Ir 0 0 an,n 1 0: Accept ann = λn as an eigenvalue, continue with 1 − → Gr = 0 1 0 and Gr− = 0 1 0 . the leading (n 1) (n 1) sub-matrix.     − × − 0 k In r 1 0 k In r 1 I − − − − − an 1,n 2 0: Separately find the eigenvalues λn 1 and λn     − − → − an 1,n 1 an 1,n from − − − , continue with the leading I 1 G− : Row (r + 1 + i) Row (r + 1 + i) ki Row (r + 1) an,n 1 an,n r ← − ×  −  for i = 1, 2, 3, , n r 1 (n 2) (n 2) sub-matrix. · · · − − − × − I Gr : Column (r + 1) Column (r + 1)+ n←r 1 − − [ki Column (r + 1 + i) ] Shift strategy: Double QR steps. i=1 × P

Applied Mathematical Methods Eigenvalue Problem of General Matrices 127, Applied Mathematical Methods Eigenvalue Problem of General Matrices 128, Introductory Remarks Introductory Remarks Inverse Iteration Reduction to Hessenberg Form* Inverse Iteration Reduction to Hessenberg Form* QR Algorithm on Hessenberg Matrices* QR Algorithm on Hessenberg Matrices* Inverse Iteration n n Inverse Iteration Recommendation With y = αj vj and y = βj vj , [RecommendationA (λi ) I]y = y gives Assumption: Matrix A has a complete set of eigenvectors. 0 j=1 j=1 − 0 0 n P P n (λi )0: a good estimate of an eigenvalue λi of A. βj [A (λi ) I]vj = αj vj − 0 j j Purpose: To find λi precisely and also to find vi . X=1 X=1 αj βj [λj (λi )0] = αj βj = . Step: Select a random vector y0 (with y0 = 1) and solve ⇒ − ⇒ λj (λi ) k k − 0 [A (λi )0I]y = y0. β is typically large and eigenvector v dominates y. − i i Avi = λi vi gives [A (λi ) I]vi = [λi (λi ) ]vi . Hence, − 0 − 0 Result: y is a good estimate of vi and [λi (λi )0]y [A (λi )0I]y = y0. 1 − ≈ − (λi )1 = (λi )0 + T y0 y Inner product with y0 gives

is an improvement in the estimate of the eigenvalue. T 1 [λi (λi )0]y0 y 1 λi (λi )0 + T . − ≈ ⇒ ≈ y0 y How to establish the result and work out an algorithm ? Applied Mathematical Methods Eigenvalue Problem of General Matrices 129, Applied Mathematical Methods Eigenvalue Problem of General Matrices 130, Introductory Remarks Introductory Remarks Inverse Iteration Reduction to Hessenberg Form* Recommendation Reduction to Hessenberg Form* QR Algorithm on Hessenberg Matrices* QR Algorithm on Hessenberg Matrices* Inverse Iteration Inverse Iteration Algorithm: Recommendation Recommendation

Start with estimate (λi )0, guess y0 (normalized). Table: Eigenvalue problem: summary of methods For k = 0, 1, 2, · · · Type Size Reduction Algorithm Post-processing I General Small Definition: Polynomial Solution of Solve [A (λi )k I]y = yk . (up to 4) Characteristic root finding linear systems − y polynomial (eigenvalues) (eigenvectors) I Symmetric Intermediate Jacobi sweeps Selective Normalize yk+1 = y . k k (say, 4–12) Jacobi rotations I 1 Tridiagonalization Sturm sequence Inverse iteration Improve (λi )k+1 = (λi )k + T . (Givens rotation property: (eigenvalue yk y or Householder Bracketing and improvement I method) bisection and eigenvectors) If yk+1 yk < , terminate. (rough eigenvalues) k − k Large Tridiagonalization QR decomposition (usually iterations Important issues Householder method) Balancing, and then I Non- Intermediate Reduction to QR decomposition Inverse iteration Update eigenvalue once in a while, not at every iteration. symmetric Large Hessenberg form iterations (eigenvectors) I (Above methods or (eigenvalues) Use some acceptable small number as artificial pivot. Gaussian elimination) General Very large Power method, I The method may not converge for defective matrix or for one (selective shift and deflation requirement) having complex eigenvalues. I Repeated eigenvalues may inhibit the process.

Applied Mathematical Methods Eigenvalue Problem of General Matrices 131, Applied Mathematical Methods Singular Value Decomposition 132, Introductory Remarks SVD Theorem and Construction Points to note Reduction to Hessenberg Form* Outline Properties of SVD QR Algorithm on Hessenberg Matrices* Pseudoinverse and Solution of Linear Systems Inverse Iteration Optimality of Pseudoinverse Solution Recommendation SVD Algorithm I Eigenvalue problem of a non-symmetric matrix is difficult! I Balancing and reduction to Hessenberg form are desirable pre-processing steps. Singular Value Decomposition I QR decomposition algorithm is typically used for reduction to SVD Theorem and Construction an upper-triangular form. Properties of SVD I Use inverse iteration to polish eigenvalue and find Pseudoinverse and Solution of Linear Systems eigenvectors. Optimality of Pseudoinverse Solution SVD Algorithm I In algebraic eigenvalue problems, different methods or combinations are suitable for different cases; regarding matrix size, symmetry and the requirements.

Necessary Exercises: 1,2 Applied Mathematical Methods Singular Value Decomposition 133, Applied Mathematical Methods Singular Value Decomposition 134, SVD Theorem and Construction SVD Theorem and Construction SVD Theorem and Construction Properties of SVD SVD Theorem and Construction Properties of SVD Pseudoinverse and Solution of Linear Systems Pseudoinverse and Solution of Linear Systems 1 Optimality of Pseudoinverse Solution Optimality of Pseudoinverse Solution Eigenvalue problem: A = UΛV− where U SVD= VAlgorithm Question: How to construct U, V and Σ? SVD Algorithm m n Do not ask for similarity. Focus on the form of the decomposition. For A R × , ∈ Guaranteed decomposition with orthogonal U, V, and T T T T T T T non-negative diagonal entries in Λ. A A = (VΣ U )(UΣV ) = VΣ ΣV = VΛV , where Λ = ΣT Σ is an n n diagonal matrix. A = UΣVT such that UT AV = Σ × σ m n 1 | SVD Theorem For any real matrix A R × , there σ m m ∈ n n 2 exist orthogonal matrices U R × and V R × such  . |  ∈ ∈ .. 0 that Σ =  |  T m n  σp  U AV = Σ R ×  |  ∈  +   −− −− −− −− − − −−  is a diagonal matrix, with diagonal entries σ1, σ2, 0,   · · · ≥  0  obtained by appending the square diagonal matrix  | ×  diag (σ1, σ2, , σp) with (m p) zero rows or (n p) Determine V and Λ. Work out Σ and we have · · · − − zero columns, where p = min(m, n). A = UΣVT AV = UΣ Singular values: σ1, σ2, , σp. ⇒ Similar result for complex matrices· · · This provides a proof as well!

Applied Mathematical Methods Singular Value Decomposition 135, Applied Mathematical Methods Singular Value Decomposition 136, SVD Theorem and Construction SVD Theorem and Construction SVD Theorem and Construction Properties of SVD Properties of SVD Properties of SVD Pseudoinverse and Solution of Linear Systems Pseudoinverse and Solution of Linear Systems Optimality of Pseudoinverse Solution Optimality of Pseudoinverse Solution SVD Algorithm For a given matrix, the SVD is unique up toSVD Algorithm From AV = UΣ, determine columns of U. (a) the same permutations of columns of U, columns of V and 1. Column Avk = σk uk , with σk = 0: determine column uk . 6 diagonal elements of Σ; Columns developed are bound to be mutually (b) the same orthonormal linear combinations among columns of orthonormal! T U and columns of V, corresponding to equal singular values; T 1 1 Verify u uj = Avi Avj = δij . and i σi σj 2. Column Avk =σk uk , withσk = 0: uk is left indeterminate (c) arbitrary orthonormal linear combinations among columns of (free). U or columns of V, corresponding to zero or non-existent singular values. 3. In the case of m < n, identically zero columns Avk = 0 for k > m: no corresponding columns of U to determine. Ordering of the singular values: 4. In the case of m > n, there will be (m n) columns of U left − σ σ σr > 0, and σr = σr = = σp = 0. indeterminate. 1 ≥ 2 ≥ · · · ≥ +1 +2 · · · Extend columns of U to an orthonormal basis. Rank(A) = Rank(Σ) = r

All three factors in the decomposition are constructed, as desired. Rank of a matrix is the same as the number of its non-zero singular values. Applied Mathematical Methods Singular Value Decomposition 137, Applied Mathematical Methods Singular Value Decomposition 138, SVD Theorem and Construction SVD Theorem and Construction Properties of SVD Properties of SVD Properties of SVD Properties of SVD Pseudoinverse and Solution of Linear Systems Pseudoinverse and Solution of Linear Systems Optimality of Pseudoinverse Solution Optimality of Pseudoinverse Solution SVD Algorithm In basis V, v = c v + c v + + cnvn =SVDVcAlgoandrithm the norm is 1 1 2 2 · · · σ1y1 given by . Av 2 vT AT Av T  .  2 Ax = UΣV x = UΣy = [u1 ur ur+1 um] A = max k k2 = max T · · · · · · σr yr k k v v v v v   kT kT T T T 2 2  0  c V A AVc c Σ Σc k σk ck   = max T T = max T = max 2 . = σ y u + σ y u + + σr yr ur   c c V Vc c c c c c 1 1 1 2 2 2 · · · P k k has non-zero components along only the first r columns of U. 2 2 P Pk σk ck U gives an orthonormal basis for the co-domain such that A = maxc c2 = σmax k k Pk k r Range(A) = < u1, u2, , ur > . For a non-singular square matrix, · · · T T 1 T 1 1 T 1 1 1 T With V x = y, v x = y , and A− = (UΣV )− = VΣ− U = V diag , , , U . k k σ σ · · · σ  1 2 n  x = y1v1 + y2v2 + + yr vr + yr+1vr+1 + ynvn. 1 1 · · · · · · Then, A− = and the condition number is k k σmin V gives an orthonormal basis for the domain such that 1 σmax κ(A) = A A− = . k k k k σmin Null(A) = < vr , vr , , vn > . +1 +2 · · ·

Applied Mathematical Methods Singular Value Decomposition 139, Applied Mathematical Methods Singular Value Decomposition 140, SVD Theorem and Construction SVD Theorem and Construction Properties of SVD Properties of SVD Pseudoinverse and Solution of Linear PropSystemserties of SVD Pseudoinverse and Solution of Linear Systems Pseudoinverse and Solution of Linear Systems Optimality of Pseudoinverse Solution Optimality of Pseudoinverse Solution SVD Algorithm Generalized inverse: G is called a generalizedSVD Algoinverserithm or g-inverse Revision of definition of norm and condition number: of A if, for b Range(A), Gb is a solution of Ax = b. The norm of a matrix is the same as its largest singular ∈ The Moore-Penrose inverse or the pseudoinverse: value, while its condition number is given by the ratio of the largest singular value to the least. A# = (UΣVT )# = (VT )#Σ#U# = VΣ#UT

Σ 0 Σ 1 0 Arranging singular values in decreasing order, with Rank(A) = r, With Σ = r , Σ# = r− . 0 0 0 0 ¯ ¯     U = [Ur U] and V = [Vr V], ρ 1 | T ρ2 T Σr 0 Vr  |  A = UΣV = [Ur U¯ ] , .. 0 0 V¯ T # . 0     Or, Σ =  |  ,  ρp  or,  |  r  +  T T  −− −− −− −− − − −−  A = Ur Σr Vr = σk uk vk .  0  k=1  | ×  X 1   , for σk = 0 or for σk > ; Efficient storage and reconstruction! σk where ρk = 6 | | 0, for σk = 0 or for σk .  | | ≤ Applied Mathematical Methods Singular Value Decomposition 141, Applied Mathematical Methods Singular Value Decomposition 142, SVD Theorem and Construction SVD Theorem and Construction Pseudoinverse and Solution of Linear PropSystemserties of SVD Optimality of Pseudoinverse Solution Properties of SVD Pseudoinverse and Solution of Linear Systems Pseudoinverse and Solution of Linear Systems Optimality of Pseudoinverse Solution Optimality of Pseudoinverse Solution Inverse-like facets and beyond SVD Algorithm Pseudoinverse solution of Ax = b: SVD Algorithm I # # r r (A ) = A. # T T T I # 1 x∗ = VΣ U b = ρk vk uk b = (uk b/σk )vk If A is invertible, then A = A− . k=1 k=1 I A#b gives the correct unique solution. X X I If Ax = b is an under-determined consistent system, then Minimize A#b selects the solution x with the minimum norm. 1 1 1 ∗ E(x) = (Ax b)T (Ax b) = xT AT Ax xT AT b + bT b I If the system is inconsistent, then A#b minimizes the least 2 − − 2 − 2 square error Ax b . k − k Condition of vanishing gradient: I If the minimizer of Ax b is not unique, then it picks up that minimizer whichk has−thek minimum norm x among such ∂E k k = 0 AT Ax = AT b minimizers. ∂x ⇒ T T T T Contrast with Tikhonov regularization: V(Σ Σ)V x = VΣ U b ⇒ T T T T Pseudoinverse solution for precision and diagnosis. (Σ Σ)V x = Σ U b ⇒ 2 T T Tikhonov’s solution for continuity of solution over σk vk x = σk uk b ⇒ T T variable A and computational efficiency. v x = u b/σk for k = 1, 2, 3, , r. ⇒ k k · · ·

Applied Mathematical Methods Singular Value Decomposition 143, Applied Mathematical Methods Singular Value Decomposition 144, SVD Theorem and Construction SVD Theorem and Construction Optimality of Pseudoinverse Solution Properties of SVD Optimality of Pseudoinverse Solution Properties of SVD Pseudoinverse and Solution of Linear Systems Pseudoinverse and Solution of Linear Systems Optimality of Pseudoinverse Solution Optimality of Pseudoinverse Solution SVD Algorithm Anatomy of the optimization through SVDSVD Algorithm

With V¯ = [vr+1 vr+2 vn], then Using basis V for domain and U for co-domain, the variables are · · · transformed as r T T T ¯ ¯ V x = y and U b = c. x = (uk b/σk )vk + Vy = x∗ + Vy. Then, Xk=1 Ax = b UΣVT x = b ΣVT x = UT b Σy = c. ⇒ ⇒ ⇒ How to minimize x 2 subject to E(x) minimum? k k A completely decoupled system! ¯ 2 Minimize E1(y) = x∗ + Vy . Usable components: yk = ck /σk for k = 1, 2, 3, , r. k k · · · For k > r, I completely redundant information (ck = 0) Since x∗ and V¯y are mutually orthogonal, I purely unresolvable conflict (ck = 0) 2 2 2 6 E1(y) = x∗ + V¯ y = x∗ + V¯ y k k k k k k SVD extracts this pure redundancy/inconsistency. is minimum when V¯ y = 0, i.e. y = 0. Setting ρk = 0 for k > r rejects it wholesale! At the same time, y is minimized, and hence x too. k k k k Applied Mathematical Methods Singular Value Decomposition 145, Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 146, SVD Theorem and Construction Group Points to note Properties of SVD Outline Field Pseudoinverse and Solution of Linear Systems Vector Space Optimality of Pseudoinverse Solution Linear Transformation SVD Algorithm Isomorphism Inner Product Space Function Space I SVD provides a complete orthogonal decomposition of the domain and co-domain of a linear transformation, separating Vector Spaces: Fundamental Concepts* out functionally distinct subspaces. Group I If offers a complete diagnosis of the pathologies of systems of Field linear equations. Vector Space I Pseudoinverse solution of linear systems satisfy meaningful Linear Transformation optimality requirements in several contexts. Isomorphism I With the existence of SVD guaranteed, many important Inner Product Space results can be established in a straightforward manner. Function Space

Necessary Exercises: 2,4,5,6,7

Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 147, Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 148, Group Group Group Field Field Field Vector Space Vector Space Linear Transformation Linear Transformation Isomorphism Isomorphism Inner Product Space Inner Product Space A set G and a binary operation, say ‘+’, fulfillingFunction Space A set F and two binary operations, say ‘+’ Fandunction‘ Space’, satisfying · Closure: a + b G a, b G Group property for addition: (F , +) is a commutative group. ∈ ∀ ∈ Associativity: a + (b + c) = (a + b) + c, a, b, c G (Denote the identity element of this group as ‘0’.) ∀ ∈ Existence of identity: 0 G such that a G, a + 0 = a = 0 + a Group property for multiplication: (F 0 , ) is a commutative ∃ ∈ ∀ ∈ − { } · Existence of inverse: a G, ( a) G such that group. (Denote the identity element of this group as ∀ ∈ ∃ − ∈ a + ( a) = 0 = ( a) + a ‘1’.) − − Distributivity: a (b + c) = a b + a c, a, b, c F . · · · ∀ ∈ Examples: (Z, +), (Z, +), (Q 0 , ), 2 5 real matrices, − { } · × Rotations etc. Concept of field: abstraction of a number system

I Commutative group Examples: (Q, +, ), (R, +, ), (C, +, ) etc. · · ·

I Subgroup I Subfield Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 149, Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 150, Group Group Vector Space Field Vector Space Field Vector Space Vector Space Linear Transformation Linear Transformation Isomorphism Isomorphism A vector space is defined by Inner Product Space Suppose V is a vector space. Inner Product Space Function Space Take a vector ξ = 0 in it. Function Space I a field F of ‘scalars’, 1 6 I a commutative group V of ‘vectors’, and Then, vectors linearly dependent on ξ1: α ξ V α F . I a binary operation between F and V, that may be called 1 1 ∈ ∀ 1 ∈ ‘scalar multiplication’, such that α, β F , a, b V; the ∀ ∈ ∀ ∈ Question: Are the elements of V exhausted? following conditions hold. Closure: αa V. If not, then take ξ2 V: linearly independent from ξ1. ∈ ∈ Identity: 1a = a. Then, α1ξ1 + α2ξ2 V α1, α2 F . Associativity: (αβ)a = α(βa). ∈ ∀ ∈ Scalar distributivity: α(a + b) = αa + αb. Question: Are the elements of V exhausted now? Vector distributivity: (α + β)a = αa + βa. · · · · · · · · · Examples: R n, C n, m n real matrices etc. Question: Will this process ever end? × Field Number system Suppose it does. ↔ Vector space Space finite dimensional vector space ↔

Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 151, Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 152, Group Group Vector Space Field Linear Transformation Field Vector Space Vector Space Linear Transformation Linear Transformation Isomorphism Isomorphism Finite dimensional vector space Inner Product Space A mapping T : V W satisfying Inner Product Space Function Space → Function Space Suppose the above process ends after n choices of linearly T(αa + βb) = αT(a) + βT(b) α, β F and a, b V ∀ ∈ ∀ ∈ independent vectors. where V and W are vector spaces over the field F . χ = α ξ + α ξ + + αnξn 1 1 2 2 · · · Question: How to describe the linear transformation T?

Then, I For V, basis ξ , ξ , , ξn 1 2 · · · I I n: dimension of the vector space For W, basis η1, η2, , ηm I · · · ordered set ξ1, ξ2, , ξn: a basis ξ V gets mapped to T(ξ ) W. · · · 1 ∈ 1 ∈ I α , α , , αn F : coordinates of χ in that basis 1 2 · · · ∈ T(ξ1) = a11η1 + a21η2 + + am1ηm Rn, Rm etc: vector spaces over the field of real numbers · · · m Similarly, enumerate T(ξj ) = i=1 aij ηi . I Subspace Matrix A = [a a aPn] codes this description! 1 2 · · · Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 153, Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 154, Group Group Linear Transformation Field Linear Transformation Field Vector Space Vector Space Linear Transformation Linear Transformation Isomorphism Isomorphism A general element χ of V can be expressed Inneras Product Space Inner Product Space Function Space Understanding: Function Space I Vector χ is an actual object in the set V and the column χ = x1ξ1 + x2ξ2 + + xnξn · · · x Rn is merely a list of its coordinates. T ∈ Coordinates in a column: x = [x1 x2 xn] I T : V W is the linear transformation and the matrix A · · · → simply stores coefficients needed to describe it. Mapping: I By changing bases of V and W, the same vector χ and the

T(χ) = x1T(ξ1) + x2T(ξ2) + + xnT(ξn), same linear transformation are now expressed by different x · · · and A, respectively. with coordinates Ax, as we know!

Summary: Matrix representation emerges as the natural description of a linear transformation between two vector spaces. I basis vectors of V get mapped to vectors in W whose coordinates are listed in columns of A, and I Exercise: Set of all T : V W form a vector space of their own!! a vector of V, having its coordinates in x, gets mapped to a → vector in W whose coordinates are obtained from Ax. Analyze and describe that vector space.

Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 155, Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 156, Group Group Isomorphism Field Isomorphism Field Vector Space Vector Space Linear Transformation Linear Transformation Consider T : V W that establishes a one-to-oneIsomorphism correspondence. Isomorphism → Inner Product Space Consider vector spaces V and W over the sameInner Profieldduct SpaceF and of the I Function Space Function Space Linear transformation T defines a one-one onto mapping, same dimension n. which is invertible. I dim V = dim W Question: Can we define an isomorphism between them? I Inverse linear transformation T 1 : W V − → Answer: Of course. As many as we want! I T defines (is) an isomorphism. I Vector spaces V and W are isomorphic to each other. The underlying field and the dimension together completely specify a vector space, up to an isomorphism. I Isomorphism is an equivalence relation. V and W are equivalent! If we need to perform some operations on vectors in one vector I All n-dimensional vector spaces over the field F are space, we may as well isomorphic to one another. 1. transform the vectors to another vector space through an I In particular, they are all isomorphic to F n. isomorphism, I The representation (columns) can be considered as the 2. conduct the required operations there, and objects (vectors) themselves. 3. map the results back to the original space through the inverse. Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 157, Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 158, Group Group Inner Product Space Field Inner Product Space Field Vector Space Vector Space Linear Transformation Linear Transformation Isomorphism Isomorphism Inner product (a, b) in a real or complex vectoInner Pror ductspace:Space a scalar Inner Product Space Function Space Inner products bring in ideas of angle and lengthFunction Spacein the geometry function p : V V F satisfying × → of vector spaces. Closure: a, b V, (a, b) F ∀ ∈ ∈ Associativity: (αa, b) = α(a, b) Orthogonality: (a, b) = 0 Distributivity: (a + b, c) = (a, c) + (b, c) Norm: : V R, such that a = (a, a) k · k → k k Conjugate commutativity: (b, a) = (a, b) Associativity: αa = α a k k | | k k p Positive definiteness: (a, a) 0; and (a, a) = 0 iff a = 0 Positive definiteness: a > 0 for a = 0 and 0 = 0 ≥ k k 6 k k Triangle inequality: a + b a + b Note: Property of conjugate commutativity forces (a, a) to be real. k k ≤ k k k k Cauchy-Schwarz inequality: (a, b) a b T T Examples: a b, a Wb in R, a∗b in C etc. ≤ k k k k A distance function or metric: d : V V R such that Inner product space: a vector space possessing an inner product V × → I Euclidean space: over R dV(a, b) = a b I Unitary space: over C k − k

Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 159, Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 160, Group Group Function Space Field Function Space Field Vector Space Vector Space Linear Transformation Linear Transformation Isomorphism Isomorphism Inner Product Space Inner Product Space Suppose we decide to represent a continuousFunctionfunctionSpace Vector space of continuous functions Function Space f : [a, b] R by the listing → First, ( , +) is a commutative group. T F vf = [f (x ) f (x ) f (x ) f (xN )] 1 2 3 · · · Next, with α, β R, x [a, b], ∈ ∀ ∈ I with a = x < x < x < < xN = b. if f (x) R, then αf (x) R 1 2 3 · · · ∈ ∈ I 1 f (x) = f (x) Note: The ‘true’ representation will require N to be infinite! · I (αβ)f (x) = α[βf (x)] I Here, vf is a real column vector. α[f1(x) + f2(x)] = αf1(x) + αf2(x) Do such vectors form a vector space? I (α + β)f (x) = αf (x) + βf (x)

Correspondingly, does the set of continuous functions I F Thus, forms a vector space over R. over [a, b] form a vector space? F I Every function in this space is an (infinite dimensional) vector. infinite dimensional vector space I Listing of values is just an obvious basis. Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 161, Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 162, Group Group Function Space Field Function Space Field Vector Space Vector Space Linear Transformation Linear Transformation Isomorphism Isomorphism Linear dependence of (non-zero) functionsInnerf1 Proandduct fSpace2 Inner product: For functions f (x) and g(xInner) inProduct, theSpaceusual inner Function Space FunctionFSpace I product between corresponding vectors: f2(x) = kf1(x) for all x in the domain I k f (x) + k f (x) = 0, x with k and k not both zero. T 1 1 2 2 1 2 (vf , vg ) = v vg = f (x )g(x ) + f (x )g(x ) + f (x )g(x ) + ∀ f 1 1 2 2 3 3 · · · Linear independence: k1f1(x) + k2f2(x) = 0 x k1 = k2 = 0 ∀ ⇒ T Weighted inner product: (vf , vg ) = v Wvg = wi f (xi )g(xi ) In general, f i For the functions, I Functions f , f , f , , fn are linearly dependent if P 1 2 3 · · · ∈ F k , k , k , , kn, not all zero, such that b ∃ 1 2 3 · · · k1f1(x) + k2f2(x) + k3f3(x) + + knfn(x) = 0 x [a, b]. (f , g) = w(x)f (x)g(x)dx · · · ∀ ∈ Za I k f (x) + k f (x) + k f (x) + + knfn(x) = 0 x [a, b] 1 1 2 2 3 3 · · · ∀ ∈ ⇒ k1, k2, k3, , kn = 0 means that functions f1, f2, f3, , fn are · · · · · · I b linearly independent. Orthogonality: (f , g) = a w(x)f (x)g(x)dx = 0 b 2 3 I R 2 Example: functions 1, x, x , x , are a set of linearly Norm: f = a w(x)[f (x)] dx · · · k k independent functions. I Orthonormal basis:q b R (fj , fk ) = w(x)fj (x)fk (x)dx = δjk j, k Incidentally, this set is a commonly used basis. a ∀ R

Applied Mathematical Methods Vector Spaces: Fundamental Concepts* 163, Applied Mathematical Methods Topics in Multivariate Calculus 164, Group Derivatives in Multi-Dimensional Spaces Points to note Field Outline Taylor’s Series Vector Space Chain Rule and Change of Variables Linear Transformation Numerical Differentiation Isomorphism An Introduction to Tensors* Inner Product Space Function Space I Matrix algebra provides a natural description for vector spaces and linear transformations. I Through isomorphisms, R n can represent all n-dimensional Topics in Multivariate Calculus real vector spaces. Derivatives in Multi-Dimensional Spaces Taylor’s Series I Through the definition of an inner product, a vector space Chain Rule and Change of Variables incorporates key geometric features of physical space. Numerical Differentiation I Continuous functions over an interval constitute an infinite An Introduction to Tensors* dimensional vector space, complete with the usual notions.

Necessary Exercises: 6,7 Applied Mathematical Methods Topics in Multivariate Calculus 165, Applied Mathematical Methods Topics in Multivariate Calculus 166, Derivatives in Multi-Dimensional Spaces Derivatives in Multi-Dimensional Spaces Derivatives in Multi-Dimensional SpacesTaylor’s Series Derivatives in Multi-Dimensional SpacesTaylor’s Series Chain Rule and Change of Variables Chain Rule and Change of Variables Numerical Differentiation Numerical Differentiation Gradient An Introduction to Tensors* An Introduction to Tensors* T Hessian ∂f ∂f ∂f ∂f f (x) (x) = ∂2f ∂2f ∂2f ∇ ≡ ∂x ∂x1 ∂x2 · · · ∂xn 2   ∂x1 ∂x2∂x1 ∂xn∂x1 2 2 · · · 2 T 2 ∂ f ∂ f ∂ f 2 Up to the first order, δf [ f (x)] δx ∂ f  ∂x1∂x2 ∂x ∂xn∂x2  ≈ ∇ H(x) = = 2 · · · Directional derivative ∂x2 . . .. .  . . . .  ∂f f (x + αd) f (x)  ∂2f ∂2f ∂2f   2  = lim −  ∂x1∂xn ∂x2∂xn ∂xn  ∂d α 0 α · · · →   2 Relationships: Meaning: f (x + δx) f (x) ∂ f (x) δx ∇ − ∇ ≈ ∂x2 ∂f ∂f ∂f T ∂f h i = , = d f (x) and = f (x) For a vector function h(x), Jacobian ∂ej ∂xj ∂d ∇ ∂gˆ k∇ k Among all unit vectors, taken as directions, ∂h ∂h ∂h ∂h J(x) = (x) = I the rate of change of a function in a direction is the same as ∂x ∂x ∂x · · · ∂x  1 2 n  the component of its gradient along that direction, and Underlying notion: δh [J(x)]δx I the rate of change along the direction of the gradient is the ≈ greatest and is equal to the magnitude of the gradient.

Applied Mathematical Methods Topics in Multivariate Calculus 167, Applied Mathematical Methods Topics in Multivariate Calculus 168, Derivatives in Multi-Dimensional Spaces Derivatives in Multi-Dimensional Spaces Taylor’s Series Taylor’s Series Chain Rule and Change of Variables Taylor’s Series Chain Rule and Change of Variables Chain Rule and Change of Variables Numerical Differentiation Numerical Differentiation Taylor’s formula in the remainder form: An Introduction to Tensors* For f (x), the total differential: An Introduction to Tensors*

f (x + δx) = f (x) + f 0(x)δx T ∂f ∂f ∂f 1 2 1 (n 1) n 1 1 (n) n df = [ f (x)] dx = dx1 + dx2 + + dxn + f 00(x)δx + + f − (x)δx − + f (xc )δx ∇ ∂x1 ∂x2 · · · ∂xn 2! · · · (n 1)! n! − Ordinary derivative or total derivative: where xc = x + tδx with 0 t 1 ≤ ≤ Mean value theorem: existence of xc df T dx Taylor’s series: = [ f (x)] dt ∇ dt 1 2 f (x + δx) = f (x) + f 0(x)δx + f 00(x)δx + df f d 2! · · · For f (t, x(t)), total derivative: = ∂ + [ f (x)]T x dt ∂t ∇ dt For a multivariate function, For f (v, x(v)) = f (v , v , , vm, x (v), x (v), , xn(v)), 1 2 · · · 1 2 · · · 1 f (x + δx) = f (x) + [δxT ]f (x) + [δxT ]2f (x) + T ∂f ∂f ∂f ∂x ∂f T ∂x ∇ 2! ∇ · · · (v, x(v)) = + (v, x) = +[ x f (v, x)] 1 T n 1 1 T n ∂vi ∂vi x ∂x ∂vi ∂vi x ∇ ∂vi + [δx ] − f (x) + [δx ] f (x + tδx)       (n 1)! ∇ n! ∇ − 2 T T 1 T ∂ f ∂x f (x + δx) f (x) + [ f (x)] δx + δx (x) δx f (v, x(v)) = v f (v, x) + (v) x f (v, x) ≈ ∇ 2 ∂x2 ⇒ ∇ ∇ ∂v ∇     Applied Mathematical Methods Topics in Multivariate Calculus 169, Applied Mathematical Methods Topics in Multivariate Calculus 170, Derivatives in Multi-Dimensional Spaces Derivatives in Multi-Dimensional Spaces Chain Rule and Change of Variables Taylor’s Series Chain Rule and Change of Variables Taylor’s Series Chain Rule and Change of Variables Chain Rule and Change of Variables Numerical Differentiation Numerical Differentiation An Introduction to Tensors* For a multiple integral An Introduction to Tensors* Let x Rm+n and h(x) R m. ∈ ∈ Partition x R m+n into z Rn and w Rm. I = f (x, y, z) dx dy dz, ∈ ∈ ∈ A System of equations h(x) = 0 means h(z, w) = 0. Z Z Z change of variables x = x(u, v, w), y = y(u, v, w), z = z(u, v, w) Question: Can we work out the function w = w(z)? gives Solution of m equations in m unknowns? I = f (x(u, v, w), y(u, v, w), z(u, v, w)) J(u, v, w) du dv dw, ¯ | | Question: If we have one valid pair (z, w), then is it possible to Z ZA Z develop w = w(z) in the local neighbourhood? ∂(x,y,z) ∂h where Jacobian determinant J(u, v, w) = ∂(u,v,w) . Answer: Yes, if Jacobian ∂w is non-singular. | | For the differential Implicit function theorem

P1(x)dx1 + P2(x)dx2 + + Pn(x)dxn, 1 · · · ∂h ∂h ∂w ∂w ∂h − ∂h + = 0 = we ask: does there exist a function f (x), ∂z ∂w ∂z ⇒ ∂z − ∂w ∂z     I of which this is the differential; Upto first order, w = w + ∂w (z z). I or equivalently, the gradient of which is P(x)? 1 ∂z 1 − h i Perfect or exact differential: can be integrated to find f .

Applied Mathematical Methods Topics in Multivariate Calculus 171, Applied Mathematical Methods Topics in Multivariate Calculus 172, Derivatives in Multi-Dimensional Spaces Derivatives in Multi-Dimensional Spaces Chain Rule and Change of Variables Taylor’s Series Numerical Differentiation Taylor’s Series Chain Rule and Change of Variables Chain Rule and Change of Variables Numerical Differentiation Numerical Differentiation Differentiation under the integral sign An Introduction to Tensors* Forward difference formula An Introduction to Tensors* How To differentiate φ(x) = φ(x, u(x), v(x)) = v(x) f (x, t) dt? f (x + δx) f (x) u(x) f 0(x) = − + (δx) In the expression δx O R Central difference formulae ∂φ ∂φ du ∂φ dv φ0(x) = + + , f (x + δx) f (x δx) 2 ∂x ∂u dx ∂v dx f 0(x) = − − + (δx ) 2δx O ∂φ v ∂f we have = (x, t)dt. f (x + δx) 2f (x) + f (x δx) 2 ∂x u ∂x f 00(x) = − − + (δx ) ∂F (x,t) δx2 O Now, considering function F (x, t) such that f (x, t) = t , R ∂ For gradient f (x) and Hessian, v ∂F ∇ φ(x) = (x, t)dt = F (x, v) F (x, u) φ(x, u, v). ∂f 1 ∂t − ≡ (x) = [f (x + δei ) f (x δei )], Zu ∂xi 2δ − − ∂φ ∂φ Using ∂v = f (x, v) and ∂u = f (x, u), 2 − ∂ f f (x + δei ) 2f (x) + f (x δei ) 2 (x) = − 2 − , and v(x) ∂f dv du ∂xi δ φ0(x) = (x, t)dt + f (x, v) f (x, u) . f (x + δe + δe ) f (x + δe δe ) u(x) ∂x dx − dx i j i j Z 2 − − ∂ f f (x δei + δej ) + f (x δei δej ) − − − − (x) = 2 Leibnitz rule ∂xi ∂xj 4δ Applied Mathematical Methods Topics in Multivariate Calculus 173, Applied Mathematical Methods Topics in Multivariate Calculus 174, Derivatives in Multi-Dimensional Spaces Derivatives in Multi-Dimensional Spaces An Introduction to Tensors* Taylor’s Series Points to note Taylor’s Series Chain Rule and Change of Variables Chain Rule and Change of Variables Numerical Differentiation Numerical Differentiation An Introduction to Tensors* An Introduction to Tensors* I Indicial notation and summation convention I Kronecker delta and Levi-Civita symbol I Gradient, Hessian, Jacobian and the Taylor’s series I Rotation of reference axes I Partial and total gradients I Tensors of order zero, or scalars I Implicit functions I Contravariant and covariant tensors of order one, or vectors I Leibnitz rule I Cartesian tensors I Numerical derivatives I Cartesian tensors of order two I Higher order tensors I Elementary tensor operations Necessary Exercises: 2,3,4,8 I Symmetric tensors I Tensor fields I · · · · · · · · ·

Applied Mathematical Methods Vector Analysis: Curves and Surfaces 175, Applied Mathematical Methods Vector Analysis: Curves and Surfaces 176, Recapitulation of Basic Notions Recapitulation of Basic Notions Outline Curves in Space Recapitulation of Basic Notions Curves in Space Surfaces* Surfaces* Dot and cross products: their implications Scalar and vector triple products Differentiation rules Vector Analysis: Curves and Surfaces Interface with matrix algebra: Recapitulation of Basic Notions Curves in Space a x = aT x, · Surfaces* (a x)b = (baT )x, and · aT x, for 2-d vectors a x = ⊥ × ∼ax, for 3-d vectors  where

0 az ay ay − a = − and ∼a = az 0 ax ⊥ ax  −    ay ax 0 −   Applied Mathematical Methods Vector Analysis: Curves and Surfaces 177, Applied Mathematical Methods Vector Analysis: Curves and Surfaces 178, Recapitulation of Basic Notions Recapitulation of Basic Notions Curves in Space Curves in Space Curves in Space Curves in Space Surfaces* Surfaces* Explicit equation: y = y(x) and z = z(x) Curve r(t) is regular if r (t) = 0 t. 0 6 ∀ Implicit equation: F (x, y, z) = 0 = G(x, y, z) I Reparametrization with respect to parameter t ∗, some Parametric equation: strictly increasing function of t r(t) = x(t)i + y(t)j + z(t)k [x(t) y(t) z(t)]T ≡ Observations I Arc length s(t) is obviously a monotonically increasing I Tangent vector: r0(t) function. I ds Speed: r0 I For a regular curve, = 0. k k dt 6 I r0 I Unit tangent: u(t) = r Then, s(t) has an inverse function. k 0k I I Length of the curve: l = b dr = b r r dt Inverse t(s) reparametrizes the curve as r(t(s)). a k k a 0 · 0 Arc length function p R R For a unit speed curve r(s), r0(s) = 1 and the unit tangent is t k k s(t) = r (τ) r (τ) dτ 0 0 u(s) = r0(s). a · Z p with ds = dr = dx2 + dy 2 + dz2 and ds = r k k dt k 0k p

Applied Mathematical Methods Vector Analysis: Curves and Surfaces 179, Applied Mathematical Methods Vector Analysis: Curves and Surfaces 180, Recapitulation of Basic Notions Recapitulation of Basic Notions Curves in Space Curves in Space Curves in Space Curves in Space Surfaces* Surfaces* Curvature: The rate at which the direction changes with arc Binormal: b = u p length. × κ(s) = u0(s) = r00(s) Serret-Frenet frame: Right-handed triad u, p, b k k k k { } Unit principal normal: I Osculating, rectifying and normal planes 1 p = u0(s) Torsion: Twisting out of the osculating plane κ With general parametrization, I rate of change of b with respect to arc length s b = u p + u p = κ(s)p p + u p = u p d r0 du d r0 2 0 0 0 0 0 r00(t) = k ku(t) + r0(t) = k ku(t) + κ(t) r0 p(t) × × × × × dt k k dt dt k k What is p0?

r/ Taking p0 = σu + τb, AC = ρ = 1/κ u I C Osculating plane p A / / b0 = u (σu + τb) = τp. I Centre of curvature z r × − r I Radius of curvature Torsion of the curve O y

x τ(s) = p(s) b0(s) − · Figure: Tangent and normal to a curve Applied Mathematical Methods Vector Analysis: Curves and Surfaces 181, Applied Mathematical Methods Vector Analysis: Curves and Surfaces 182, Recapitulation of Basic Notions Recapitulation of Basic Notions Curves in Space Curves in Space Surfaces* Curves in Space Surfaces* Surfaces*

We have u0 and b0. What is p0? Parametric surface equation:

From p = b u, r(u, v) = x(u, v)i+y(u, v)j+z(u, v)k [x(u, v) y(u, v) z(u, v)]T × ≡

p0 = b0 u + b u0 = τp u + b κp = κu + τb. Tangent vectors ru and rv define a tangent plane . × × − × × − T

N = ru rv is normal to the surface and the unit normal is Serret-Frenet formulae × N ru rv u0 = κp, n = = × . p = κu + τb, N ru rv 0 −  k k k × k b = τp 0 −  Question: How does n vary over the surface?  Intrinsic representation of a curve is complete with κ(s) and τ(s). Information on local geometry: curvature tensor The arc-length parametrization of a curve is completely I Normal and principal curvatures determined by its curvature κ(s) and torsion τ(s) functions, except for a rigid body motion. I Local shape: convex, concave, saddle, cylindrical, planar

Applied Mathematical Methods Vector Analysis: Curves and Surfaces 183, Applied Mathematical Methods Scalar and Vector Fields 184, Recapitulation of Basic Notions Differential Operations on Field Functions Points to note Curves in Space Outline Integral Operations on Field Functions Surfaces* Integral Theorems Closure

I Parametric equation is the general and most convenient representation of curves and surfaces. I Arc length is the natural parameter and the Serret-Frenet Scalar and Vector Fields frame offers the natural frame of reference. Differential Operations on Field Functions I Curvature and torsion are the only inherent properties of a Integral Operations on Field Functions curve. Integral Theorems Closure I The local shape of a surface patch can be understood through an analysis of its curvature tensor.

Necessary Exercises: 1,2,3,6 Applied Mathematical Methods Scalar and Vector Fields 185, Applied Mathematical Methods Scalar and Vector Fields 186, Differential Operations on Field Functions Differential Operations on Field Functions Differential Operations on Field FunctionsIntegral Operations on Field Functions Differential Operations on Field FunctionsIntegral Operations on Field Functions Integral Theorems Integral Theorems Scalar point function or scalar field φ(x, y, zClosure): R 3 R Closure → Gradient Vector point function or vector field V(x, y, z): R 3 R3 → The del or nabla ( ) operator ∂φ ∂φ ∂φ ∇ grad φ φ = i + j + k ∂ ∂ ∂ ≡ ∇ ∂x ∂y ∂z i + j + k ∇ ≡ ∂x ∂y ∂z is orthogonal to the level surfaces.

I is a vector, Flow fields: φ gives the velocity vector. ∇ −∇ I it signifies a differentiation, and Divergence I it operates from the left side. Laplacian operator: For V(x, y, z) Vx (x, y, z)i + Vy (x, y, z)j + Vz (x, y, z)k, ≡ 2 2 2 2 ∂ ∂ ∂ ∂Vx ∂Vy ∂Vz 2 + 2 + 2 = ?? div V V = + + ∇ ≡ ∂x ∂y ∂z ∇ · ∇ ≡ ∇ · ∂x ∂y ∂z Laplace’s equation: Divergence of ρV: flow rate of mass per unit volume out of the ∂2φ ∂2φ ∂2φ + + = 0 control volume. ∂x2 ∂y 2 ∂z2 Similar relation between field and flux in electromagnetics. Solution of 2φ = 0: harmonic function ∇

Applied Mathematical Methods Scalar and Vector Fields 187, Applied Mathematical Methods Scalar and Vector Fields 188, Differential Operations on Field Functions Differential Operations on Field Functions Differential Operations on Field FunctionsIntegral Operations on Field Functions Differential Operations on Field FunctionsIntegral Operations on Field Functions Integral Theorems Integral Theorems Closure Closure Composite operations Curl Operator is linear. ∇ i j k (φ + ψ) = φ + ψ, curl V V = ∂ ∂ ∂ ∇ ∇ ∇ ≡ ∇ × ∂x ∂y ∂z (V + W) = V + W, and V V V ∇ · ∇ · ∇ · x y z (V + W) = V + W. ∇ × ∇ × ∇ × ∂Vz ∂ Vy ∂Vx ∂Vz ∂Vy ∂Vx = i + j + k ∂y − ∂z ∂z − ∂x ∂x − ∂y       Considering the products φψ, φV, V W, and V W; · × If V = ω r represents the velocity field, then angular velocity (φψ) = ψ φ + φ ψ × ∇ ∇ ∇ (φV) = φ V + φ V 1 ∇ · ∇ · ∇ · ω = curl V. (φV) = φ V + φ V ∇ × ∇ × ∇ × 2 (V W) = (W )V +(V )W +W ( V)+V ( W) ∇ · ·∇ ·∇ × ∇× × ∇× Curl represents rotationality. (V W) = W ( V) V ( W) ∇ · × · ∇ × − · ∇ × (V W) = (W )V W( V) (V )W + V( W) ∇ × × · ∇ − ∇ · − · ∇ ∇ · Connections between electric and magnetic fields! ∂ ∂ ∂ Note: the expression V Vx + Vy + Vz is an operator! · ∇ ≡ ∂x ∂y ∂z Applied Mathematical Methods Scalar and Vector Fields 189, Applied Mathematical Methods Scalar and Vector Fields 190, Differential Operations on Field Functions Differential Operations on Field Functions Differential Operations on Field FunctionsIntegral Operations on Field Functions Integral Operations on Field FunctionsIntegral Operations on Field Functions Integral Theorems Integral Theorems Second order differential operators Closure Line integral along curve C: Closure

I = V dr = (Vx dx + Vy dy + Vz dz) div grad φ ( φ) C · C ≡ ∇ · ∇ Z Z curl grad φ ( φ) For a parametrized curve r(t), t [a, b], ≡ ∇ × ∇ ∈ div curl V ( V) ≡ ∇ · ∇ × b dr curl curl V ( V) I = V dr = V dt. ≡ ∇ × ∇ × C · a · dt grad div V ( V) Z Z ≡ ∇ ∇ · For simple (non-intersecting) paths contained in a simply Important identities: connected region, equivalent statements: I div grad φ ( φ) = 2φ Vx dx + Vy dy + Vz dz is an exact differential. ≡ ∇ · ∇ ∇ I V = φ for some φ(r). curl grad φ ( φ) = 0 ∇ ≡ ∇ × ∇ I V dr is independent of path. div curl V ( V) = 0 C · ≡ ∇ · ∇ × I Circulation V dr = 0 around any closed path. curl curl V ( V) R · ≡ ∇ × ∇ × I curl V = 0. = ( V) 2V = grad div V 2V H ∇ ∇ · − ∇ − ∇ I Field V is conservative.

Applied Mathematical Methods Scalar and Vector Fields 191, Applied Mathematical Methods Scalar and Vector Fields 192, Differential Operations on Field Functions Differential Operations on Field Functions Integral Operations on Field FunctionsIntegral Operations on Field Functions Integral Theorems Integral Operations on Field Functions Integral Theorems Integral Theorems Closure Green’s theorem in the plane Closure Surface integral over an orientable surface S: R: closed bounded region in the xy-plane C: boundary, a piecewise smooth closed curve J = V dS = V ndS F1(x, y) and F2(x, y): first order continuous functions · · ZS Z ZS Z ∂F ∂F (F dx + F dy) = 2 1 dx dy For r(u, w), dS = ru rw du dw and 1 2 ∂x − ∂y k × k IC ZR Z   J = V ndS = V (r r ) du dw. u w y S · R · × y D Z Z Z Z d y (x) 2 R

x1 (y) B Volume integrals of point functions over a region T : R A x2 (y)

c y (x) M = φdv and F = Vdv C 1 O a x O x Z ZT Z Z ZT Z b (a) Simple domain (b) General domain

Figure: Regions for proof of Green’s theorem in the plane Applied Mathematical Methods Scalar and Vector Fields 193, Applied Mathematical Methods Scalar and Vector Fields 194, Differential Operations on Field Functions Differential Operations on Field Functions Integral Theorems Integral Operations on Field Functions Integral Theorems Integral Operations on Field Functions Integral Theorems Integral Theorems Closure Closure Proof: Gauss’s divergence theorem T : a closed bounded region ∂F b y2(x) ∂F 1 dxdy = 1 dydx S: boundary, a piecewise smooth closed orientable ∂y ∂y ZR Z Za Zy1(x) surface b F(x, y, z): a first order continuous vector function = [F x, y (x) F x, y (x) ]dx 1{ 2 } − 1{ 1 } Za a b div Fdv = F ndS = F x, y (x) dx F x, y (x) dx T S · − 1{ 2 } − 1{ 1 } Z Z Z Z Z Zb Za = F (x, y)dx Interpretation of the definition extended to finite domains. − 1 IC

∂F d x2(y) ∂F ∂Fx ∂Fy ∂Fz 2 dxdy = 2 dxdy = F (x, y)dy + + dx dy dz = (Fx nx +Fy ny +Fz nz )dS ∂x ∂x 2 T ∂x ∂y ∂z S ZR Z Zc Zx1(y) IC Z Z Z   Z Z ∂Fz ∂F2 ∂F1 To show: dx dy dz = Fz nz dS Difference: C (F1dx + F2dy) = R ∂x ∂y dx dy T ∂z S − First consider a region, the boundary of which is intersected at In alternativeH form, F dr = R R curl F k dx dy. R R R R R C · R · most twice by any line parallel to a coordinate axis. H R R

Applied Mathematical Methods Scalar and Vector Fields 195, Applied Mathematical Methods Scalar and Vector Fields 196, Differential Operations on Field Functions Differential Operations on Field Functions Integral Theorems Integral Operations on Field Functions Integral Theorems Integral Operations on Field Functions Integral Theorems Integral Theorems Closure Closure

Lower and upper segments of S: z = z1(x, y) and z = z2(x, y). Green’s identities (theorem) z ∂Fz 2 ∂Fz Region T and boundary S: as required in premises of dx dy dz = dz dx dy Gauss’s theorem T ∂z R z1 ∂z Z Z Z Z Z Z  φ(x, y, z) and ψ(x, y, z): second order continuous scalar = [Fz x, y, z2(x, y) Fz x, y, z1(x, y) ]dx dy functions { } − { } ZR Z R: projection of T on the xy-plane φ ψ ndS = (φ 2ψ + φ ψ)dv ∇ · ∇ ∇ · ∇ ZS Z Z ZT Z Projection of area element of the upper segment: nz dS = dx dy (φ ψ ψ φ) ndS = (φ 2ψ ψ 2φ)dv Projection of area element of the lower segment: nz dS = dx dy S ∇ − ∇ · T ∇ − ∇ − Z Z Z Z Z ∂Fz Thus, T ∂z dx dy dz = S Fz nz dS. Direct consequences of Gauss’s theorem R R R R R Sum of three such components leads to the result. To establish, apply Gauss’s divergence theorem on φ ψ, and then ∇ on ψ φ as well. Extension to arbitrary regions by a suitable subdivision of domain! ∇ Applied Mathematical Methods Scalar and Vector Fields 197, Applied Mathematical Methods Scalar and Vector Fields 198, Differential Operations on Field Functions Differential Operations on Field Functions Integral Theorems Integral Operations on Field Functions Integral Theorems Integral Operations on Field Functions Integral Theorems Integral Theorems Closure Closure Stokes’s theorem Represent S as z = z(x, y) f (x, y). S: a piecewise smooth surface ≡ C: boundary, a piecewise smooth simple closed curve T ∂f ∂f T Unit normal n = [nx ny nz ] is proportional to [ ∂x ∂y 1] . F(x, y, z): first order continuous vector function − ∂z ny = nz F dr = curl F ndS − ∂y · · IC ZS Z n: unit normal given by the right hand clasp rule on C ∂Fx ∂Fx ∂Fx ∂Fx ∂z ny nz dS = + nz dS S ∂z − ∂y − S ∂y ∂z ∂y For F(x, y, z) = Fx (x, y, z)i, Z Z   Z Z  

Over projection R of S on xy-plane, φ(x, y) = Fx (x, y, z(x, y)). ∂Fx ∂Fx ∂Fx ∂Fx Fx dx = j k ndS = ny nz dS. C S ∂z − ∂y · S ∂z − ∂y ∂φ I Z Z   Z Z   LHS = dx dy = φ(x, y)dx = Fx dx − ∂y ZR Z IC 0 IC First, consider a surface S intersected at most once by any line Similar results for Fy (x, y, z)j and Fz (x, y, z)k. parallel to a coordinate axis.

Applied Mathematical Methods Scalar and Vector Fields 199, Applied Mathematical Methods Polynomial Equations 200, Differential Operations on Field Functions Basic Principles Points to note Integral Operations on Field Functions Outline Analytical Solution Integral Theorems General Polynomial Equations Closure Two Simultaneous Equations Elimination Methods* Advanced Techniques*

I The ‘del’ operator ∇ I Gradient, divergence and curl Polynomial Equations I Composite and second order operators Basic Principles I Line, surface and volume intergals Analytical Solution General Polynomial Equations I Green’s, Gauss’s and Stokes’s theorems Two Simultaneous Equations I Applications in physics (and engineering) Elimination Methods* Advanced Techniques*

Necessary Exercises: 1,2,3,6,7 Applied Mathematical Methods Polynomial Equations 201, Applied Mathematical Methods Polynomial Equations 202, Basic Principles Basic Principles Basic Principles Analytical Solution Analytical Solution Analytical Solution General Polynomial Equations General Polynomial Equations Two Simultaneous Equations Two Simultaneous Equations Fundamental theorem of algebra Elimination Methods* Quadratic equation Elimination Methods* Advanced Techniques* Advanced Techniques* n n 1 n 2 2 p(x) = a0x + a1x − + a2x − + + an 1x + an 2 b √b 4ac · · · − ax + bx + c = 0 x = −  − ⇒ 2a has exactly n roots x1, x2, , xn; with · · · Method of completing the square: p(x) = a0(x x1)(x x2)(x x3) (x xn). − − − · · · − b b 2 b2 c b 2 b2 4ac x2 + x + = x + = − In general, roots are complex. a 2a 4a2 − a ⇒ 2a 4a2     Multiplicity: A root of p(x) with multiplicity k satisfies Cubic equations (Cardano): (k 1) p(x) = p0(x) = p00(x) = = p − (x) = 0. · · · x3 + ax2 + bx + c = 0 I Descartes’ rule of signs Completing the cube? I Bracketing and separation Substituting y = x + k, I Synthetic division and deflation y 3 + (a 3k)y 2 + (b 2ak + 3k 2)y + (c bk + ak 2 k3) = 0. − − − − p(x) = f (x)q(x) + r(x) Choose the shift k = a/3.

Applied Mathematical Methods Polynomial Equations 203, Applied Mathematical Methods Polynomial Equations 204, Basic Principles Basic Principles Analytical Solution Analytical Solution Analytical Solution Analytical Solution General Polynomial Equations General Polynomial Equations Two Simultaneous Equations Two Simultaneous Equations 3 Elimination Methods* Elimination Methods* y + py + q = 0 Advanced Techniques* Quartic equations (Ferrari) Advanced Techniques* 3 3 3 Assuming y = u + v, we have y = u + v + 3uv(u + v). a 2 a2 x4+ax3+bx2+cx+d = 0 x2 + x = b x2 cx d ⇒ 2 4 − − − uv = p/3   −   u3 + v 3 = q For a perfect square, − 4p3 and hence (u3 v 3)2 = q2 + . a y 2 a2 ay y 2 − 27 x2 + x + = b + y x2 + c x + d 2 2 4 − 2 − 4 − Solution:         Under what condition, the new RHS will be a perfect square? q q2 p3 u3, v 3 = + = A, B (say). − 2  4 27 ay 2 a2 y 2 r c 4 b + y d = 0 2 − − 4 − 4 − 2 2 u = A1, A1ω, A1ω , and v = B1, B1ω, B1ω       Resolvent of a quartic: y = A + B , y = A ω + B ω2 and y = A ω2 + B ω. 1 1 1 2 1 1 3 1 1 y 3 by 2 + (ac 4d)y + (4bd a2d c2) = 0 − − − − At least one of the roots is real!! Applied Mathematical Methods Polynomial Equations 205, Applied Mathematical Methods Polynomial Equations 206, Basic Principles Basic Principles Analytical Solution Analytical Solution General Polynomial Equations Analytical Solution General Polynomial Equations General Polynomial Equations Two Simultaneous Equations Two Simultaneous Equations Elimination Methods* Analytical solution of the general quintic equation?Elimination Methods* Advanced Techniques* Advanced Techniques* Procedure Galois: group theory: I Frame the cubic resolvent. A general quintic, or higher degree, equation is not I Solve this cubic equation. solvable by radicals. I Pick up one solution as y. General polynomial equations: iterative algorithms I Insert this y to form I Methods for nonlinear equations a y 2 I Methods specific to polynomial equations x2 + x + = (ex + f )2. 2 2 Solution through the companion matrix   I Split it into two quadratic equations as Roots of a polynomial equation are the same as the eigenvalues of its companion matrix. 2 a y x + x + = (ex + f ). 0 0 0 an 2 2  · · · − 1 0 0 an 1 I Solve each of the two quadratic equations to obtain a total of  . . ·.· · . − − .  Companion matrix: ...... four solutions of the original quartic equation.    0 0 0 a   · · · − 2   0 0 1 a   · · · − 1   

Applied Mathematical Methods Polynomial Equations 207, Applied Mathematical Methods Polynomial Equations 208, Basic Principles Basic Principles General Polynomial Equations Analytical Solution Two Simultaneous Equations Analytical Solution General Polynomial Equations General Polynomial Equations Two Simultaneous Equations Two Simultaneous Equations Elimination Methods* Elimination Methods* Advanced Techniques* Advanced Techniques* Bairstow’s method 2 2 p1x + q1xy + r1y + u1x + v1y + w1 = 0 to separate out factors of small degree. 2 2 p2x + q2xy + r2y + u2x + v2y + w2 = 0 Attempt to separate real linear factors? Rearranging, a x2 + b x + c = 0 Real quadratic factors 1 1 1 2 a2x + b2x + c2 = 0 2 Synthetic division with a guess factor x + q1x + q2: Cramer’s rule: remainder r x + r x2 x 1 1 2 = − = T T b1c2 b2c1 a1c2 a2c1 a1b2 a2b1 r = [r1 r2] is a vector function of q = [q1 q2] . − − − b c b c a c a c x = 1 2 − 2 1 = 1 2 − 2 1 ⇒ − a1c2 a2c1 −a1b2 a2b1 Iterate over (q1, q2) to make (r1, r2) zero. − − Consistency condition: Newton-Raphson (Jacobian based) iteration: see exercise. (a b a b )(b c b c ) (a c a c )2 = 0 1 2 − 2 1 1 2 − 2 1 − 1 2 − 2 1 A 4th degree equation in y Applied Mathematical Methods Polynomial Equations 209, Applied Mathematical Methods Polynomial Equations 210, Basic Principles Basic Principles Elimination Methods* Analytical Solution Advanced Techniques* Analytical Solution General Polynomial Equations General Polynomial Equations Two Simultaneous Equations Two Simultaneous Equations Elimination Methods* Three or more independent equations in as manyEliminationunknoMethods*wns? The method operates similarly even if the degreesAdvanced Tofechniques*the original Advanced Techniques* equations in y are higher. I Cascaded elimination? Objections! I Exploitation of special structures through clever heuristics What about the degree of the eliminant equation? (mechanisms kinematics literature)

Two equations in x and y of degrees n1 and n2: I Gr¨obner basis representation x-eliminant is an equation of degree n1n2 in y (algebraic geometry) Maximum number of solutions: I Bezout number = n1n2 Continuation or homotopy method by Morgan Note: Deficient systems may have less number of solutions. For solving the system f(x) = 0, identify another structurally similar system g(x) = 0 with known solutions and construct the parametrized system Classical methods of elimination I Sylvester’s dialytic method h(x) = tf(x) + (1 t)g(x) = 0 for t [0, 1]. − ∈ I Bezout’s method Track each solution from t = 0 to t = 1.

Applied Mathematical Methods Polynomial Equations 211, Applied Mathematical Methods Solution of Nonlinear Equations and Systems 212, Basic Principles Methods for Nonlinear Equations Points to note Analytical Solution Outline Systems of Nonlinear Equations General Polynomial Equations Closure Two Simultaneous Equations Elimination Methods* Advanced Techniques* I Roots of cubic and quartic polynomials by the methods of Cardano and Ferrari I For higher degree polynomials, Solution of Nonlinear Equations and Systems I Bairstow’s method: a clever implementation of Methods for Nonlinear Equations Newton-Raphson method for polynomials Systems of Nonlinear Equations I Eigenvalue problem of a companion matrix Closure I Reduction of a system of polynomial equations in two unknowns by elimination

Necessary Exercises: 1,3,4,6 Applied Mathematical Methods Solution of Nonlinear Equations and Systems 213, Applied Mathematical Methods Solution of Nonlinear Equations and Systems 214, Methods for Nonlinear Equations Methods for Nonlinear Equations Methods for Nonlinear Equations Systems of Nonlinear Equations Methods for Nonlinear Equations Systems of Nonlinear Equations Closure Closure w y = x Fixed point iteration y y = g(x) u v Algebraic and transcendental equations in the form Rearrange f (x) = 0 in the form x = g(x). m f (x) = 0 l Example: Practical problem: to find one real root (zero) of f (x) For f (x) = tan x x 3 2, − − b a n possible rearrangements: 1 3 g (x) = tan (x + 2) d c Example of f (x): x 3 2x + 5, x 3 ln x sin x + 2, etc. 1 − − − g (x) = (tan x 2)1/3 2 f tan x 2− e g3(x) = x2− g If f (x) is continuous, then Iteration: xk+1 = g(xk ) O x p q x r x Bracketing: f (x )f (x ) < 0 there must be a root of f (x) 0 1 ⇒ between x0 and x1. Figure: Fixed point iteration x0+x1 Bisection: Check the sign of f ( 2 ). Replace either x0 or x1 x0+x1 with 2 . If x∗ is the unique solution in interval J and g (x) h < 1 in J, then any x J converges to x . | 0 | ≤ 0 ∈ ∗

Applied Mathematical Methods Solution of Nonlinear Equations and Systems 215, Applied Mathematical Methods Solution of Nonlinear Equations and Systems 216, Methods for Nonlinear Equations Methods for Nonlinear Equations Methods for Nonlinear Equations Systems of Nonlinear Equations Methods for Nonlinear Equations Systems of Nonlinear Equations Closure Closure Newton-Raphson method Secant method and method of false position First order Taylor series f(x) f(x) a In the Newton-Raphson formula, f(x ) f (x + δx) f (x) + f 0(x)δx 0 ≈ f (xk ) f (xk 1) f 0(x) − − From f (xk + δx) = 0, xk xk 1 ≈ − − δx = f (xk )/f 0(xk ) e − xk xk 1 xk+1 = xk − − f (xk ) Iteration: f (xk ) f (xk 1) ⇒ − − − xk = xk f (xk )/f (xk ) +1 − 0 Draw the chord or x1 x2 x3 x* Convergence criterion: O x0 x 2 b f x* secant to f (x) through O f (x)f 00(x) < f 0(x) g d x0 x | | | | (xk 1, f (xk 1)) and (xk , f (xk )). f(x1 ) − − Draw tangent to f (x). c Take its x-intercept. Take its x-intercept. Figure: Method of false position Figure: Newton-Raphson method Special case: Maintain a bracket over the root at every iteration. 2 Merit: quadratic speed of convergence: xk x = c xk x | +1 − ∗| | − ∗| The method of false position or regula falsi Demerit: If the starting point is not appropriate, haphazard wandering, oscillations or outright divergence! Convergence is guaranteed! Applied Mathematical Methods Solution of Nonlinear Equations and Systems 217, Applied Mathematical Methods Solution of Nonlinear Equations and Systems 218, Methods for Nonlinear Equations Methods for Nonlinear Equations Methods for Nonlinear Equations Systems of Nonlinear Equations Systems of Nonlinear Equations Systems of Nonlinear Equations Closure Closure Quadratic interpolation method or Muller method

Evaluate f (x) at three points y f1(x1, x2, , xn) = 0, and model y = a + bx + cx 2. · · · f (x , x , , xn) = 0, Set y = 0 and solve for x. 2 1 2 · · ·

(x0 ,y0 ) Quadratic · · · · · · · · · · · · Interpolation fn(x1, x2, , xn) = 0. Inverse quadratic interpolation Inverse · · · Quadratic Evaluate f (x) at three points Interpolation f(x) = 0 (x ,y ) 2 1 1 x3 and model x = a + by + cy . O I x3 x Number of variables and number of equations? (x2 ,y2 ) Set y = 0 to get x = a. I No bracketing! I Figure: Interpolation schemes Fixed point iteration schemes x = g(x)? Newton’s method for systems of equations Van Wijngaarden-Dekker Brent method ∂f I maintains the bracket, f(x + δx) = f(x) + (x) δx + f(x) + J(x)δx ∂x · · · ≈ I uses inverse quadratic interpolation, and   I 1 accepts outcome if within bounds, else takes a bisection step. xk = xk [J(xk )]− f(xk ) ⇒ +1 − Opportunistic manoeuvring between a fast method and a safe one! with the usual merits and demerits!

Applied Mathematical Methods Solution of Nonlinear Equations and Systems 219, Applied Mathematical Methods Solution of Nonlinear Equations and Systems 220, Methods for Nonlinear Equations Methods for Nonlinear Equations Closure Systems of Nonlinear Equations Points to note Systems of Nonlinear Equations Closure Closure Modified Newton’s method

1 I Iteration schemes for solving f (x) = 0 xk+1 = xk αk [J(xk )]− f(xk ) − I Newton (or Newton-Raphson) iteration for a system of Broyden’s secant method equations 1 xk = xk [J(xk )]− f(xk ) Jacobian is not evaluated at every iteration, but gets +1 − developed through updates. I Optimization formulation of a multi-dimensional root finding problem Optimization-based formulation

Global minimum of the function Necessary Exercises: 1,2,3 f(x) 2 = f 2 + f 2 + + f 2 k k 1 2 · · · n

Levenberg-Marquardt method Applied Mathematical Methods Optimization: Introduction 221, Applied Mathematical Methods Optimization: Introduction 222, The Methodology of Optimization The Methodology of Optimization Outline Single-Variable Optimization The Methodology of Optimization Single-Variable Optimization Conceptual Background of Multivariate Optimization Conceptual Background of Multivariate Optimization

I Parameters and variables I The statement of the optimization problem Optimization: Introduction Minimize f (x) The Methodology of Optimization subject to g(x) 0, Single-Variable Optimization ≤ h(x) = 0. Conceptual Background of Multivariate Optimization I Optimization methods I Sensitivity analysis I Optimization problems: unconstrained and constrained I Optimization problems: linear and nonlinear I Single-variable and multi-variable problems

Applied Mathematical Methods Optimization: Introduction 223, Applied Mathematical Methods Optimization: Introduction 224, The Methodology of Optimization The Methodology of Optimization Single-Variable Optimization Single-Variable Optimization Single-Variable Optimization Single-Variable Optimization Conceptual Background of Multivariate Optimization Conceptual Background of Multivariate Optimization For a function f (x), a point x ∗ is defined as a relative (local) minimum if  such that f (x) f (x ) x [x , x + ]. Higher order analysis: From Taylor’s series, ∃ ≥ ∗ ∀ ∈ ∗ − ∗ f( x ) ∆f = f (x∗ + δx) f (x ∗) − 1 2 1 3 1 iv 4 = f 0(x∗)δx + f 00(x∗)δx + f 000(x∗)δx + f (x∗)δx + 2! 3! 4! · · ·

For an extremum to occur at point x ∗, the lowest order

O a x x x x x x x 1 2 3 4 5 6 b derivative with non-zero value should be of even order. Figure: Schematic of optima of a univariate function If f 0(x∗) = 0, then I Optimality criteria x∗ is a stationary point, a candidate for an extremum. I First order necessary condition: If x ∗ is a local minimum or Evaluate higher order derivatives till one of them is found to be non-zero. maximum point and if f 0(x∗) exists, then f 0(x∗) = 0. I If its order is odd, then x ∗ is an inflection point. Second order necessary condition: If x ∗ is a local minimum point I and f (x ) exists, then f (x ) 0. If its order is even, then x ∗ is a local minimum or maximum, 00 ∗ 00 ∗ ≥ as the derivative value is positive or negative, respectively. Second order sufficient condition: If f 0(x∗) = 0 and f 00(x∗) > 0 then x∗ is a local minimum point. Applied Mathematical Methods Optimization: Introduction 225, Applied Mathematical Methods Optimization: Introduction 226, The Methodology of Optimization The Methodology of Optimization Single-Variable Optimization Single-Variable Optimization Single-Variable Optimization Single-Variable Optimization Conceptual Background of Multivariate Optimization Conceptual Background of Multivariate Optimization Iterative methods of line search Bracketing: Methods based on gradient root finding x1 < x2 < x3 with f (x1) f (x2) f (x3) I Newton’s method ≥ ≤ Exhaustive search method or its variants f 0(xk ) xk+1 = xk Direct optimization algorithms − f (xk ) 00 I Fibonacci search uses a pre-defined number N, of function I Secant method evaluations, and the Fibonacci sequence xk xk 1 xk+1 = xk − − f 0(xk ) F0 = 1, F1 = 1, F2 = 2, , Fj = Fj 2 + Fj 1, − f 0(xk ) f 0(xk 1) · · · − − · · · − − I Method of cubic estimation to tighten a bracket with economized number of function point of vanishing gradient of the cubic fit with evaluations. I Golden section search uses a constant ratio f (xk 1), f (xk ), f 0(xk 1) and f 0(xk ) − − √5 1 I Method of quadratic estimation τ = − 0.618, 2 ≈ point of vanishing gradient of the quadratic fit the golden section ratio, of interval reduction, that is through three points determined as the limiting case of N and the actual → ∞ Disadvantage: treating all stationary points alike! number of steps is decided by the accuracy desired.

Applied Mathematical Methods Optimization: Introduction 227, Applied Mathematical Methods Optimization: Introduction 228, The Methodology of Optimization The Methodology of Optimization Conceptual Background of MultivariateSingle-VOptimizationariable Optimization Conceptual Background of MultivariateSingle-VOptimizationariable Optimization Conceptual Background of Multivariate Optimization Conceptual Background of Multivariate Optimization Convexity Unconstrained minimization problem Set S Rn is a convex set if ⊆ x is called a local minimum of f (x) if δ such that ∗ ∃ x1, x2 S and α (0, 1), αx1 + (1 α)x2 S. f (x) f (x ) for all x satisfying x x < δ. ∀ ∈ ∈ − ∈ ≥ ∗ k − ∗k Function f (x) over a convex set S: a convex function if x , x S and α (0, 1), ∀ 1 2 ∈ ∈ Optimality criteria f (αx1 + (1 α)x2) αf (x1) + (1 α)f (x2). From Taylor’s series, − ≤ − Chord approximation is an overestimate at intermediate points! T 1 T f (x) f (x∗) = [g(x∗)] δx + δx [H(x∗)]δx + . − 2 · · · x2 f(x )

For x∗ to be a local minimum, f(x 2) X2 f(x ) necessary condition: g(x∗) = 0 and H(x∗) is positive semi-definite, X1 1 sufficient condition: g(x ) = 0 and H(x ) is positive definite. ∗ ∗ O x1 O x1 x2 x

Indefinite Hessian matrix characterizes a saddle point. Figure: A convex domain Figure: A convex function Applied Mathematical Methods Optimization: Introduction 229, Applied Mathematical Methods Optimization: Introduction 230, The Methodology of Optimization The Methodology of Optimization Conceptual Background of MultivariateSingle-VOptimizationariable Optimization Conceptual Background of MultivariateSingle-VOptimizationariable Optimization Conceptual Background of Multivariate Optimization Conceptual Background of Multivariate Optimization

First order characterization of convexity Quadratic function

From f (αx + (1 α)x ) αf (x ) + (1 α)f (x ), 1 1 − 2 ≤ 1 − 2 q(x) = xT Ax + bT x + c 2 f (x + α(x x )) f (x ) f (x ) f (x ) 2 1 − 2 − 2 . Gradient q(x) = Ax + b and Hessian = A is constant. 1 − 2 ≥ α ∇

T As α 0, f (x1) f (x2) + [ f (x2)] (x1 x2). I If A is positive definite, then the unique solution of Ax = b → ≥ ∇ − − is the only minimum point. Tangent approximation is an underestimate at intermediate points! I If A is positive semi-definite and b Range(A), then the − ∈ Second order characterization: Hessian is positive semi-definite. entire subspace of solutions of Ax = b are global minima. − I If A is positive semi-definite but b / Range(A), then the − ∈ Convex programming problem: convex function over convex set function is unbounded! A local minimum is also a global minimum, and all minima are connected in a convex set. Note: A quadratic problem (with positive definite Hessian) acts as Note: Convexity is a stronger condition than unimodality! a benchmark for optimization algorithms.

Applied Mathematical Methods Optimization: Introduction 231, Applied Mathematical Methods Optimization: Introduction 232, The Methodology of Optimization The Methodology of Optimization Conceptual Background of MultivariateSingle-VOptimizationariable Optimization Conceptual Background of MultivariateSingle-VOptimizationariable Optimization Conceptual Background of Multivariate Optimization Conceptual Background of Multivariate Optimization Optimization Algorithms Convergence of algorithms: notions of guarantee and speed From the current point, move to another point, hopefully better. Global convergence: the ability of an algorithm to approach and converge to an optimal solution for an arbitrary Which way to go? How far to go? Which decision is first? problem, starting from an arbitrary point I Strategies and versions of algorithms: Practically, a sequence (or even subsequence) of monotonically decreasing errors is enough. Trust Region: Develop a local quadratic model Local convergence: the rate/speed of approach, measured by p, T 1 T f (xk + δx) = f (xk ) + [g(xk )] δx + δx Fk δx, where 2 xk+1 x∗ and minimize it in a small trust region around xk . β = lim k − pk < k xk x∗ ∞ (Define trust region with dummy boundaries.) →∞ k − k Line search: Identify a descent direction d and minimize the k I Linear, quadratic and superlinear rates of function along it through the univariate function convergence for p = 1, 2 and intermediate. φ(α) = f (xk + αdk ). I Comparison among algorithms with linear rates I Exact or accurate line search of convergence is by the convergence ratio β. I Inexact or inaccurate line search I Armijo, Goldstein and Wolfe conditions Applied Mathematical Methods Optimization: Introduction 233, Applied Mathematical Methods Multivariate Optimization 234, The Methodology of Optimization Direct Methods Points to note Single-Variable Optimization Outline Steepest Descent (Cauchy) Method Conceptual Background of Multivariate Optimization Newton’s Method Hybrid (Levenberg-Marquardt) Method Least Square Problems

I Theory and methods of single-variable optimization I Optimality criteria in multivariate optimization Multivariate Optimization I Convexity in optimization Direct Methods I The quadratic function Steepest Descent (Cauchy) Method I Trust region Newton’s Method I Line search Hybrid (Levenberg-Marquardt) Method Least Square Problems I Global and local convergence

Necessary Exercises: 1,2,5,7,8

Applied Mathematical Methods Multivariate Optimization 235, Applied Mathematical Methods Multivariate Optimization 236, Direct Methods Direct Methods Direct Methods Steepest Descent (Cauchy) Method Direct Methods Steepest Descent (Cauchy) Method Newton’s Method Newton’s Method Hybrid (Levenberg-Marquardt) Method Hybrid (Levenberg-Marquardt) Method Least Square Problems Nelder and Mead’s simplex method Least Square Problems Direct search methods using only function values Simplex in n-dimensional space: polytope formed by n + 1 vertices I Cyclic coordinate search Nelder and Mead’s method iterates over simplices that are I Rosenbrock’s method non-degenerate (i.e. enclosing non-zero hypervolume). I Hooke-Jeeves pattern search First, n + 1 suitable points are selected for the starting simplex. I Box’s complex method Among vertices of the current simplex, identify the worst point xw , I Nelder and Mead’s simplex search the best point xb and the second worst point xs . I Powell’s conjugate directions method Need to replace xw with a good point. Useful for functions, for which derivative either does not exist at all points in the domain or is computationally costly to evaluate. Centre of gravity of the face not containing xw :

n+1 1 Note: When derivatives are easily available, gradient-based x = x c n i algorithms appear as mainstream methods. i=1,i=w X6

Reflect xw with respect to xc as xr = 2xc xw . Consider options. − Applied Mathematical Methods Multivariate Optimization 237, Applied Mathematical Methods Multivariate Optimization 238, Direct Methods Direct Methods Direct Methods Steepest Descent (Cauchy) Method Steepest Descent (Cauchy) Method Steepest Descent (Cauchy) Method Newton’s Method Newton’s Method Hybrid (Levenberg-Marquardt) Method Hybrid (Levenberg-Marquardt) Method Default xnew = xr . Least Square Problems Least Square Problems Revision possibilities: From a point xk , a move through α units in direction dk :

T 2 f(x b ) f(x s ) f(x w ) f (xk + αdk ) = f (xk ) + α[g(xk )] dk + (α ) Expansion Default Positive Negative O Contraction Contraction T xnew Descent direction dk : For α > 0, [g(xk )] dk < 0 xr xw xnew xw xr = x new xw xr xw xr Direction of steepest descent: dk = gk [ or dk = gk / gk ] − − k k Figure: Nelder and Mead’s simplex method Minimize φ(α) = f (xk + αdk ).

1. For f (xr ) < f (xb), expansion: Exact line search: xnew = xc + α(xc xw ), α > 1. − T 2. For f (xr ) f (xw ), negative contraction: φ0(αk ) = [g(xk + αk dk )] dk = 0 ≥ xnew = xc β(xc xw ), 0 < β < 1. − − Search direction tangential to the contour surface at (xk + αk dk ). 3. For f (xs ) < f (xr ) < f (xw ), positive contraction:

xnew = xc + β(xc xw ), with 0 < β < 1. Note: Next direction dk = g(xk ) orthogonal to dk − +1 − +1 Replace xw with xnew . Continue with new simplex.

Applied Mathematical Methods Multivariate Optimization 239, Applied Mathematical Methods Multivariate Optimization 240, Direct Methods Direct Methods Steepest Descent (Cauchy) Method Steepest Descent (Cauchy) Method Steepest Descent (Cauchy) Method Steepest Descent (Cauchy) Method Newton’s Method Newton’s Method Hybrid (Levenberg-Marquardt) Method Hybrid (Levenberg-Marquardt) Method Least Square Problems Least Square Problems Analysis on a quadratic function Steepest descent algorithm 1 T T 1. Select a starting point x0, set k = 0 and several parameters: For minimizing q(x) = 2 x Ax + b x, the error function: tolerance G on gradient, absolute tolerance A on reduction 1 T in function value, relative tolerance R on reduction in E(x) = (x x∗) A(x x∗) function value and maximum number of iterations M. 2 − − 2 2. If gk G , STOP. Else dk = gk / gk . E(xk+1) κ(A) 1 k k ≤ − k k Convergence ratio: − E(xk ) ≤ κ(A)+1 3. Line search: Obtain αk by minimizing φ(α) = f (xk + αdk ), Local convergence is poor.  α > 0. Update xk+1 = xk + αk dk .

4. If f (xk+1) f (xk ) A + R f (xk ) ,STOP. Else k k + 1. | − | ≤ | | ← Importance of steepest descent method 5. If k > M, STOP. Else go to step 2. I conceptual understanding I Very good global convergence. initial iterations in a completely new problem I spacer steps in other sophisticated methods But, why so many “STOPS”? Re-scaling of the problem through change of variables? Applied Mathematical Methods Multivariate Optimization 241, Applied Mathematical Methods Multivariate Optimization 242, Direct Methods Direct Methods Newton’s Method Steepest Descent (Cauchy) Method Newton’s Method Steepest Descent (Cauchy) Method Newton’s Method Newton’s Method Hybrid (Levenberg-Marquardt) Method Hybrid (Levenberg-Marquardt) Method Second order approximation of a function: Least Square Problems Least Square Problems

T 1 T Modified Newton’s method f (x) f (xk ) + [g(xk )] (x xk ) + (x xk ) H(xk )(x xk ) ≈ − 2 − − I Replace the Hessian by Fk = H(xk ) + γI . Vanishing of gradient I Replace full Newton’s step by a line search.

g(x) g(xk ) + H(xk )(x xk ) ≈ − gives the iteration formula Algorithm

1 1. Select x0, tolerance  and δ > 0. Set k = 0. xk+1 = xk [H(xk )]− g(xk ). − 2. Evaluate gk = g(xk ) and H(xk ). Excellent local convergence property! Choose γ, find Fk = H(xk ) + γI , solve Fk dk = gk for dk . − 3. Line search: obtain αk to minimize φ(α) = f (xk + αdk ). xk+1 x∗ k − 2k β Update xk+1 = xk + αk dk . xk x ≤ k − ∗k 4. Check convergence: If f (xk ) f (xk ) < , STOP. | +1 − | Caution: Does not have global convergence. Else, k k + 1 and go to step 2. 1 ← If H(xk ) is positive definite then dk = [H(xk )] g(xk ) − − is a descent direction.

Applied Mathematical Methods Multivariate Optimization 243, Applied Mathematical Methods Multivariate Optimization 244, Direct Methods Direct Methods Hybrid (Levenberg-Marquardt) MethoSteepd est Descent (Cauchy) Method Least Square Problems Steepest Descent (Cauchy) Method Newton’s Method Newton’s Method Hybrid (Levenberg-Marquardt) Method Hybrid (Levenberg-Marquardt) Method Methods of deflected gradients Least Square Problems Least Square Problems Linear least square problem:

xk+1 = xk αk [Mk ]gk − y(θ) = x φ (θ) + x φ (θ) + + xnφn(θ) 1 1 2 2 · · · I identity matrix in place of Mk : steepest descent step For measured values y(θi ) = yi , I 1 Mk = Fk− : step of modified Newton’s method n I 1 T Mk = [H(xk )]− and αk = 1: pure Newton’s step ei = xk φk (θi ) yi = [Φ(θi )] x yi . − − 1 k=1 In Mk = [H(xk ) + λk I ]− , tune parameter λk over iterations. X I Error vector: e = Ax y Initial value of λ: large enough to favour steepest descent − trend Last square fit: I Improvement in an iteration: λ reduced by a factor 1 2 1 T I Increase in function value: step rejected and λ increased Minimize E = 2 i ei = 2 e e Opportunism systematized! P Note: Cost of evaluating the Hessian remains a bottleneck. Pseudoinverse solution and its variants Useful for problems where Hessian estimates come cheap! Applied Mathematical Methods Multivariate Optimization 245, Applied Mathematical Methods Multivariate Optimization 246, Direct Methods Direct Methods Least Square Problems Steepest Descent (Cauchy) Method Least Square Problems Steepest Descent (Cauchy) Method Newton’s Method Newton’s Method Hybrid (Levenberg-Marquardt) Method Hybrid (Levenberg-Marquardt) Method Nonlinear least square problem Least Square Problems Least Square Problems For model function in the form Levenberg-Marquardt algorithm 1. Select x0, evaluate E(x0). Select tolerance , initial λ and its y(θ) = f (θ, x) = f (θ, x , x , , xn), 1 2 · · · update factor. Set k = 0. ¯ T T square error function 2. Evaluate gk and Hk = J J + λ diag(J J). Solve H¯ k δx = gk . Evaluate E(xk + δx). − 1 T 1 2 1 2 E(x) = e e = e = [f (θi , x) yi ] 3. If E(xk + δx) E(xk ) < , STOP. 2 2 i 2 − | − | i i 4. If E(x + δx) < E(x ), then decrease λ, X X k k T update xk+1 = xk + δx, k k + 1. Gradient: g(x) = E(x) = [f (θi , x) yi ] f (θi , x) = J e ← ∇ i − ∇ Else increase λ. ∂2 T ∂2 T Hessian: H(x) = 2 E(x) =PJ J + ei 2 f (θi , x) J J 5. Go to step 2. ∂x i ∂x ≈ Combining a modified form λ diag(JPT J) δx = g(x) of steepest Professional procedure for nonlinear least square problems and also − descent formula with Newton’s formula, for solving systems of nonlinear equations in the form h(x) = 0. Levenberg-Marquardt step: [JT J + λ diag(JT J)]δx = g(x) −

Applied Mathematical Methods Multivariate Optimization 247, Applied Mathematical Methods Methods of Nonlinear Optimization* 248, Direct Methods Conjugate Direction Methods Points to note Steepest Descent (Cauchy) Method Outline Quasi-Newton Methods Newton’s Method Closure Hybrid (Levenberg-Marquardt) Method Least Square Problems

I Simplex method of Nelder and Mead I Steepest descent method with its global convergence Methods of Nonlinear Optimization* I Newton’s method for fast local convergence Conjugate Direction Methods I Levenberg-Marquardt method for equation solving and least Quasi-Newton Methods squares Closure

Necessary Exercises: 1,2,3,4,5,6 Applied Mathematical Methods Methods of Nonlinear Optimization* 249, Applied Mathematical Methods Methods of Nonlinear Optimization* 250, Conjugate Direction Methods Conjugate Direction Methods Conjugate Direction Methods Quasi-Newton Methods Conjugate Direction Methods Quasi-Newton Methods Closure Closure Conjugacy of directions: Question: How to find a set of n conjugate directions? Two vectors d1 and d2 are mutually conjugate with T respect to a symmetric matrix A, if d1 Ad2 = 0. Gram-Schmidt procedure is a poor option! Linear independence of conjugate directions: Conjugate gradient method Conjugate directions with respect to a positive definite Starting from d = g , matrix are linearly independent. 0 − 0

n dk = gk + βk dk Expanding subspace property: In R , with conjugate vectors +1 − +1 d0, d1, , dn 1 with respect to symmetric positive definite A, { · · · n− } Imposing the condition of conjugacy of dk+1 with dk , for any x R , the sequence x , x , x , , xn generated as 0 ∈ { 0 1 2 · · · } T T T gk dk gk+1Adk gk+1(gk+1 gk ) xk+1 = xk + αk dk , with αk = T , βk = T = T − −dk Adk dk Adk αk dk Adk where gk = Axk + b, has the property that Resulting d conjugate to all the earlier directions, for x minimizes q(x) = 1 xT Ax + bT x on the line k+1 k 2 a quadratic problem. xk 1 + αdk 1, as well as on the linear variety x0 + k , − − B where k is the span of d0, d1, , dk 1. B · · · −

Applied Mathematical Methods Methods of Nonlinear Optimization* 251, Applied Mathematical Methods Methods of Nonlinear Optimization* 252, Conjugate Direction Methods Conjugate Direction Methods Conjugate Direction Methods Quasi-Newton Methods Conjugate Direction Methods Quasi-Newton Methods Closure Closure Using k in place of k + 1 in the formula for dk+1, Extension to general (non-quadratic) functions I Varying Hessian A: determine the step size by line search. dk = gk + βk 1dk 1 − − − I After n steps, minimum not attained. T T T T T gk gk But, gk dk = gk gk implies guaranteed descent. gk dk = gk gk and αk = T − ⇒ − dk Adk Globally convergent, with superlinear rate of convergence. I Polak-Ribiere formula: What to do after n steps? Restart or continue? T Algorithm gk+1(gk+1 gk ) βk = T − 1. Select x0 and tolerances G , D . Evaluate g0 = f (x0). g gk ∇ k 2. Set k = 0 and dk = gk . − No need to know A! 3. Line search: find αk ; update xk+1 = xk + αk dk . Further, 4. Evaluate gk+1 = f (xk+1). If gk+1 G , STOP. T ∇ k k ≤ gk+1(gk+1 gk ) T T T T 5. Find βk = T − (Polak-Ribiere) gk+1dk = 0 gk+1gk = βk 1(gk + αk dk A)dk 1 = 0. gk gk ⇒ − − T gk+1gk+1 or βk = T (Fletcher-Reeves). Fletcher-Reeves formula: gk gk T Obtain dk+1 = gk+1 + βk dk . gk+1gk+1 T − β = dk dk+1 k T 6. If 1 < D, reset g0 = gk+1and go to step 2. g g dk dk+1 k k − k k k k Else, k k + 1 and go to step 3. ←

Applied Mathematical Methods Methods of Nonlinear Optimization* 253, Applied Mathematical Methods Methods of Nonlinear Optimization* 254, Conjugate Direction Methods Conjugate Direction Methods Conjugate Direction Methods Quasi-Newton Methods Conjugate Direction Methods Quasi-Newton Methods Closure Closure Powell’s conjugate direction method For q(x) = 1 xT Ax + bT x, suppose 2 Algoithm T x1 = xA + α1d such that d g1 = 0 and T 1. Select x0,  and a set of n linearly independent (preferably x2 = xB + α2d such that d g2 = 0. normalized) directions d , d , , dn; possibly di = ei . 1 2 · · · T T Then, d A(x x ) = d (g g ) = 0. 2. Line search along dn and update x1 = x0 + αdn; set k = 1. 2 − 1 2 − 1 n 3. Line searches along d1, d2, , dn in sequence to obtain Parallel subspace property: In R , consider two parallel n · · · z = x + α d . linear varieties = v + k and = v + k , with k j=1 j j S1 1 B S2 2 B k = d , d , , dk , k < n. 4. New conjugate direction d = z xk . If d < , STOP. B { 1 2 · · · } P − k k If x1 and x2 5. Reassign directions dj dj+1 for j = 1, 2, , (n 1) and 1 T T ← · · · − minimize q(x) = 2 x Ax+b x on 1 and 2, respectively, dn = d/ d . S S k k then x x is conjugate to d , d , , dk . (Old d1 gets discarded at this step.) 2 − 1 1 2 · · · 6. Line search and update xk = z + αdn; set k k + 1 and +1 ← Assumptions imply g1, g2 k and hence go to step 3. ⊥ B T T (g g ) k d A(x x ) = d (g g ) = 0 for i = 1, 2, , k. 2− 1 ⊥ B ⇒ i 2− 1 i 2− 1 · · ·

Applied Mathematical Methods Methods of Nonlinear Optimization* 255, Applied Mathematical Methods Methods of Nonlinear Optimization* 256, Conjugate Direction Methods Conjugate Direction Methods Conjugate Direction Methods Quasi-Newton Methods Quasi-Newton Methods Quasi-Newton Methods Closure Closure I Variable metric methods x0-x1 and b-z1: x1-z1 is conjugate to b-z1. I b-z1-x2 and c-d-z2: c-d, d-z2 and x2-z2 are mutually attempt to construct the inverse Hessian Bk . conjugate. pk = xk xk and qk = gk gk qk Hpk +1 − +1 − ⇒ ≈ x3 x3 1 1 With n such steps, B = PQ− : update and construct Bk H− . z T ≈ d 2 Rank one correction: Bk+1 = Bk + ak zk zk ? Rank two correction: x 2 c T T Bk+1 = Bk + ak zk zk + bk wk wk x2 z x1 1 Davidon-Fletcher-Powell (DFP) method Select x , tolerance  and B = In. For k = 0, 1, 2, , x1 a b 0 0 I · · · x0 dk = Bk gk . I − Line search for αk ; update pk = αk dk , xk+1 = xk + pk , Figure: Schematic of Powell’s conjugate direction method qk = gk+1 gk . I − If pk <  or qk < , STOP. k k k k T T Performance of Powell’s method approaches that of the I DFP pk pk Bk qk qk Bk Rank two correction: Bk+1 = Bk + T T . conjugate gradient method! pk qk − qk Bk qk Applied Mathematical Methods Methods of Nonlinear Optimization* 257, Applied Mathematical Methods Methods of Nonlinear Optimization* 258, Conjugate Direction Methods Conjugate Direction Methods

Quasi-Newton Methods Quasi-Newton Methods Closure Quasi-Newton Methods 23. Closure Closure

Properties of DFP iterations: Methods

1. If Bk is symmetric and positive definite, then so is Bk+1. of Table 23.1: Summary of performance of optimization methods

2. For quadratic function with positive definite Hessian H, Nonlinear Cauchy Newton Levenberg-Marquardt DFP/BFGS FR/PR Powell T (Steepest (Hybrid) (Quasi-Newton) (Conjugate (Direction pi Hpj = 0 for 0 i < j k, Descent) (Deflected Gradient) (Variable Metric) Gradient) Set) ≤ ≤ For Quadratic Optimiza and Bk+1Hpi = pi for 0 i k. Problems: ≤ ≤ Convergence steps N 1 N n n n2 Indefinite Unknown tion* Implications: Evaluations Nf 2f Nf (n + 1)f (n + 1)f n2f Ng 2g Ng (n + 1)g (n + 1)g 1. Positive definiteness of inverse Hessian estimate is never lost. 1H NH 2. Successive search directions are conjugate directions. Equivalent function evaluations N(2n + 1) 2n2 + 2n + 1 N(2n2 + 1) 2n2 + 3n + 1 2n2 + 3n + 1 n2 3. With B = I, the algorithm is a conjugate gradient method. 0 Line searches N 0 N or 0 n n n2

4. For a quadratic problem, the inverse Hessian gets completely Storage Vector Matrix Matrix Matrix Vector Matrix constructed after n steps. Performance in general problems Slow Risky Costly Flexible Good Okay Practically good for Unknown Good NL Eqn. systems Bad Large Small Variants: Broyden-Fletcher-Goldfarb-Shanno (BFGS) start-up functions NL least squares functions problems problems method and the Broyden family of methods 197

Applied Mathematical Methods Methods of Nonlinear Optimization* 259, Applied Mathematical Methods Constrained Optimization 260, Conjugate Direction Methods Constraints Points to note Quasi-Newton Methods Outline Optimality Criteria Closure Sensitivity Duality* Structure of Methods: An Overview*

I Conjugate directions and the expanding subspace property I Conjugate gradient method Constrained Optimization I Powell-Smith direction set method Constraints Optimality Criteria I The quasi-Newton concept in professional optimization Sensitivity Duality* Structure of Methods: An Overview* Necessary Exercises: 1,2,3 Applied Mathematical Methods Constrained Optimization 261, Applied Mathematical Methods Constrained Optimization 262, Constraints Constraints Constraints Optimality Criteria Constraints Optimality Criteria Sensitivity Sensitivity Duality* Duality* Constrained optimization problem: Structure of Methods: An Overview* Structure of Methods: An Overview* Constraint qualification Minimize f (x) subject to gi (x) 0 for i = 1, 2, , l, or g(x) 0; h1(x), h2(x) etc are linearly independent, i.e. h(x) is ≤ · · · ≤ ∇ ∇ ∇ and hj (x) = 0 for j = 1, 2, , m, or h(x) = 0. full-rank. · · ·

Conceptually, “minimize f (x), x Ω”. If a feasible point x0, with h(x0) = 0, satisfies the constraint ∈ Equality constraints reduce the domain to a surface or a manifold, qualification condition, we call it a regular point. possessing a tangent plane at every point. Gradient of the vector function h(x): At a regular feasible point x0, tangent plane T T = y : [ h(x )] y = 0 ∂h M { ∇ 0 } ∂x1 T  ∂h  gives the collection of feasible directions. ∂x2 h(x) [ h1(x) h2(x) hm(x)] . , ∇ ≡ ∇ ∇ · · · ∇ ≡  .   .  Equality constraints reduce the dimension of the problem.  ∂hT     ∂xn    Variable elimination? ∂h T related to the usual Jacobian as Jh(x) = = [ h(x)] . ∂x ∇

Applied Mathematical Methods Constrained Optimization 263, Applied Mathematical Methods Constrained Optimization 264, Constraints Constraints Constraints Optimality Criteria Optimality Criteria Optimality Criteria Sensitivity Sensitivity Duality* Duality* Structure of Methods: An Overview* Structure of Methods: An Overview* Active inequality constraints gi (x0) = 0: Suppose x∗ is a regular point with (a) included among hj (x0) I active inequality constraints: g (x) 0 ≤ I inactive constraints: g(i)(x) 0 for the tangent plane. ≤ Columns of h(x ) and g(a)(x ): basis for orthogonal Cone of feasible directions: ∇ ∗ ∇ ∗ complement of the tangent plane T T [ h(x0)] d = 0 and [ gi (x0)] d 0 for i I Basis of the tangent plane: D = [d d dk ] ∇ ∇ ≤ ∈ 1 2 · · · Then, [D h(x ) g(a)(x )]: basis of R n where I is the set of indices of active inequality constraints. ∇ ∗ ∇ ∗ Now, f (x ) is a vector in R n. Handling inequality constraints: −∇ ∗ I Active set strategy maintains a list of active constraints, z (a) keeps checking at every step for a change of scenario and f (x∗) = [D h(x∗) g (x∗)] λ −∇ ∇ ∇   updates the list by inclusions and exclusions. µ(a) I Slack variable strategy replaces all the inequality constraints (a)   with unique z, λ and µ for a given f (x∗). by equality constraints as gi (x) + xn+i = 0 with the inclusion ∇ of non-negative slack variables (xn+i ). What can you say if x∗ is a solution to the NLP problem? Applied Mathematical Methods Constrained Optimization 265, Applied Mathematical Methods Constrained Optimization 266, Constraints Constraints Optimality Criteria Optimality Criteria Optimality Criteria Optimality Criteria Sensitivity Sensitivity Duality* Duality* Components of f (x ) in the tangent planeStructuremustof Methods:beAnzero.Overview* Finally, what about the feasible directions inStructurethe cone?of Methods: An Overview* ∇ ∗ Answer: Negative gradient f (x∗) can have no component (a) (a) −∇ z = 0 f (x∗) = [ h(x∗)]λ + [ g (x∗)]µ (a) (a) ⇒ −∇ ∇ ∇ towards decreasing gi (x), i.e. µi 0, i. (i) ≥ ∀ For inactive constraints, insisting on µ = 0, Combining it with µ(i) = 0, µ 0. i ≥ (a) First order necessary conditions or Karusch-Kuhn-Tucker (a) (i) µ f (x∗) = [ h(x∗)]λ + [ g (x∗) g (x∗)] , −∇ ∇ ∇ ∇ µ(i) (KKT) conditions: If x∗ is a regular point of the constraints and   a solution to the NLP problem, then there exist Lagrange or multiplier vectors, λ and µ, such that f (x ) + [ h(x )]λ + [ g(x )]µ = 0 ∇ ∗ ∇ ∗ ∇ ∗ Optimality: f (x∗) + [ h(x∗)]λ + [ g(x∗)]µ = 0, µ 0; g(a)(x) µ(a) ∇ ∇ ∇ ≥ where g(x) = and µ = . Feasibility: h(x∗) = 0, g(x∗) 0; g(i)(x) µ(i) T ≤     Complementarity: µ g(x∗) = 0. (a) (i) Notice: g (x∗) = 0 and µ = 0 µi gi (x∗) = 0 i, or T ⇒ ∀ Convex programming problem: Convex objective function f (x) µ g(x∗) = 0. and convex domain (convex gi (x) and linear hj (x)): Now, components in g(x) are free to appear in any order. KKT conditions are sufficient as well!

Applied Mathematical Methods Constrained Optimization 267, Applied Mathematical Methods Constrained Optimization 268, Constraints Constraints Optimality Criteria Optimality Criteria Optimality Criteria Optimality Criteria Sensitivity Sensitivity Duality* Duality* Lagrangian function: Structure of Methods: An Overview* Structure of Methods: An Overview* Including contributions from all active constraints, L(x, λ, µ) = f (x) + λT h(x) + µT g(x) 2 Necessary conditions for a stationary point of the Lagrangian: d T T f (z(t)) = z˙ (0) HL(x∗)z˙ (0) + [ x L(x∗, λ, µ)] ¨z(0) 0, dt2 ∇ ≥ t=0 x L = 0, λL = 0 ∇ ∇ 2 where H (x) = ∂ L = H(x) + λ H (x) + µ H (x). Second order conditions L ∂x2 j j hj i i gi Consider curve z(t) in the tangent plane with z(0) = x∗. First order necessary condition Pmakes the secondP term vanish!

d 2 d Second order necessary condition: f (z(t)) = [ f (z(t))T z˙ (t)] dt2 dt ∇ The Hessian matrix of the Lagrangian function is positive t=0 t=0 T T semi-definite on the tangent plane . = z˙ (0) H(x∗)z˙(0) + [ f (x∗)] ¨z(0) 0 M ∇ ≥ Sufficient condition: x L = 0 and HL(x) positive definite on . Similarly, from hj (z(t)) = 0, ∇ M n n T T Restriction of the mapping HL(x∗) : R R on subspace ? z˙ (0) Hh (x∗)z˙ (0) + [ hj (x∗)] ¨z(0) = 0. j ∇ → M Applied Mathematical Methods Constrained Optimization 269, Applied Mathematical Methods Constrained Optimization 270, Constraints Constraints Optimality Criteria Optimality Criteria Sensitivity Optimality Criteria Sensitivity Sensitivity Duality* Duality* Structure of Methods: An Overview* Structure of Methods: An Overview* Take y , operate HL(x ) on it, project the image back to . Suppose original objective and constraint functions as ∈ M ∗ M Restricted mapping LM : f (x, p), g(x, p) and h(x, p) M → M Question: Matrix representation for LM of size (n m) (n m)? By choosing parameters (p), we arrive at x∗. Call it x∗(p). − × − Select local orthonormal basis D R n (n m) for . Question: How does f (x∗(p), p) depend on p? ∈ × − M n m n For arbitrary z R , map y = Dz R as HLy = HLDz. Total gradients ∈ − ∈ Its component along d : dT H Dz ¯ pf (x∗(p), p) = px∗(p) x f (x∗, p) + pf (x∗, p), i i L ∇ ∇ ∇ ∇ ¯ ph(x∗(p), p) = px∗(p) x h(x∗, p) + ph(x∗, p) = 0, Hence, projection back on : ∇ ∇ ∇ ∇ M T and similarly for g(x∗(p), p). LM z = D HLDz, In view of x L = 0, from KKT conditions, T The (n m) (n m) matrix LM = D HLD: the restriction! ∇ − × − ¯ pf (x∗(p), p) = pf (x∗, p) + [ ph(x∗, p)]λ + [ pg(x∗, p)]µ Second order necessary/sufficient condition: LM p.s.d./p.d. ∇ ∇ ∇ ∇

Applied Mathematical Methods Constrained Optimization 271, Applied Mathematical Methods Constrained Optimization 272, Constraints Constraints Sensitivity Optimality Criteria Duality* Optimality Criteria Sensitivity Sensitivity Duality* Duality* Structure of Methods: An Overview* Dual problem: Structure of Methods: An Overview* Sensitivity to constraints Reformulation of a problem in terms of the Lagrange multipliers. In particular, in a revised problem, with h(x) = c and g(x) d, ≤ Suppose x as a local minimum for the problem using p = c, ∗ Minimize f (x) subject to h(x) = 0, pf (x∗, p) = 0, ph(x∗, p) = I and pg(x∗, p) = 0. ∇ ∇ − ∇ with Lagrange multiplier (vector) λ∗.

¯ c f (x∗(p), p) = λ ∇ − f (x∗) + [ h(x∗)]λ∗ = 0 ∇ ∇ Similarly, using p = d, we get ¯ d f (x (p), p) = µ. ∇ ∗ − If HL(x∗) is positive definite (assumption of local duality), then x∗ Lagrange multipliers λ and µ signify costs of pulling the minimum is also a local minimum of point in order to satisfy the constraints! T f¯(x) = f (x) + λ∗ h(x). I Equality constraint: both sides infeasible, sign of λj identifies one side or the other of the hypersurface. If we vary λ around λ∗, the minimizer of

I Inequality constraint: one side is feasible, no cost of pulling T L(x, λ) = f (x) + λ h(x) from that side, so µi 0. ≥ varies continuously with λ. Applied Mathematical Methods Constrained Optimization 273, Applied Mathematical Methods Constrained Optimization 274, Constraints Constraints Duality* Optimality Criteria Duality* Optimality Criteria Sensitivity Sensitivity Duality* Duality* Structure of Methods: An Overview* Structure of Methods: An Overview* In the neighbourhood of λ∗, define the dual function Hessian of the dual function: Φ(λ) = min L(x, λ) = min[f (x) + λT h(x)]. H (λ) = x(λ) x h(x(λ)) x x φ ∇λ ∇

Differentiating x L(x(λ), λ) = 0, we have For a pair x, λ , the dual solution is feasible if and only ∇ { } if the primal solution is optimal. T x(λ)HL(x(λ), λ) + [ x h(x(λ))] = 0. ∇λ ∇ Define x(λ) as the local minimizer of L(x, λ). Solving for x(λ) and substituting, ∇λ T T 1 Φ(λ) = L(x(λ), λ) = f (x(λ)) + λ h(x(λ)) H (λ) = [ x h(x(λ))] [HL(x(λ), λ)]− x h(x(λ)), φ − ∇ ∇ First derivative: negative definite!

Φ(λ) = λx(λ) x L(x(λ), λ) + h(x(λ)) = h(x(λ)) At λ∗, x(λ∗) = x , Φ(λ∗) = h(x ) = 0, H (λ∗) is negative ∇ ∇ ∇ ∗ ∇ ∗ φ definite and the dual function is maximized. For a pair x, λ , the dual solution is optimal if and only { } if the primal solution is feasible. Φ(λ∗) = L(x∗, λ∗) = f (x∗)

Applied Mathematical Methods Constrained Optimization 275, Applied Mathematical Methods Constrained Optimization 276, Constraints Constraints Duality* Optimality Criteria Structure of Methods: An Overview* Optimality Criteria Sensitivity Sensitivity Duality* Duality* Consolidation (including all constraints) Structure of Methods: An Overview* For a problem of n variables, with m active Structureconstraints,of Methods: An Overview* I Assuming local convexity, the dual function: nature and dimension of working spaces Penalty methods (R n): Minimize the penalized function Φ(λ, µ) = min L(x, λ, µ) = min[f (x) + λT h(x) + µT g(x)]. x x q(c, x) = f (x) + cP(x). I Constraints on the dual: x L(x, λ, µ) = 0, optimality of the ∇ 1 2 1 2 primal. Example: P(x) = 2 h(x) + 2 [max(0, g(x))] . n m k k I Corresponding to inequality constraints of the primal problem, Primal methods (R − ): Work only in feasible domain, restricting non-negative variables µ in the dual problem. steps to the tangent plane. I First order necessary conditons for the dual optimality: Example: Gradient projection method. equivalent to the feasibility of the primal problem. Dual methods (R m): Transform the problem to the space of I The dual function is concave globally! Lagrange multipliers and maximize the dual. I Example: Augmented Lagrangian method. Under suitable conditions, Φ(λ∗) = L(x∗, λ∗) = f (x∗). Lagrange methods (R m+n): Solve equations appearing in the KKT I The Lagrangian L(x, λ, µ) has a saddle point in the combined conditions directly. space of primal and dual variables: positive curvature along x Example: Sequential quadratic programming. directions and negative curvature along λ and µ directions. Applied Mathematical Methods Constrained Optimization 277, Applied Mathematical Methods Linear and Quadratic Programming Problems* 278, Constraints Linear Programming Points to note Optimality Criteria Outline Quadratic Programming Sensitivity Duality* Structure of Methods: An Overview*

I Constraint qualification I KKT conditions Linear and Quadratic Programming Problems* I Second order conditions Linear Programming I Basic ideas for solution strategy Quadratic Programming

Necessary Exercises: 1,2,3,4,5,6

Applied Mathematical Methods Linear and Quadratic Programming Problems* 279, Applied Mathematical Methods Linear and Quadratic Programming Problems* 280, Linear Programming Linear Programming Linear Programming Quadratic Programming Linear Programming Quadratic Programming Standard form of an LP problem: The simplex method N M M N T Suppose x R , b R and A R full-rank, with M < N. Minimize f (x) = c x, ∈ ∈ ∈ × subject to Ax = b, x 0; with b 0. ≥ ≥ IM xB + A0xNB = b0 Preprocessing to cast a problem to the standard form M N M Basic and non-basic variables: xB R and xNB R − I Maximization: Minimize the negative function. ∈ ∈ Basic feasible solution: xB = b0 0 and xNB = 0 I Variables of unrestricted sign: Use two variables. ≥ At every iteration, I Inequality constraints: Use slack/surplus variables. I selection of a non-basic variable to enter the basis I Negative RHS: Multiply with 1. − I edge of travel selected based on maximum rate of descent Geometry of an LP problem I no qualifier: current vertex is optimal I Infinite domain: does a minimum exist? I selection of a basic variable to leave the basis I Finite convex polytope: existence guaranteed I based on the first constraint becoming active along the edge I I Operating with vertices sufficient as a strategy no constraint ahead: function is unbounded I Extension with slack/surplus variables: original solution space I elementary row operations: new basic feasible solution a subspace in the extented space, x 0 marking the domain Two-phase method: Inclusion of a pre-processing phase with ≥ I Essence of the non-negativity condition of variables artificial variables to develop a basic feasible solution Applied Mathematical Methods Linear and Quadratic Programming Problems* 281, Applied Mathematical Methods Linear and Quadratic Programming Problems* 282, Linear Programming Linear Programming Linear Programming Quadratic Programming Quadratic Programming Quadratic Programming

General perspective A quadratic objective function and linear constraints define LP problem: a QP problem. T T Minimize f (x, y) = c1 x + c2 y; subject to A x + A y = b , A x + A y b , y 0. Equations from the KKT conditions: linear! 11 12 1 21 22 ≤ 2 ≥ Lagrangian: Lagrange methods are the natural choice! L(x, y, λ, µ, ν) = cT x + cT y 1 2 With equality constraints only, T T T + λ (A11x + A12y b1) + µ (A21x + A22y b2) ν y − − − 1 Optimality conditions: Minimize f (x) = xT Qx + cT x, subject to Ax = b. 2 T T T T c1 + A λ + A µ = 0 and ν = c2 + A λ + A µ 0 11 21 12 22 ≥ First order necessary conditions: T T Substituting back, optimal function value: f ∗ = λ b1 µ b2 ∂f ∂f − − T Sensitivity to the constraints: ∗ = λ and ∗ = µ Q A x∗ c ∂b1 − ∂b2 − = − Dual problem: A 0 λ b       T T maximize Φ(λ, µ) = b1 λ b2 µ; T T− − T T Solution of this linear system yields the complete result! subject to A11λ + A21µ = c1, A12λ + A22µ c2, µ 0. − ≥ − ≥ Caution: This coefficient matrix is indefinite. Notice the symmetry between the primal and dual problems.

Applied Mathematical Methods Linear and Quadratic Programming Problems* 283, Applied Mathematical Methods Linear and Quadratic Programming Problems* 284, Linear Programming Linear Programming Quadratic Programming Quadratic Programming Quadratic Programming Quadratic Programming Active set method Linear complementary problem (LCP)

1 T T Slack variable strategy with inequality constraints Minimize f (x) = 2 x Qx + c x; 1 T T subject to A1x = b1, Minimize x Qx + c x, subject to Ax b, x 0. 2 ≤ ≥ A2x b2. ≤ KKT conditions: With x, y, µ, ν 0, Start the iterative process from a feasible point. ≥ I Construct active set of constraints as Ax = b. Qx + c + AT µ ν = 0, I − From the current point xk , with x = xk + dk , Ax + y = b, 1 T T f (x) = (x + d )T Q(x + d ) + cT (x + d ) x ν = µ y = 0. 2 k k k k k k 1 Denoting = dT Qd + (c + Qx )T d + f (x ). 2 k k k k k x ν c Q AT I Since gk f (xk ) = c + Qxk , subsidiary quadratic program: z = , w = , q = and M = , ≡ ∇ µ y b A 0 1 T T        −  minimize 2 dk Qdk + gk dk subject to Adk = 0. T I w Mz = q, w z = 0. Examining solution dk and Lagrange multipliers, decide to − terminate, proceed or revise the active set. Find mutually complementary non-negative w and z. Applied Mathematical Methods Linear and Quadratic Programming Problems* 285, Applied Mathematical Methods Linear and Quadratic Programming Problems* 286, Linear Programming Linear Programming Quadratic Programming Quadratic Programming Points to note Quadratic Programming

If q 0, then w = q, z = 0 is a solution! ≥ T Lemke’s method: artificial variable z0 with e = [1 1 1 1] : · · · I Fundamental issues and general perspective of the linear Iw Mz ez = q programming problem − − 0 I The simplex method With z = max( qi ), 0 − I Quadratic programming w = q + ez 0 and z = 0: basic feasible solution I 0 ≥ The active set method I Lemke’s method via the linear complementary problem I Evolution of the basis similar to the simplex method. I Out of a pair of w and z variables, only one can be there in any basis. Necessary Exercises: 1,2,3,4,5 I At every step, one variable is driven out of the basis and its partner called in. I The step driving out z0 flags termination.

Handling of equality constraints? Very clumsy!!

Applied Mathematical Methods Interpolation and Approximation 287, Applied Mathematical Methods Interpolation and Approximation 288, Polynomial Interpolation Polynomial Interpolation Outline Piecewise Polynomial Interpolation Polynomial Interpolation Piecewise Polynomial Interpolation Interpolation of Multivariate Functions Interpolation of Multivariate Functions A Note on Approximation of Functions A Note on Approximation of Functions Modelling of Curves and Surfaces* Problem: To develop an analytical representationModelling ofofCurvesa functionand Surfaces* from information at discrete data points. Purpose I Evaluation at arbitrary points Interpolation and Approximation I Polynomial Interpolation Differentiation and/or integration Piecewise Polynomial Interpolation I Drawing conclusion regarding the trends or nature Interpolation of Multivariate Functions Interpolation: one of the ways of function representation A Note on Approximation of Functions I sampled data are exactly satisfied Modelling of Curves and Surfaces* Polynomial: a convenient class of basis functions For yi = f (xi ) for i = 0, 1, 2, , n with x < x < x < < xn, · · · 0 1 2 · · · 2 n p(x) = a + a x + a x + + anx . 0 1 2 · · ·

Find the coefficients such that p(xi ) = f (xi ) for i = 0, 1, 2, , n. · · · Values of p(x) for x [x , xn] interpolate n + 1 values ∈ 0 of f (x), an outside estimate is extrapolation. Applied Mathematical Methods Interpolation and Approximation 289, Applied Mathematical Methods Interpolation and Approximation 290, Polynomial Interpolation Polynomial Interpolation Polynomial Interpolation Piecewise Polynomial Interpolation Polynomial Interpolation Piecewise Polynomial Interpolation Interpolation of Multivariate Functions Interpolation of Multivariate Functions A Note on Approximation of Functions A Note on Approximation of Functions Modelling of Curves and Surfaces* Lagrange interpolation Modelling of Curves and Surfaces* To determine p(x), solve the linear system Basis functions: 2 n n (x x ) 1 x0 x0 x0 a f (x ) j=0,j=k j · · · 0 0 L (x) = 6 − 1 x x2 xn k n 1 1 1 a1 f (x1) Qj=0,j=k (xk xj )  2 · · · n      6 − 1 x2 x2 x2 a = f (x ) ? · · · 2 2 Q(x x0)(x x1) (x xk 1)(x xk+1) (x xn)  ......      = − − · · · − − − · · · −  . . . . .      (xk x0)(xk x1) (xk xk 1)(xk xk+1) (xk xn)  2 n  · · · · · · − − · · · − − − · · · −  1 xn x x   an   f (xn)   n · · · n      Interpolating polynomial:       Vandermonde matrix: invertible, but typically ill-conditioned! p(x) = α0L0(x) + α1L1(x) + α2L2(x) + + αnLn(x) · · · Invertibility means existence and uniqueness of polynomial p(x). At the data points, Lk (xi ) = δik .

Coefficient matrix identity and αi = f (xi ). Two polynomials p1(x) and p2(x) matching the function f (x) at x , x , x , , xn imply 0 1 2 · · · Lagrange interpolation formula: n-th degree polynomial ∆p(x) = p (x) p (x) with n 1 − 2 n + 1 roots! p(x) = f (xk )Lk (x) = L (x)f (x )+L (x)f (x )+ +Ln(x)f (xn) 0 0 1 1 · · · k=0 ∆p 0 p1(x) = p2(x): p(x) is unique. X ≡ ⇒ Existence of p(x) is a trivial consequence!

Applied Mathematical Methods Interpolation and Approximation 291, Applied Mathematical Methods Interpolation and Approximation 292, Polynomial Interpolation Polynomial Interpolation Polynomial Interpolation Piecewise Polynomial Interpolation Piecewise Polynomial Interpolation Piecewise Polynomial Interpolation Interpolation of Multivariate Functions Interpolation of Multivariate Functions A Note on Approximation of Functions A Note on Approximation of Functions Two interpolation formulae Modelling of Curves and Surfaces* Piecewise linear interpolation Modelling of Curves and Surfaces* I one costly to determine, but easy to process f (xi ) f (xi 1) f (x) = f (xi 1) + − − (x xi 1) for x [xi 1, xi ] I − xi xi 1 − − ∈ − the other trivial to determine, costly to process − − Handy for many uses with dense data. But, not differentiable. Newton interpolation for an intermediate trade-off: n 1 Piecewise cubic interpolation p(x) = c0 + c1(x x0) + c2(x x0)(x x1) + + cn i=0− (x xi ) − − − · · · − With function values and derivatives at (n + 1) points, Hermite interpolation Q n cubic Hermite segments uses derivatives as well as function values.

(ni 1) Data for the j-th segment: Data: f (xi ), f 0(xi ), , f − (xi ) at x = xi , for i = 0, 1, , m: · · · m · · · I At (m + 1) points, a total of n + 1 = ni conditions f (xj 1) = fj 1, f (xj ) = fj , f 0(xj 1) = fj0 1 and f 0(xj ) = fj0 i=0 − − − − Limitations of single-polynomial interpolationP Interpolating polynomial: 2 3 With large number of data points, polynomial degree is high. pj (x) = a0 + a1x + a2x + a3x I Computational cost and numerical imprecision Coefficients a0, a1, a2, a3: linear combinations of fj 1, fj , fj0 1, fj0 − − I Lack of representative nature due to oscillations Composite function 1 continuous at knot points. C Applied Mathematical Methods Interpolation and Approximation 293, Applied Mathematical Methods Interpolation and Approximation 294, Polynomial Interpolation Polynomial Interpolation Piecewise Polynomial Interpolation Piecewise Polynomial Interpolation Piecewise Polynomial Interpolation Piecewise Polynomial Interpolation Interpolation of Multivariate Functions Interpolation of Multivariate Functions A Note on Approximation of Functions A Note on Approximation of Functions General formulation through normalizationMoofdellingintervalsof Curves and Surfaces* Modelling of Curves and Surfaces* Spline interpolation x = xj 1 + t(xj xj 1), t [0, 1] − − − ∈ Spline: a drafting tool to draw a smooth curve through key points. With g(t) = f (x(t)), g 0(t) = (xj xj 1)f 0(x(t)); − − Data: fi = f (xi ), for x < x < x < < xn. 0 1 2 · · · g0 = fj 1, g1 = fj , g00 = (xj xj 1)fj0 1 and g10 = (xj xj 1)fj0. − − − − − − If kj = f 0(xj ), then Cubic polynomial for the j-th segment: pj (x) can be determined in terms of fj 1, fj , kj 1, kj 2 3 − − qj (t) = α0 + α1t + α2t + α3t and pj+1(x) in terms of fj , fj+1, kj , kj+1.

Modular expression: Then, pj00(xj ) = pj00+1(xj ): a linear equation in kj 1, kj and kj+1 − 1 1 From n 1 interior knot points, t t − qj (t) = [α0 α1 α2 α3]   = [g0 g1 g 0 g 0 ] W   = Gj WT n 1 linear equations in derivative values k0, k1, , kn. t2 0 1 t2 − · · ·  t3   t3  Prescribing k and k , a diagonally dominant tridiagonal system!     0 n     Packaging data, interpolation type and variable terms separately! A spline is a smooth interpolation, with 2 continuity. Question: How to supply derivatives? And, why? C

Applied Mathematical Methods Interpolation and Approximation 295, Applied Mathematical Methods Interpolation and Approximation 296, Polynomial Interpolation Polynomial Interpolation Interpolation of Multivariate FunctionsPiecewise Polynomial Interpolation Interpolation of Multivariate FunctionsPiecewise Polynomial Interpolation Interpolation of Multivariate Functions Interpolation of Multivariate Functions A Note on Approximation of Functions A Note on Approximation of Functions Piecewise bilinear interpolation Modelling of Curves and Surfaces* Alternative local formula through reparametrizationModelling of Curves and Surfaces* x xi 1 y yj 1 With u = − − and v = − − , denoting xi xi 1 yj yj 1 Data: f (x, y) over a dense rectangular grid − − − −

x = x0, x1, x2, , xm and y = y0, y1, y2, , yn fi 1,j 1 = g0,0, fi,j 1 = g1,0, fi 1,j = g0,1 and fi,j = g1,1; · · · · · · − − − − Rectangular domain: (x, y) : x x xm, y y yn bilinear interpolation: { 0 ≤ ≤ 0 ≤ ≤ }

For xi 1 x xi and yj 1 y yj , α0,0 α0,1 1 − ≤ ≤ − ≤ ≤ g(u, v) = [1 u] for u, v [0, 1]. α1,0 α1,1 v ∈     a0,0 a0,1 1 f (x, y) = a0,0 + a1,0x + a0,1y + a1,1xy = [1 x] Values at four corner points fix the coefficient matrix as a a y  1,0 1,1    α α 1 0 g g 1 1 With data at four corner points, coefficient matrix determined from 0,0 0,1 = 0,0 0,1 − . α α 1 1 g g 0 1  1,0 1,1   −   1,0 1,1    1 xi 1 a0,0 a0,1 1 1 fi 1,j 1 fi 1,j T T − = − − − . Concisely, g(u, v) = U W Gi,j WV in which 1 xi a1,0 a1,1 yj 1 yj fi,j 1 fi,j      −   −  1 1 1 1 fi 1,j 1 fi 1,j 0 U = , V = , W = − , Gi,j = − − − . Approximation only continuous. u v 0 1 fi,j 1 fi,j C        −  Applied Mathematical Methods Interpolation and Approximation 297, Applied Mathematical Methods Interpolation and Approximation 298, Polynomial Interpolation Polynomial Interpolation Interpolation of Multivariate FunctionsPiecewise Polynomial Interpolation A Note on Approximation of FunctionsPiecewise Polynomial Interpolation Interpolation of Multivariate Functions Interpolation of Multivariate Functions A Note on Approximation of Functions A Note on Approximation of Functions Piecewise bicubic interpolation Modelling of Curves and Surfaces* Modelling of Curves and Surfaces* ∂f ∂f ∂2f A common strategy of function approximation is to Data: f , ∂x , ∂y and ∂x∂y over grid points With normalizing parameters u and v, I express a function as a linear combination of a set of basis functions (which?), and ∂g ∂f ∂g ∂f = (xi xi 1) , = (yj yj 1) , and I ∂u − − ∂x ∂v − − ∂y determine coefficients based on some criteria (what?). ∂2g ∂2f ∂u∂v = (xi xi 1)(yj yj 1) ∂x∂y − − − − Criteria:

In (x, y) : xi 1 x xi , yj 1 y yj or (u, v) : u, v [0, 1] , Interpolatory approximation: Exact agreement with sampled data { − ≤ ≤ − ≤ ≤ } { ∈ } T T Least square approximation: Minimization of a sum (or integral) of g(u, v) = U W Gi,j WV, square errors over sampled data with U = [1 u u2 u3]T , V = [1 v v 2 v 3]T , and Minimax approximation: Limiting the largest deviation

g(0, 0) g(0, 1) gv (0, 0) gv (0, 1) Basis functions: g(1, 0) g(1, 1) gv (1, 0) gv (1, 1) Gi,j =   . polynomials, sinusoids, orthogonal eigenfunctions or gu(0, 0) gu(0, 1) guv (0, 0) guv (0, 1) field-specific heuristic choice  g (1, 0) g (1, 1) g (1, 0) g (1, 1)   u u uv uv   

Applied Mathematical Methods Interpolation and Approximation 299, Applied Mathematical Methods Basic Methods of Numerical Integration 300, Polynomial Interpolation Newton-Cotes Integration Formulae Points to note Piecewise Polynomial Interpolation Outline Richardson Extrapolation and Romberg Integration Interpolation of Multivariate Functions Further Issues A Note on Approximation of Functions Modelling of Curves and Surfaces*

I Lagrange, Newton and Hermite interpolations I Piecewise polynomial functions and splines Basic Methods of Numerical Integration I Bilinear and bicubic interpolation of bivariate functions Newton-Cotes Integration Formulae Direct extension to vector functions: curves and surfaces! Richardson Extrapolation and Romberg Integration Further Issues

Necessary Exercises: 1,2,4,6 Applied Mathematical Methods Basic Methods of Numerical Integration 301, Applied Mathematical Methods Basic Methods of Numerical Integration 302, Newton-Cotes Integration Formulae Newton-Cotes Integration Formulae Newton-Cotes Integration Formulae Richardson Extrapolation and Romberg Integration Newton-Cotes Integration Formulae Richardson Extrapolation and Romberg Integration Further Issues Further Issues Mid-point rule b xi 1+xi Selecting x as x¯ = − , J = f (x)dx i∗ i 2 a n Z xi b Divide [a, b] into n sub-intervals with f (x)dx hf (x¯i ) and f (x)dx h f (x¯i ). ≈ ≈ xi 1 a i Z − Z X=1 a = x0 < x1 < x2 < < xn 1 < xn = b, · · · − Error analysis: From Taylor’s series of f (x) about x¯i , b a xi xi 2 where xi xi 1 = h = − . (x x¯i ) − − n f (x)dx = f (x¯i ) + f 0(x¯i )(x x¯i ) + f 00(x¯i ) − + dx xi 1 xi 1 − 2 · · · n Z − Z −   ¯ 3 5 J = hf (xi∗) = h[f (x1∗) + f (x2∗) + + f (xn∗)] h h iv · · · = hf (x¯i ) + f 00(x¯i ) + f (x¯i ) + , i=1 24 1920 · · · X Taking xi∗ [xi 1, xi ] as xi 1 and xi , we get summations J1 and J2. third order accurate! ∈ − − Over the entire domain [a, b], As n (i.e. h 0), if J1 and J2 approach the same → ∞ → n n n limit, then function f (x) is integrable over interval [a, b]. b h3 h2 f (x)dx h f (x¯i )+ f 00(x¯i ) = h f (x¯i )+ (b a)f 00(ξ), ≈ 24 24 − a i i i A rectangular rule or a one-point rule Z X=1 X=1 X=1 Question: Which point to take as x ∗? for ξ [a, b] (from mean value theorem): second order accurate. i ∈

Applied Mathematical Methods Basic Methods of Numerical Integration 303, Applied Mathematical Methods Basic Methods of Numerical Integration 304, Newton-Cotes Integration Formulae Newton-Cotes Integration Formulae Newton-Cotes Integration Formulae Richardson Extrapolation and Romberg Integration Newton-Cotes Integration Formulae Richardson Extrapolation and Romberg Integration Further Issues Further Issues Trapezoidal rule Error estimate of trapezoidal rule Approximating function f (x) with a linear interpolation, xi 3 5 x h h h iv i h f (x)dx = [f (xi 1) + f (xi )] f 00(x¯i ) f (x¯i ) + − f (x)dx [f (xi 1) + f (xi )] xi 1 2 − 12 − 480 · · · − Z − xi 1 ≈ 2 Z − and Over an extended domain, b n 1 − 1 1 b n 1 2 f (x)dx h f (x0) + f (xi ) + f (xn) . 1 − h a ≈ 2 2 f (x)dx = h f (x ) + f (xn) + f (xi ) (b a)f 00(ξ)+ . Z " i=1 # 2{ 0 } −12 − · · · X a " i # Taylor series expansions about the mid-point: Z X=1 2 3 4 h h h h iv The same order of accuracy as the mid-point rule! f (xi 1) = f (x¯i ) f 0(x¯i ) + f 00(x¯i ) f 000(x¯i ) + f (x¯i ) − − 2 8 − 48 384 − · · · 2 3 4 Different sources of merit h h h h iv f (xi ) = f (x¯i ) + f 0(x¯i ) + f 00(x¯i ) + f 000(x¯i ) + f (x¯i ) + I Mid-point rule: Use of mid-point leads to symmetric 2 8 48 384 · · · error-cancellation. 3 5 h h h iv I [f (xi 1) + f (xi )] = hf (x¯i ) + f 00(x¯i ) + f (x¯i ) + Trapezoidal rule: Use of end-points allows double utilization ⇒ 2 − 8 384 · · · of boundary points in adjacent intervals. xi h3 h5 iv Recall f (x)dx = hf (x¯i ) + f 00(x¯i ) + f (x¯i ) + . xi 1 24 1920 − · · · How to use both the merits? R Applied Mathematical Methods Basic Methods of Numerical Integration 305, Applied Mathematical Methods Basic Methods of Numerical Integration 306, Newton-Cotes Integration Formulae Newton-Cotes Integration Formulae Newton-Cotes Integration Formulae Richardson Extrapolation and Romberg Integration Richardson Extrapolation and RombergRichardsonIntegrationExtrapolation and Romberg Integration Further Issues Further Issues Simpson’s rules To determine quantity F Divide [a, b] into an even number (n = 2m) of intervals. I using a step size h, estimate F (h) Fit a quadratic polynomial over a panel of two intervals. I error terms: hp, hq, hr etc (p < q < r) I For this panel of length 2h, two estimates: F = limδ 0 F (δ)? → I plot F (h), F (αh), F (α2h) (with α < 1) and extrapolate? M(f ) = 2hf (xi ) and T (f ) = h[f (xi 1) + f (xi+1)] − p q 1 F (h) = F + ch + (h ) 3 5 h h iv pO q J = M(f ) + f 00(xi ) + f (xi ) + 2 F (αh) = F + c(αh) + (h ) 3 60 · · · 2 2 p O q 3 5 4 F (α h) = F + c(α h) + (h ) 2h h iv O J = T (f ) f 00(xi ) f (xi ) + − 3 − 15 · · · Eliminate c and determine (better estimates of) F : Simpson’s one-third rule (with error estimate): p F (αh) α F (h) q r 3 F1(h) = − = F + c1h + (h ) xi+1 5 p h h iv 1 α O f (x)dx = [f (xi 1) + 4f (xi ) + f (xi+1)] f (xi ) 2 − p − F (α h) α F (αh) q r xi 1 3 − 90 Z − 5 F1(αh) = − = F + c1(αh) + (h ) 1 αp O Fifth (not fourth) order accurate! − q F1(αh) α F1(h) r Still better estimate: 6 F2(h) = 1−αq = F + (h ) A four-point rule: Simpson’s three-eighth rule − O Still higher order rules NOT advisable! Richardson extrapolation

Applied Mathematical Methods Basic Methods of Numerical Integration 307, Applied Mathematical Methods Basic Methods of Numerical Integration 308, Newton-Cotes Integration Formulae Newton-Cotes Integration Formulae Richardson Extrapolation and RombergRichardsonIntegrationExtrapolation and Romberg Integration Further Issues Richardson Extrapolation and Romberg Integration Further Issues Further Issues Trapezoidal rule for J = b f (x)dx: p = 2, q = 4, r = 6 etc a Featured functions: adaptive quadrature 2 4 6 T (f ) = JR + ch + dh + eh + I (xi xi 1) · · · With prescribed tolerance , assign quota i = b− a− of − 1 error to every interval [xi 1, xi ]. With α = 2 , half the sum available for successive levels. − I For each interval, find two estimates of the integral and Romberg integration estimate the error. I Trapezoidal rule with h = H: find J11. I If error estimate is not within quota, then subdivide. I With h = H/2, find J12. 1 2 Function as tabulated data J12 2 J11 4J12 J11 J22 = − = − . 1 2 3 I Only trapezoidal rule applicable? 1 2
− I Fit a spline over data points and integrate the segments? I If J22 J12 is within tolerance,
STOP. Accept J J22. | − | ≈ I With h = H/4, find J . 13 Improper integral: Newton-Cotes closed formulae not applicable! 4 1 I 4J13 J12 J23 2 J22 16J23 J22 Open Newton-Cotes formulae J23 = − and J33 = − = − . 1 4 3 1
15 I Gaussian quadrature − 2 I If J33 J23 is within tolerance, STOPwith
J J33. | − | ≈ Applied Mathematical Methods Basic Methods of Numerical Integration 309, Applied Mathematical Methods Advanced Topics in Numerical Integration* 310, Newton-Cotes Integration Formulae Gaussian Quadrature Points to note Richardson Extrapolation and Romberg Integration Outline Multiple Integrals Further Issues

I Definition of an integral and integrability I Closed Newton-Cotes formulae and their error estimates I Richardson extrapolation as a general technique Advanced Topics in Numerical Integration* Gaussian Quadrature I Romberg integration Multiple Integrals I Adaptive quadrature

Necessary Exercises: 1,2,3,4

Applied Mathematical Methods Advanced Topics in Numerical Integration* 311, Applied Mathematical Methods Advanced Topics in Numerical Integration* 312, Gaussian Quadrature Gaussian Quadrature Gaussian Quadrature Multiple Integrals Gaussian Quadrature Multiple Integrals n A typical quadrature formula: a weighted sum i=0 wi fi Gauss-Legendre quadrature I fi : function value at i-th sampled point P 1 I wi : corresponding weight f (x)dx = w1f (x1) + w2f (x2) 1 Newton-Cotes formulae: Z− Four variables: Insist that it is exact for 1, x, x 2 and x3. I Abscissas (xi ’s) of sampling prescribed I Coefficients or weight values determined to eliminate 1 dominant error terms w1 + w2 = dx = 2, 1 Z− Gaussian quadrature rules: 1 I no prescription of quadrature points w1x1 + w2x2 = xdx = 0, 1 I Z− only the ‘number’ of quadrature points prescribed 1 2 2 2 2 I locations as well as weights contribute to the accuracy criteria w1x1 + w2x2 = x dx = 1 3 Z− I with n integration points, 2n degrees of freedom 1 3 3 3 I can be made exact for polynomials of degree up to 2n 1 and w1x1 + w2x2 = x dx = 0. − 1 I best locations: interior points Z− x = x , w = w w = w = 1, x = 1 , x = 1 I open quadrature rules: can handle integrable singularities 1 − 2 1 2 ⇒ 1 2 1 − √3 2 √3 Applied Mathematical Methods Advanced Topics in Numerical Integration* 313, Applied Mathematical Methods Advanced Topics in Numerical Integration* 314, Gaussian Quadrature Gaussian Quadrature Gaussian Quadrature Multiple Integrals Gaussian Quadrature Multiple Integrals Two-point Gauss-Legendre quadrature formula General Framework for n-point formula 1 f (x)dx = f ( 1 ) + f ( 1 ) 1 √3 √3 f (x): a polynomial of degree 2n 1 − − − Exact for any cubicR polynomial: parallels Simpson’s rule! p(x): Lagrange polynomial through the n quadrature points Three-point quadrature rule along similar lines: f (x) p(x): a (2n 1)-degree polynomial having n of its roots at − − 1 5 3 8 5 3 the quadrature points f (x)dx = f + f (0) + f 1 9 − 5! 9 9 5! Then, with φ(x) = (x x1)(x x2) (x xn), Z− r r − − · · · − A large number of formulae: Consult mathematical handbooks. f (x) p(x) = φ(x)q(x). For domain of integration [a, b], − n 1 i Quotient polynomial: q(x) = − α x a + b b a b a i=0 i x = + − t and dx = − dt Direct integration: 2 2 2 P With scaling and relocation, 1 1 1 n 1 − i f (x)dx = p(x)dx + φ(x) αi x dx b 1 b a Z 1 Z 1 Z 1 " i=0 # f (x)dx = − f [x(t)]dt − − − X a 2 1 Z Z− How to make the second term vanish?

Applied Mathematical Methods Advanced Topics in Numerical Integration* 315, Applied Mathematical Methods Advanced Topics in Numerical Integration* 316, Gaussian Quadrature Gaussian Quadrature Gaussian Quadrature Multiple Integrals Gaussian Quadrature Multiple Integrals

Choose quadrature points x , x , , xn so that φ(x) is orthogonal 1 2 · · · Weight functions in Gaussian quadrature to all polynomials of degree less than n. Legendre polynomial What is so great about exact integration of polynomials? Demand something else: generalization Gauss-Legendre quadrature Exact integration of polynomials times function W (x) 1. Choose Pn(x), Legendre polynomial of degree n, as φ(x).

2. Take its roots x , x , , xn as the quadrature points. Given weight function W (x) and number (n) of quadrature points, 1 2 · · · 3. Fit Lagrange polynomial of f (x), using these n points. work out the locations (xj ’s) of the n points and the corresponding weights (wj ’s), so that integral p(x) = L1(x)f (x1) + L2(x)f (x2) + + Ln(x)f (xn) · · · b n 4. W (x)f (x)dx = wj f (xj ) 1 1 n 1 Za j=1 f (x)dx = p(x)dx = f (xj ) Lj (x)dx X 1 1 j 1 Z− Z− X=1 Z− is exact for an arbitrary polynomial f (x) of degree up to 1 (2n 1). Weight values: wj = 1 Lj (x)dx, for j = 1, 2, , n − − · · · R Applied Mathematical Methods Advanced Topics in Numerical Integration* 317, Applied Mathematical Methods Advanced Topics in Numerical Integration* 318, Gaussian Quadrature Gaussian Quadrature Gaussian Quadrature Multiple Integrals Multiple Integrals Multiple Integrals

A family of orthogonal polynomials with increasing degree: b g2(x) S = f (x, y) dy dx quadrature points: roots of n-th member of the family. Za Zg1(x)

g2(x) b F (x) = f (x, y) dy and S = F (x)dx For different kinds of functions and different domains, ⇒ Zg1(x) Za I Gauss-Chebyshev quadrature with complete flexibility of individual quadrature methods. I Gauss-Laguerre quadrature Double integral on rectangular domain I Gauss-Hermite quadrature Two-dimensional version of Simpson’s one-third rule: I · · · · · · · · · 1 1 Several singular functions and infinite domains can be handled. f (x, y)dxdy 1 1 Z− Z− A very special case: = w0f (0, 0) + w1[f ( 1, 0) + f (1, 0) + f (0, 1) + f (0, 1)] − − For W (x) = 1, Gauss-Legendre quadrature! + w2[f ( 1, 1) + f ( 1, 1) + f (1, 1) + f (1, 1)] − − − −

Exact for bicubic functions: w0 = 16/9, w1 = 4/9 and w2 = 1/9.

Applied Mathematical Methods Advanced Topics in Numerical Integration* 319, Applied Mathematical Methods Advanced Topics in Numerical Integration* 320, Gaussian Quadrature Gaussian Quadrature Multiple Integrals Multiple Integrals Points to note Multiple Integrals

Monte Carlo integration

I = f (x)dV I Basic strategy of Gauss-Legendre quadrature ZΩ I Requirements: Formulation of a double integral from fundamental principle I I a simple volume V enclosing the domain Ω Monte Carlo integration I a point classification scheme Generating random points in V ,

f (x) if x Ω, Necessary Exercises: 2,5,6 F (x) = ∈ 0 otherwise . 

N V I F (xi ) ≈ N i X=1 Estimate of I (usually) improves with increasing N. Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 321, Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 322, Single-Step Methods Single-Step Methods Outline Practical Implementation of Single-Step Methods Single-Step Methods Practical Implementation of Single-Step Methods Systems of ODE’s Systems of ODE’s Multi-Step Methods* Multi-Step Methods*

Initial value problem (IVP) of a first order ODE:

dy = f (x, y), y(x0) = y0 Numerical Solution of Ordinary Differential Equations dx Single-Step Methods To determine: y(x) for x [a, b] with x0 = a. Practical Implementation of Single-Step Methods ∈ Systems of ODE’s Numerical solution: Start from the point (x0, y0). Multi-Step Methods* I y1 = y(x1) = y(x0 + h) =? I Found (x1, y1). Repeat up to x = b.

Information at how many points are used at every step? I Single-step method: Only the current value I Multi-step method: History of several recent steps

Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 323, Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 324, Single-Step Methods Single-Step Methods Single-Step Methods Practical Implementation of Single-Step Methods Single-Step Methods Practical Implementation of Single-Step Methods Systems of ODE’s Systems of ODE’s Multi-Step Methods* Multi-Step Methods* Euler’s method Initial slope for the entire step: is it a good idea? dy I At (xn, yn), evaluate slope = f (xn, yn). dx y y I For a small step h, C C C1 y 3 Q* Q2 C2 ∆y yn+1 = yn + hf (xn, yn) 3 Q C1 C3 Q y y P 1 Repitition of such steps constructs y(x). 0 0 First order truncated Taylor’s series: P1 O x0 x1 x2 x3 x O x0 x1 x Expected error: (h2) O Figure: Euler’s method Figure: Improved Euler’s method Accumulation over steps Total error: (h) O Improved Euler’s method or Heun’s method Euler’s method is a first order method. y¯n+1 = yn + hf (xn, yn) Question: Total error = Sum of errors over the steps? h yn+1 = yn + [f (xn, yn) + f (xn+1, y¯n+1)] Answer: No, in general. 2 The order of Heun’s method is two. Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 325, Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 326, Single-Step Methods Single-Step Methods Single-Step Methods Practical Implementation of Single-Step Methods Single-Step Methods Practical Implementation of Single-Step Methods Systems of ODE’s Systems of ODE’s Multi-Step Methods* Multi-Step Methods* Runge-Kutta methods With continuous choice of w2, Second order method: a family of second order Runge Kutta (RK2) formulae

k1 = hf (xn, yn), k2 = hf (xn + αh, yn + βk1) Popular form of RK2: with choice w2 = 1, k = w1k1 + w2k2, h k and xn+1 = xn + h, yn+1 = yn + k 1 k1 = hf (xn, yn), k2 = hf (xn + 2 , yn + 2 ) x = x + h, y = y + k Force agreement up to the second order. n+1 n n+1 n 2 Fourth order Runge-Kutta method (RK4): yn+1 = yn + w1hf (xn, yn) + w2h[f (xn, yn) + αhfx (xn, yn) + βk1fy (xn, yn) + ] k = hf (x , y ) · · · 1 n n 2 h k1 = yn + (w1 + w2)hf (xn, yn) + h w2[αfx (xn, yn) + βf (xn, yn)fy (xn, yn)] + k2 = hf (xn + 2 , yn + 2 ) · · · h k2 k3 = hf (xn + 2 , yn + 2 ) From Taylor’s series, using y 0 = f (x, y) and y 00 = fx + ffy , k4 = hf (xn + h, yn + k3) 2 h 1 y(xn+1) = yn + hf (xn, yn) + [fx (xn, yn) + f (xn, yn)fy (xn, yn)] + k = (k + 2k + 2k + k ) 2 · · · 6 1 2 3 4

1 1 w1 + w2 = 1, αw2 = βw2 = α = β = , w1 = 1 w2 xn+1 = xn + h, yn+1 = yn + k 2 ⇒ 2w2 −

Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 327, Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 328, Single-Step Methods Single-Step Methods Practical Implementation of Single-StepPracticalMethoImplementationdsof Single-Step Methods Practical Implementation of Single-StepPracticalMethoImplementationdsof Single-Step Methods Systems of ODE’s Systems of ODE’s Multi-Step Methods* Multi-Step Methods* Question: How to decide whether the error is within tolerance? Evaluation of a step: Additional estimates: ∆ > : Step size is too large for accuracy. I handle to monitor the error Subdivide the interval. I further efficient algorithms ∆ << : Step size is inefficient! Runge-Kutta method with adaptive step size In an interval [xn, xn + h], Start with a large step size. (1) 5 Keep subdividing intervals whenever ∆ > . yn+1 = yn+1 + ch + higher order terms Fast marching over smooth segments and small steps in h Over two steps of size 2 , zones featured with rapid changes in y(x). h 5 y (2) = y + 2c + higher order terms Runge-Kutta-Fehlberg method n+1 n+1 2   With six function values, Difference of two estimates: An RK4 formula embedded in an RK5 formula 15 ∆ = y (1) y (2) ch5 I two independent estimates and an error estimate! n+1 − n+1 ≈ 16 (2) (1) (2) 16y y RKF45 in professional implementations Best available value: y = y ∆ = n+1− n+1 n∗+1 n+1 − 15 15 Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 329, Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 330, Single-Step Methods Single-Step Methods Systems of ODE’s Practical Implementation of Single-Step Methods Systems of ODE’s Practical Implementation of Single-Step Methods Systems of ODE’s Systems of ODE’s Multi-Step Methods* Multi-Step Methods* Methods for a single first order ODE State space formulation is directly applicable when directly applicable to a first order vector ODE the highest order derivatives can be solved explicitly. A typical IVP with an ODE system: The resulting form of the ODE’s: normal system of ODE’s dy Example: = f(x, y), y(x ) = y dx 0 0 d 2x dy dx 2 dx d 2y An n-th order ODE: convert into a system of first order ODE’s y 3 + 2x + 4 = 0 dt2 − dt dt dt dt2 Defining state vector z(x) = [y(x) y (x) y (n 1)(x)]T ,       r 0 − 3 2 3/2 · · · xy d y d y t dz e y + 2x + 1 = e− work out dx to form the state space equation. dt3 − dt2   (n 1) T T Initial condition: z(x0) = [y(x0) y 0(x0) y − (x0)] dx dy d2y · · · State vector: z(t) = x dt y dt dt2 A system of higher order ODE’s with the highest order derivatives With three trivial derivativesh z10 (t) = z2, z30 (it) = z4 and z40 (t) = z5 of orders n , n , n , , nk and the other two obtained from the given ODE’s, 1 2 3 · · · I Cast into the state space form with the state vector of dz we get the state space equations as dt = f(t, z). dimension n = n + n + n + + nk 1 2 3 · · ·

Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 331, Applied Mathematical Methods Numerical Solution of Ordinary Differential Equations 332, Single-Step Methods Single-Step Methods Multi-Step Methods* Practical Implementation of Single-Step Methods Points to note Practical Implementation of Single-Step Methods Systems of ODE’s Systems of ODE’s Multi-Step Methods* Multi-Step Methods* Single-step methods: every step a brand new IVP! Why not try to capture the trend? A typical multi-step formula: I Euler’s and Runge-Kutta methods I Step size adaptation yn+1 = yn + h[c0f (xn+1, yn+1) + c1f (xn, yn) I State space formulation of dynamic systems + c2f (xn 1, yn 1) + c3f (xn 2, yn 2) + ] − − − − · · · Determine coefficients by demanding the exactness for leading polynomial terms. Necessary Exercises: 1,2,5,6

Explicit methods: c0 = 0, evaluation easy, but involves extrapolation. Implicit methods: c = 0, difficult to evaluate, but better stability. 0 6 Predictor-corrector methods Example: Adams-Bashforth-Moulton method Applied Mathematical Methods ODE Solutions: Advanced Issues 333, Applied Mathematical Methods ODE Solutions: Advanced Issues 334, Stability Analysis Stability Analysis Outline Implicit Methods Stability Analysis Implicit Methods Stiff Differential Equations Stiff Differential Equations Boundary Value Problems Adaptive RK4 is an extremely successful methoBoundad.ry Value Problems But, its scope has a limitation. Focus of explicit methods (such as RK) is accuracy and efficiency. ODE Solutions: Advanced Issues The issue of stabilty is handled indirectly. Stability Analysis Implicit Methods Stabilty of explicit methods Stiff Differential Equations For the ODE system y0 = f(x, y), Euler’s method gives 2 Boundary Value Problems yn = yn + f(xn, yn)h + (h ). +1 O Taylor’s series of the actual solution: 2 y(xn ) = y(xn) + f(xn, y(xn))h + (h ) +1 O Discrepancy or error:

∆n+1 = yn+1 y(xn+1) − 2 = [yn y(xn)] + [f(xn, yn) f(xn, y(xn))]h + (h ) − − O ∂f 2 = ∆n + (xn, ¯yn)∆n h + (h ) (I + hJ)∆n ∂y O ≈  

Applied Mathematical Methods ODE Solutions: Advanced Issues 335, Applied Mathematical Methods ODE Solutions: Advanced Issues 336, Stability Analysis Stability Analysis Stability Analysis Implicit Methods Stability Analysis Implicit Methods Stiff Differential Equations Stiff Differential Equations Boundary Value Problems Boundary Value Problems

Euler’s step magnifies the error by a factor (I + hJ). 3 Using J loosely as the representative Jacobian, UNSTABLE 2 UNSTABLE

n RK2 ∆n (I + hJ) ∆ . Euler +1 ≈ 1 1 ) For stability, ∆n+1 0 as n . λ 0 → → ∞ Im(h O

Eigenvalues of (I + hJ) must fall within the unit circle −1 z = 1. By shift theorem, eigenvalues of hJ must fall RK4 | | −2 inside the unit circle with the centre at z0 = 1. − −3 −5 −4 −3 −2 −1 0 1 2 3 2Re (λ) Re(hλ) 1 + hλ < 1 h < − | | ⇒ λ 2 | | Figure: Stability regions of explicit methods Note: Same result for single ODE w 0 = λw, with complex λ. For second order Runge-Kutta method, Question: What do these stability regions mean with reference to h2λ2 the system eigenvalues? ∆n = 1 + hλ + ∆n +1 2 Question: How does the step size adaptation of RK4 operate on a   system with eigenvalues on the left half of complex plane? z2 Region of stability in the plane of z = hλ: 1 + z + 2 < 1 Step size adaptation tackles instability by its symptom!

Applied Mathematical Methods ODE Solutions: Advanced Issues 337, Applied Mathematical Methods ODE Solutions: Advanced Issues 338, Stability Analysis Stability Analysis Implicit Methods Implicit Methods Implicit Methods Implicit Methods Stiff Differential Equations Stiff Differential Equations Boundary Value Problems Boundary Value Problems Backward Euler’s method 2

1.5

1 STABLE yn+1 = yn + f(xn+1, yn+1)h STABLE 0.5 UNSTABLE ) λ Solve it? Is it worth solving? 0 Im(h O

−0.5

−1 ∆n+1 yn+1 y(xn+1) STABLE ≈ − −1.5

= [y y(x )] + h[f(x , y ) f(x , y(x ))] −2 n n n+1 n+1 n+1 n+1 −1.5 −1 −0.5 0 0.5 1 1.5 2 2.5 3 3.5 − − Re(hλ) = ∆n + hJ(xn+1, ¯yn+1)∆n+1 Notice the flip in the form of this equation. Figure: Stability region of backward Euler’s method

1 How to solve g(yn+1) = yn + hf(xn+1, yn+1) yn+1 = 0 for yn+1? ∆n+1 (I hJ)− ∆n − ≈ − Typical Newton’s iteration: Stability: eigenvalues of (I hJ) outside the unit circle z = 1 − | | (k+1) (k) 1 (k) (k) yn+1 = yn+1 + (I hJ)− yn yn+1 + hf xn+1, yn+1 2Re (λ) − − hλ 1 > 1 h > Semi-implicit Euler’s method hfor local solution: i | − | ⇒ λ 2 | | 1 yn = yn + h(I hJ)− f(xn , yn) Absolute stability for a stable ODE, i.e. one with Re (λ) < 0 +1 − +1

Applied Mathematical Methods ODE Solutions: Advanced Issues 339, Applied Mathematical Methods ODE Solutions: Advanced Issues 340, Stability Analysis Stability Analysis Stiff Differential Equations Implicit Methods Stiff Differential Equations Implicit Methods Stiff Differential Equations Stiff Differential Equations 2t 300t Boundary Value Problems e− e− Boundary Value Problems Example: IVP of a mass-spring-damper system: (c) c = 302, k = 600: x = −298

−3 −3 x¨ + cx˙ + kx = 0, x(0) = 0, x˙ (0) = 1 x 10 x 10 4 4 (a) c = 3, k = 2: x = e t e 2t − − 3 3 − 24t 25t (b) c = 49, k = 600: x = e− e− − 2 2

1 1

1 1 x x 0 0

0.8 0.8 −1 −1

0.6 0.6

−2 −2 0.4 0.4

0.2 0.2 −3 −3

x 0 x 0 −4 −4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 t t −0.2 −0.2 −0.4 −0.4 (c) With RK4 (d) With implicit Euler −0.6 −0.6

−0.8 −0.8

−1 −1 0 0.5 1 1.5 2 2.5 3 3.5 4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 t t Figure: Solutions of a mass-spring-damper system: stiff situation (a) Case of c = 3, k = 2 (b) Case of c = 49, k = 600 To solve stiff ODE systems, Figure: Solutions of a mass-spring-damper system: ordinary situations use implicit method, preferably with explicit Jacobian. Applied Mathematical Methods ODE Solutions: Advanced Issues 341, Applied Mathematical Methods ODE Solutions: Advanced Issues 342, Stability Analysis Stability Analysis Boundary Value Problems Implicit Methods Boundary Value Problems Implicit Methods Stiff Differential Equations Stiff Differential Equations Boundary Value Problems Boundary Value Problems A paradigm shift from the initial value problems Shooting method I A ball is thrown with a particular velocity. What trajectory follows the strategy to adjust trials to hit a target. does the ball follow? Consider the 2-point BVP I How to throw a ball such that it hits a particular window at a neighbouring house after 15 seconds? y0 = f(x, y), g1(y(a)) = 0, g2(y(b)) = 0,

where g Rn1 , g Rn2 and n + n = n. Two-point BVP in ODE’s: 1 ∈ 2 ∈ 1 2 boundary conditions at two values of the independent I Parametrize initial state: y(a) = h(p) with p R n2 . variable ∈ I Guess n2 values of p to define IVP Methods of solution y0 = f(x, y), y(a) = h(p). I Shooting method I Finite difference (relaxation) method I Solve this IVP for [a, b] and evaluate y(b). I I Finite element method Define error vector E(p) = g2(y(b)).

Applied Mathematical Methods ODE Solutions: Advanced Issues 343, Applied Mathematical Methods ODE Solutions: Advanced Issues 344, Stability Analysis Stability Analysis Boundary Value Problems Implicit Methods Boundary Value Problems Implicit Methods Stiff Differential Equations Stiff Differential Equations Boundary Value Problems Finite difference (relaxation) method Boundary Value Problems Objective: To solve E(p) = 0 adopts a global perspective.

∂E From current vector p, n2 perturbations as p + ei δ: Jacobian ∂p 1. Discretize domain [a, b]: grid of points a = x0 < x1 < x2 < < xN 1 < xN = b. Each Newton’s step: solution of n2 + 1 initial value · · · − problems! Function values y(xi ): n(N + 1) unknowns 2. Replace the ODE over intervals by finite difference equations. Considering mid-points, a typical (vector) FDE: I Computational cost xi + xi 1 yi + yi 1 I yi yi 1 hf − , − = 0, for i = 1, 2, 3, , N Convergence not guaranteed (initial guess important) − − − 2 2 · · ·   nN (scalar) equations Merits of shooting method 3. Assemble additional n equations from boundary conditions. I Very few parameters to start 4. Starting from a guess solution over the grid, solve this system. I In many cases, it is found quite efficient. (Sparse Jacobian is an advantage.) Iterative schemes for solution of systems of linear equations. Applied Mathematical Methods ODE Solutions: Advanced Issues 345, Applied Mathematical Methods Existence and Uniqueness Theory 346, Stability Analysis Well-Posedness of Initial Value Problems Points to note Implicit Methods Outline Uniqueness Theorems Stiff Differential Equations Extension to ODE Systems Boundary Value Problems Closure

I Numerical stability of ODE solution methods I Computational cost versus better stability of implicit methods I Multiscale responses leading to stiffness: failure of explicit Existence and Uniqueness Theory methods Well-Posedness of Initial Value Problems Uniqueness Theorems I Implicit methods for stiff systems Extension to ODE Systems I Shooting method for two-point boundary value problems Closure I Relaxation method for boundary value problems

Necessary Exercises: 1,2,3,4,5

Applied Mathematical Methods Existence and Uniqueness Theory 347, Applied Mathematical Methods Existence and Uniqueness Theory 348, Well-Posedness of Initial Value Problems Well-Posedness of Initial Value Problems Well-Posedness of Initial Value ProblemsUniqueness Theorems Well-Posedness of Initial Value ProblemsUniqueness Theorems Extension to ODE Systems Extension to ODE Systems Pierre Simon de Laplace (1749-1827):Closure Initial value problem Closure ”We may regard the present state of the y = f (x, y), y(x ) = y universe as the effect of its past and the 0 0 0 cause of its future. An intellect which at a From (x, y), the trajectory develops according to y 0 = f (x, y). certain moment would know all forces that The new point: (x + δx, y + f (x, y)δx) The slope now: f (x + δx, y + f (x, y)δx) set nature in motion, and all positions of all items of which nature is composed, if this Question: Was the old direction of approach valid? With δx 0, directions appropriate, if intellect were also vast enough to submit → lim f (x, y) = f (x¯, y(x¯)), these data to analysis, it would embrace in a x x¯ single formula the movements of the greatest → i.e. if f (x, y) is continuous. bodies of the universe and those of the If f (x, y) = , then y = and trajectory is vertical. ∞ 0 ∞ tiniest atom; for such an intellect nothing For the same value of x, several values of y! would be uncertain and the future just like y(x) not a function, unless f (x, y) = , i.e. f (x, y) is bounded. the past would be present before its eyes.” 6 ∞ Applied Mathematical Methods Existence and Uniqueness Theory 349, Applied Mathematical Methods Existence and Uniqueness Theory 350, Well-Posedness of Initial Value Problems Well-Posedness of Initial Value Problems Well-Posedness of Initial Value ProblemsUniqueness Theorems Well-Posedness of Initial Value ProblemsUniqueness Theorems Extension to ODE Systems Extension to ODE Systems Peano’s theorem: If f (x, y) is continuous Closureand bounded in a Example: Closure rectangle R = (x, y) : x x0 < h, y y0 < k , with y 1 { | − | | − | } y 0 = − , y(0) = 1 f (x, y) M < , then the IVP y = f (x, y), y(x ) = y has a x | | ≤ ∞ 0 0 0 solution y(x) defined in a neighbourhood of x0. y 1 Function f (x, y) = −x undefined at (0, 1). y y    

(x0,y0)  (x ,y )  0 0

 Premises of existence theorem not satisfied.                       

y +k 

                       y +k

0 

                       0

¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¡ ¡                        

¥ ¤ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡                        

¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¥ ¤

 Mh                       

k ¡ ¡ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ 

                       k

¡ ¡ ¡ ¡ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤                         Mh

¡ ¡ ¡ ¡ ¡ ¡ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤                        

¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¥ ¤ ¡ ¥ ¤ ¡ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤ ¥ ¤                        

§ ¦ § ¦ § ¦ § ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢ £ ¢ £ ¢                        

§ ¦ § ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢ y y

 But, premises here are sufficient, not necessary!                       

£ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ § ¦ 0 0                        

£ ¢ £ ¢ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ £ ¢§ ¦ § ¦ § ¦ § ¦                        

£ ¢ £ ¢ £ ¢ £ ¢ £ ¢§ ¦ £ ¢§ ¦ £ ¢ § ¦ £ ¢ § ¦ £ ¢ § ¦ £ ¢ § ¦ £ ¢ § ¦ § ¦ § ¦ § ¦ § ¦ § ¦                         Mh

£ ¢ £ ¢ £ ¢ £ ¢ £ ¢ £ ¢ £ ¢ § ¦ £ ¢ § ¦ £ ¢ § ¦ £ ¢ § ¦ £ ¢ § ¦ § ¦ § ¦ § ¦ § ¦ § ¦ § ¦ § ¦                        

£ ¢ £ ¢ £ ¢ £ ¢ £ ¢ £ ¢ £ ¢ £ ¢ £ ¢ § ¦ £ ¢ § ¦ £ ¢ § ¦ § ¦ § ¦ § ¦ § ¦ § ¦ § ¦ § ¦ § ¦ § ¦ k 

                       k Mh

                       

                       



                       Result inconclusive.

                       

y −k y −k  0 0 













h h  h h

 The IVP has solutions: y(x) = 1 + cx for all values of c.

















© ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨ © ¨                                                                  The solution is not unique.

O x 0 −h x 0 x 0+h x O x 0−h x0 x0+h x

2 (a) Mh <= k (b) Mh >= k Example: y = y , y(0) = 0 0 | | Figure: Regions containing the trajectories Existence theorem guarantees a solution. But, there are two solutions: Guaranteed neighbourhood: y(x) = 0 and y(x) = sgn(x) x 2/4. [x δ, x + δ], where δ = min(h, k ) > 0 0 − 0 M

Applied Mathematical Methods Existence and Uniqueness Theory 351, Applied Mathematical Methods Existence and Uniqueness Theory 352, Well-Posedness of Initial Value Problems Well-Posedness of Initial Value Problems Well-Posedness of Initial Value ProblemsUniqueness Theorems Well-Posedness of Initial Value ProblemsUniqueness Theorems Extension to ODE Systems Extension to ODE Systems Closure Closure Physical system to mathematical model Continuous dependence on initial condition I Mathematical solution Suppose that for IVP y 0 = f (x, y), y(x0) = y0, I Interpretation about the physical system I unique solution: y1(x). Meanings of non-uniqueness of a solution Applying a small perturbation to the initial condition, the new IVP: y = f (x, y), y(x ) = y +  I Mathematical model admits of extraneous solution(s)? 0 0 0 I unique solution: y (x) I Physical system itself can exhibit alternative behaviours? 2 Question: By how much y2(x) differs from y1(x) for x > x0? Indeterminacy of the solution Large difference: solution sensitive to initial condition I Mathematical model of the system is not complete. I Practically unreliable solution The initial value problem is not well-posed. Well-posed IVP: After existence, next important question: An initial value problem is said to be well-posed if there exists a solution to it, the solution is unique and it Uniqueness of a solution depends continuously on the initial conditions. Applied Mathematical Methods Existence and Uniqueness Theory 353, Applied Mathematical Methods Existence and Uniqueness Theory 354, Well-Posedness of Initial Value Problems Well-Posedness of Initial Value Problems Uniqueness Theorems Uniqueness Theorems Uniqueness Theorems Uniqueness Theorems Extension to ODE Systems Extension to ODE Systems Lipschitz condition: Closure Closure x f (x, y) f (x, z) L y z E 0(x) E 0(x) | − | ≤ | − | 2L dx 2L(x x0) E(x) ≤ ⇒ E(x) ≤ − L: finite positive constant (Lipschitz constant) Zx0 2L(x x ) Theorem: If f (x, y) is a continuous function satisfying a Integrating, E(x) E(x0)e − 0 . ≤ Lipschitz condition on a strip Hence, S = (x, y) : a < x < b, < y < , then for any L(x x0) { −∞ ∞} y1(x) y2(x) e − (y1)0 (y2)0 . point (x0, y0) S, the initial value problem of | − | ≤ | − | ∈ y 0 = f (x, y), y(x0) = y0 is well-posed. Since x [a, b], eL(x x0) is finite. ∈ − Assume y1(x) and y2(x): solutions of the ODE y 0 = f (x, y) with L(x x ) (y ) (y ) =  y (x) y (x) e − 0  initial conditions y(x0) = (y1)0 and y(x0) = (y2)0 | 1 0 − 2 0| ⇒ | 1 − 2 | ≤ 2 Consider E(x) = [y1(x) y2(x)] . − continuous dependence of the solution on initial condition E 0(x) = 2(y y )(y 0 y 0 ) = 2(y y )[f (x, y ) f (x, y )] 1 − 2 1 − 2 1 − 2 1 − 2 In particular, (y ) = (y ) = y y (x) = y (x) x [a, b]. Applying Lipschitz condition, 1 0 2 0 0 ⇒ 1 2 ∀ ∈ 2 E 0(x) 2L(y y ) = 2LE(x). The initial value problem is well-posed. | | ≤ 1 − 2 Need to consider the case of E (x) 0 only. 0 ≥

Applied Mathematical Methods Existence and Uniqueness Theory 355, Applied Mathematical Methods Existence and Uniqueness Theory 356, Well-Posedness of Initial Value Problems Well-Posedness of Initial Value Problems Uniqueness Theorems Uniqueness Theorems Extension to ODE Systems Uniqueness Theorems Extension to ODE Systems Extension to ODE Systems A weaker theorem (hypotheses are stronger):Closure Closure ∂f For ODE System Picard’s theorem: If f (x, y) and ∂y are continuous and bounded on a rectangle dy = f(x, y), y(x ) = y R = (x, y) : a < x < b, c < y < d , then for every dx 0 0 { } (x , y ) R, the IVP y = f (x, y), y(x ) = y has a 0 0 ∈ 0 0 0 unique solution in some neighbourhood x x h. | − 0| ≤ I Lipschitz condition: From the mean value theorem, f(x, y) f(x, z) L y z ∂f k − k ≤ k − k f (x, y ) f (x, y ) = (ξ)(y y ). 1 − 2 ∂y 1 − 2 I Scalar function E(x) generalized as

2 T ∂f E(x) = y1(x) y2(x) = (y1 y2) (y1 y2) With Lipschitz constant L = sup ∂y , k − k − − ∂f ∂f Lipschitz condition is satisfied ‘lavishly’ ! I Partial derivative replaced by the Jacobian A = ∂y ∂y I Note: All these theorems give only sufficient conditions! Boundedness to be inferred from the boundedness of its norm Hypotheses of Picard’s theorem Lipschitz condition With these generalizations, the formulations work as usual. ⇒ ⇒ Well-posedness Existence and uniqueness ⇒ Applied Mathematical Methods Existence and Uniqueness Theory 357, Applied Mathematical Methods Existence and Uniqueness Theory 358, Well-Posedness of Initial Value Problems Well-Posedness of Initial Value Problems Extension to ODE Systems Uniqueness Theorems Closure Uniqueness Theorems Extension to ODE Systems Extension to ODE Systems Closure Closure IVP of linear first order ODE system A practical by-product of existence and uniqueness results: y0 = A(x)y + g(x), y(x0) = y0 I important results concerning the solutions

Rate function: f(x, y) = A(x)y + g(x) A sizeable segment of current research: ill-posed problems Continuity and boundedness of the coefficient functions I Dynamics of some nonlinear systems in A(x) and g(x) are sufficient for well-posedness. I Chaos: sensitive dependence on initial conditions An n-th order linear ordinary differential equation For boundary value problems, (n) (n 1) (n 2) y +P1(x)y − +P2(x)y − + +Pn 1(x)y 0+Pn(x)y = R(x) · · · − No general criteria for existence and uniqueness State vector: z = [y y y y (n 1)]T 0 00 · · · − With z10 = z2, z20 = z3, , zn0 1 = zn and zn0 from the ODE, Note: Taking clue from the shooting method, a BVP in ODE’s · · · − I can be visualized as a complicated root-finding problem! state space equation in the form z0 = A(x)z + g(x)

Continuity and boundedness of P1(x), P2(x), , Pn(x) Multiple solutions or non-existence of solution is no surprise. · · · and R(x) guarantees well-posedness.

Applied Mathematical Methods Existence and Uniqueness Theory 359, Applied Mathematical Methods First Order Ordinary Differential Equations 360, Well-Posedness of Initial Value Problems Formation of Differential Equations and Their Solutions Points to note Uniqueness Theorems Outline Separation of Variables Extension to ODE Systems ODE’s with Rational Slope Functions Closure Some Special ODE’s Exact Differential Equations and Reduction to the Exact Form First Order Linear (Leibnitz) ODE and Associated Forms Orthogonal Trajectories Modelling and Simulation I For a solution of initial value problems, questions of existence, uniqueness and continuous dependence on initial condition are First Order Ordinary Differential Equations of crucial importance. Formation of Differential Equations and Their Solutions Separation of Variables I These issues pertain to aspects of practical relevance ODE’s with Rational Slope Functions regarding a physical system and its dynamic simulation Some Special ODE’s I Lipschitz condition is the tightest (avaliable) criterion for Exact Differential Equations and Reduction to the Exact Form deciding these questions regarding well-posedness First Order Linear (Leibnitz) ODE and Associated Forms Orthogonal Trajectories Modelling and Simulation Necessary Exercises: 1,2 Applied Mathematical Methods First Order Ordinary Differential Equations 361, Applied Mathematical Methods First Order Ordinary Differential Equations 362, Formation of Differential Equations and Their Solutions Formation of Differential Equations and Their Solutions Formation of Differential Equations andSeparationTheirof VariablesSolutions Separation of Variables Separation of Variables ODE’s with Rational Slope Functions ODE’s with Rational Slope Functions Some Special ODE’s Some Special ODE’s Exact Differential Equations and Reduction to the Exact Form ODE form with separable variables: Exact Differential Equations and Reduction to the Exact Form A differential equation represents a class of Firstfunctions.Order Linear (Leibnitz) ODE and Associated Forms First Order Linear (Leibnitz) ODE and Associated Forms Orthogonal Trajectories Orthogonal Trajectories Modelling and Simulation Modelling and Simulation k dy φ(x) Example: y(x) = cx y 0 = f (x, y) = or ψ(y)dy = φ(x)dx ⇒ dx ψ(y) dy k 1 d2y k 2 With dx = ckx − and dx2 = ck(k 1)x − , − Solution as quadrature: d 2y dy 2 dy xy = x y dx2 dx − dx ψ(y)dy = φ(x)dx + c.   Z Z A compact ‘intrinsic’ description. Separation of variables through substitution Important terms Example: I Order and degree of differential equations y 0 = g(αx + βy + γ) I Homogeneous and non-homogeneous ODE’s Substitute v = αx + βy + γ to arrive at Solution of a differential equation dv dv = α + βg(v) x = + c I general, particular and singular solutions dx ⇒ α + βg(v) Z

Applied Mathematical Methods First Order Ordinary Differential Equations 363, Applied Mathematical Methods First Order Ordinary Differential Equations 364, Formation of Differential Equations and Their Solutions Formation of Differential Equations and Their Solutions ODE’s with Rational Slope FunctionsSeparation of Variables Some Special ODE’s Separation of Variables ODE’s with Rational Slope Functions ODE’s with Rational Slope Functions Some Special ODE’s Some Special ODE’s Exact Differential Equations and Reduction to the Exact Form Exact Differential Equations and Reduction to the Exact Form f1(x, y) First Order Linear (Leibnitz) ODE and Associated Forms Clairaut’s equation First Order Linear (Leibnitz) ODE and Associated Forms y = Orthogonal Trajectories Orthogonal Trajectories 0 Modelling and Simulation y = xy 0 + f (y 0) Modelling and Simulation f2(x, y)

If f1 and f2 are homogeneous functions of n-th degree, then Substitute p = y 0 and differentiate: substitution y = ux separates variables x and u. dp dp dp dy φ1(y/x) du φ1(u) dx φ2(u) p = p + x + f 0(p) [x + f 0(p)] = 0 = u+x = = du dx dx ⇒ dx dx φ (y/x) ⇒ dx φ (u) ⇒ x φ (u) uφ (u) 2 2 1 − 2 dp a1x+b1y+c1 For y = , coordinate shift dx = 0 means y 0 = p = m (constant) 0 a2x+b2y+c2 I dy dY family of straight lines y = mx + f (m) as general solution x = X + h, y = Y + k y 0 = = ⇒ dx dX Singular solution: produces dY a1X + b1Y + (a1h + b1k + c1) x = f 0(p) and y = f (p) pf 0(p) = . − − dX a2X + b2Y + (a2h + b2k + c2) Choose h and k such that Singular solution is the envelope of the family of straight lines that constitute the general solution. a1h + b1k + c1 = 0 = a2h + b2k + c2.

If the system is inconsistent, then substitute u = a2x + b2y. Applied Mathematical Methods First Order Ordinary Differential Equations 365, Applied Mathematical Methods First Order Ordinary Differential Equations 366, Formation of Differential Equations and Their Solutions Formation of Differential Equations and Their Solutions Some Special ODE’s Separation of Variables Exact Differential Equations and ReductionSeparation of Vtoariablesthe Exact Form ODE’s with Rational Slope Functions ODE’s with Rational Slope Functions Some Special ODE’s Some Special ODE’s Second order ODE’s with the function notExact DifferentialappearingEquations and Reduction to the Exact Form Mdx + Ndy: an exact differential if Exact Differential Equations and Reduction to the Exact Form First Order Linear (Leibnitz) ODE and Associated Forms First Order Linear (Leibnitz) ODE and Associated Forms explicitly Orthogonal Trajectories Orthogonal Trajectories Modelling and Simulation ∂φ ∂φ Mo∂dellingM and∂SimulationN f (x, y , y ) = 0 M = and N = , or, = 0 00 ∂x ∂y ∂y ∂x Substitute y 0 = p and solve f (x, p, p0) = 0 for p(x). ∂M ∂N M(x, y)dx + N(x, y)dy = 0 is an exact ODE if ∂y = ∂x Second order ODE’s with independent variable not appearing ∂φ ∂φ explicitly With M(x, y) = ∂x and N(x, y) = ∂y , f (y, y 0, y 00) = 0 ∂φ ∂φ dx + dy = 0 dφ = 0. Use y 0 = p and ∂x ∂y ⇒ dp dp dy dp dp Solution: φ(x, y) = c y 00 = = = p f (y, p, p ) = 0. dx dy dx dy ⇒ dy Working rule:

Solve for p(y). φ1(x, y) = M(x, y)dx+g1(y) and φ2(x, y) = N(x, y)dy+g2(x) Resulting equation solved through a quadrature as Z Z Determine g1(y) and g2(x) from φ1(x, y) = φ2(x, y) = φ(x, y). dy dy M N = p(y) x = x + . If ∂ = ∂ , but ∂ (FM) = ∂ (FN)? dx ⇒ 0 p(y) ∂y 6 ∂x ∂y ∂x Z F : Integrating factor

Applied Mathematical Methods First Order Ordinary Differential Equations 367, Applied Mathematical Methods First Order Ordinary Differential Equations 368, Formation of Differential Equations and Their Solutions Formation of Differential Equations and Their Solutions First Order Linear (Leibnitz) ODE andSepaAssoration of Vciatedariables Forms First Order Linear (Leibnitz) ODE andSepaAssoration of Vciatedariables Forms ODE’s with Rational Slope Functions ODE’s with Rational Slope Functions Some Special ODE’s Some Special ODE’s General first order linear ODE: Exact Differential Equations and Reduction to the Exact Form Bernoulli’s equation Exact Differential Equations and Reduction to the Exact Form First Order Linear (Leibnitz) ODE and Associated Forms First Order Linear (Leibnitz) ODE and Associated Forms dy Orthogonal Trajectories dy Orthogonal Trajectories + P(x)y = Q(x)Modelling and Simulation + P(x)y = Q(x)yMok delling and Simulation dx dx 1 k dz k dy Leibnitz equation Substitution: z = y − , = (1 k)y − gives dx − dx For integrating factor F (x), dz + (1 k)P(x)z = (1 k)Q(x), dy d dF dx − − F (x) + F (x)P(x)y = [F (x)y] = F (x)P(x). dx dx ⇒ dx in the Leibnitz form. Riccati equation Separating variables, 2 dF y 0 = a(x) + b(x)y + c(x)y = P(x)dx ln F = P(x)dx. F ⇒ If one solution y (x) is known, then propose y(x) = y (x) + z(x). Z Z Z 1 1 2 y 0 (x) + z0(x) = a(x) + b(x)[y1(x) + z(x)] + c(x)[y1(x) + z(x)] Integrating factor: F (x) = eR P(x)dx 1 2 Since y10 (x) = a(x) + b(x)y1(x) + c(x)[y1(x)] , z (x) = [b(x) + 2c(x)y (x)]z(x) + c(x)[z(x)]2, yeR P(x)dx = Q(x)eR P(x)dx dx + C 0 1 Z in the form of Bernoulli’s equation. Applied Mathematical Methods First Order Ordinary Differential Equations 369, Applied Mathematical Methods First Order Ordinary Differential Equations 370, Formation of Differential Equations and Their Solutions Formation of Differential Equations and Their Solutions Orthogonal Trajectories Separation of Variables Points to note Separation of Variables ODE’s with Rational Slope Functions ODE’s with Rational Slope Functions Some Special ODE’s Some Special ODE’s Exact Differential Equations and Reduction to the Exact Form Exact Differential Equations and Reduction to the Exact Form In xy-plane, one-parameter equation φ(x, y,Firstc) Order= 0:Linear (Leibnitz) ODE and Associated Forms First Order Linear (Leibnitz) ODE and Associated Forms Orthogonal Trajectories Orthogonal Trajectories a family of curves Modelling and Simulation Modelling and Simulation I Meaning and solution of ODE’s Differential equation of the family of curves: I Separating variables dy I Exact ODE’s and integrating factors = f1(x, y) dx I Linear (Leibnitz) equations Slope of curves orthogonal to φ(x, y, c) = 0: I Orthogonal families of curves

dy 1 = dx −f1(x, y) Necessary Exercises: 1,3,5,7 Solving this ODE, another family of curves ψ(x, y, k) = 0. Orthogonal trajectories

If φ(x, y, c) = 0 represents the potential lines (contours), then ψ(x, y, k) = 0 will represent the streamlines!

Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 371, Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 372, Introduction Introduction Outline Homogeneous Equations with Constant Coefficients Introduction Homogeneous Equations with Constant Coefficients Euler-Cauchy Equation Euler-Cauchy Equation Theory of the Homogeneous Equations Theory of the Homogeneous Equations Basis for Solutions Second order ODE: Basis for Solutions f (x, y, y 0, y 00) = 0 Special case of a linear (non-homogeneous) ODE: Second Order Linear Homogeneous ODE’s y + P(x)y + Q(x)y = R(x) Introduction 00 0 Homogeneous Equations with Constant Coefficients Non-homogeneous linear ODE with constant coefficients: Euler-Cauchy Equation

Theory of the Homogeneous Equations y 00 + ay 0 + by = R(x) Basis for Solutions For R(x) = 0, linear homogeneous differential equation

y 00 + P(x)y 0 + Q(x)y = 0

and linear homogeneous ODE with constant coefficients

y 00 + ay 0 + by = 0 Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 373, Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 374, Introduction Introduction Homogeneous Equations with ConstantHomogeneousCoefficientsEquations with Constant Coefficients Homogeneous Equations with ConstantHomogeneousCoefficientsEquations with Constant Coefficients Euler-Cauchy Equation Euler-Cauchy Equation Theory of the Homogeneous Equations Theory of the Homogeneous Equations Basis for Solutions Basis for Solutions I 2 a Real and equal (a = 4b): λ1 = λ2 = λ = 2 y 00 + ay 0 + by = 0 − λx only solution in hand: y1 = e Assume λx λx 2 λx Method to develop another solution? y = e y 0 = λe and y 00 = λ e . I λx ⇒ Verify that y2 = xe is another solution. 2 λx Substitution: (λ + aλ + b)e = 0 λx y(x) = c1y1(x) + c2y2(x) = (c1 + c2x)e Auxiliary equation: λ2 + aλ + b = 0 I Complex conjugate (a2 < 4b): λ = a iω 1,2 − 2  Solve for λ1 and λ2: ( a +iω)x ( a iω)x Solutions: eλ1x and eλ2x y(x) = c1e − 2 + c2e − 2 − ax = e− 2 [c1(cos ωx + i sin ωx) + c2(cos ωx i sin ωx)] Three cases ax − = e 2 [A cos ωx + B sin ωx], I Real and distinct (a2 > 4b): λ = λ − 1 6 2 with A = c1 + c2, B = i(c1 c2). λ1x λ2x y(x) = c1y1(x) + c2y2(x) = c1e + c2e −ax I A third form: y(x) = Ce− 2 cos(ωx α) −

Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 375, Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 376, Introduction Introduction Euler-Cauchy Equation Homogeneous Equations with Constant Coefficients Theory of the Homogeneous EquationsHomogeneous Equations with Constant Coefficients Euler-Cauchy Equation Euler-Cauchy Equation Theory of the Homogeneous Equations Theory of the Homogeneous Equations Basis for Solutions Basis for Solutions 2 x y 00 + axy 0 + by = 0 Substituting y = x k , auxiliary (or indicial) equation: y 00 + P(x)y 0 + Q(x)y = 0 k2 + (a 1)k + b = 0 Well-posedness of its IVP: − The initial value problem of the ODE, with arbitrary 2 initial conditions y(x0) = Y0, y 0(x0) = Y1, has a unique 1. Roots real and distinct [(a 1) > 4b]: k1 = k2. − 6 solution, as long as P(x) and Q(x) are continuous in the k1 k2 y(x) = c1x + c2x . interval under question. 2 a 1 2. Roots real and equal [(a 1) = 4b]: k = k = k = − . − 1 2 − 2 k y(x) = (c1 + c2 ln x)x . At least two linearly independent solutions: 2 a 1 I 3. Roots complex conjugate [(a 1) < 4b]: k = − iν. y1(x): IVP with initial conditions y(x0) = 1, y 0(x0) = 0 − 1,2 − 2  a 1 a 1 I − − y2(x): IVP with initial conditions y(x0) = 0, y 0(x0) = 1 y(x) = x− 2 [A cos(ν ln x)+B sin(ν ln x)] = Cx − 2 cos(ν ln x α). − c1y1(x) + c2y2(x) = 0 c1 = c2 = 0 Alternative approach: substitution ⇒ dx dt 1 At most two linearly independent solutions? x = et t = ln x, = et = x and = , etc. ⇒ dt dx x Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 377, Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 378, Introduction Introduction Theory of the Homogeneous EquationsHomogeneous Equations with Constant Coefficients Theory of the Homogeneous EquationsHomogeneous Equations with Constant Coefficients Euler-Cauchy Equation Euler-Cauchy Equation Theory of the Homogeneous Equations Theory of the Homogeneous Equations Wronskian of two solutions y1(x) and y2(xBasis): for Solutions If y1(x) and y2(x) are linearly dependent, thenBasis foyr2Solutions= ky1.

y1 y2 W (y1, y2) = y1y20 y2y10 = y1(ky10 ) (ky1)y10 = 0 W (y , y ) = = y y 0 y y 0 − − 1 2 y y 1 2 − 2 1 10 20 In particular, W [y (x ), y (x )] = 0 1 0 2 0 Solutions y1 and y2 are linea rly dep endent, if and only if x0 ∃ Conversely, if there is a value x0, where such that W [y1(x0), y2(x0)] = 0. y1(x0) y2(x0) I W [y1(x0), y2(x0)] = 0 W [y1(x), y2(x)] = 0 x. W [y1(x0), y2(x0)] = = 0, y10 (x0) y20 (x0) I ⇒ ∀ W [y1(x1), y2(x1)] = 0 W [y1(x), y2(x)] = 0 x, and y1(x) 6 ⇒ 6 ∀ then for and y2(x) are linearly independent solutions. y (x ) y (x ) c 1 0 2 0 1 = 0, Complete solution: y (x ) y (x ) c  10 0 20 0   2  If y1(x) and y2(x) are two linearly independent solutions, coefficient matrix is singular. then the general solution is c Choose non-zero 1 and frame y(x) = c y + c y , satisfying c 1 1 2 2  2  y(x) = c1y1(x) + c2y2(x). IVP y 00 + Py 0 + Qy = 0, y(x0) = 0, y 0(x0) = 0. And, the general solution is the complete solution . Therefore, y(x) = 0 y and y are linearly dependent. ⇒ 1 2 No third linearly independent solution. No singular solution.

Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 379, Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 380, Introduction Introduction Theory of the Homogeneous EquationsHomogeneous Equations with Constant Coefficients Basis for Solutions Homogeneous Equations with Constant Coefficients Euler-Cauchy Equation Euler-Cauchy Equation Theory of the Homogeneous Equations Theory of the Homogeneous Equations Basis for Solutions For completely describing the solutions, we Basisneedfor Solutions Pick a candidate solution Y (x), choose a point x0, evaluate two linearly independent solutions. functions y1, y2, Y and their derivatives at that point, frame No guaranteed procedure to identify two basis members! y1(x0) y2(x0) C1 Y (x0) = If one solution y (x) is available, then to find another? y10 (x0) y20 (x0) C2 Y 0(x0) 1       Reduction of order C and ask for solution 1 . Assume the second solution as C  2  y2(x) = u(x)y1(x) Unique solution for C1, C2. Hence, particular solution

and determine u(x) such that y2(x) satisfies the ODE. y ∗(x) = C1y1(x) + C2y2(x)

is the “unique” solution of the IVP u00y1 + 2u0y10 + uy100 + P(u0y1 + uy10 ) + Quy1 = 0

y 00 + Py 0 + Qy = 0, y(x0) = Y (x0), y 0(x0) = Y 0(x0). u00y + 2u0y 0 + Pu0y + u(y 00 + Py 0 + Qy ) = 0. ⇒ 1 1 1 1 1 1 But, that is the candidate function Y (x)! Hence, Y (x) = y (x). ∗ Since y100 + Py10 + Qy1 = 0, we have y1u00 + (2y10 + Py1)u0 = 0 Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 381, Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 382, Introduction Introduction Basis for Solutions Homogeneous Equations with Constant Coefficients Basis for Solutions Homogeneous Equations with Constant Coefficients Euler-Cauchy Equation Euler-Cauchy Equation Theory of the Homogeneous Equations Theory of the Homogeneous Equations y10 Function space perspective: Denoting u0 = U, U0 + (2 + P)U = 0. Basis for Solutions Basis for Solutions y1 Operator ‘D’ means differentiation, operates on an infinite Rearrangement and integration of the reduced equation: dimensional function space as a linear transformation. I It maps all constant functions to zero. dU dy1 2 R Pdx + 2 + Pdx = 0 Uy1 e = C = 1 (choose). I It has a one-dimensional null space. U y1 ⇒ Second derivative or D 2 is an operator that has a two-dimensional Then, null space, c + c x, with basis 1, x . 1 2 { } 1 R Pdx Examples of composite operators u0 = U = 2 e− , y1 I ax (D + a) has a null space ce− . Integrating, I a (xD + a) has a null space cx − . 1 Pdx R 2 u(x) = 2 e− dx, A second order linear operator D + P(x)D + Q(x) possesses a y1 Z two-dimensional null space. and I 2 1 R Pdx Solution of [D + P(x)D + Q(x)]y = 0: description of the y (x) = y (x) e− dx. 2 1 y 2 null space, or a basis for it.. Z 1 I Analogous to solution of Ax = 0, i.e. development of a basis Note: The factor u(x) is never constant! for Null(A).

Applied Mathematical Methods Second Order Linear Homogeneous ODE’s 383, Applied Mathematical Methods Second Order Linear Non-Homogeneous ODE’s 384, Introduction Linear ODE’s and Their Solutions Points to note Homogeneous Equations with Constant Coefficients Outline Method of Undetermined Coefficients Euler-Cauchy Equation Method of Variation of Parameters Theory of the Homogeneous Equations Closure Basis for Solutions

I Second order linear homogeneous ODE’s I Wronskian and related results Second Order Linear Non-Homogeneous ODE’s I Solution basis Linear ODE’s and Their Solutions I Reduction of order Method of Undetermined Coefficients I Null space of a differential operator Method of Variation of Parameters Closure

Necessary Exercises: 1,2,3,7,8 Applied Mathematical Methods Second Order Linear Non-Homogeneous ODE’s 385, Applied Mathematical Methods Second Order Linear Non-Homogeneous ODE’s 386, Linear ODE’s and Their Solutions Linear ODE’s and Their Solutions Linear ODE’s and Their Solutions Method of Undetermined Coefficients Linear ODE’s and Their Solutions Method of Undetermined Coefficients Method of Variation of Parameters Method of Variation of Parameters Closure Closure The Complete Analogy Procedure to solve y 00 + P(x)y 0 + Q(x)y = R(x) 1. First, solve the corresponding homogeneous equation, obtain a Table: Linear systems and mappings: algebraic and differential basis with two solutions and construct

In ordinary vector space In infinite-dimensional function space yh(x) = c1y1(x) + c2y2(x).

Ax = b y 00 + Py 0 + Qy = R 2. Next, find one particular solution yp(x) of the NHE and The system is consistent. P(x), Q(x), R(x) are continuous. compose the complete solution A solution x∗ A solution yp(x) Alternative solution: ¯x Alternative solution: y¯(x) y(x) = yh(x) + yp(x) = c1y1(x) + c2y2(x) + yp(x). ¯x x∗ satisfies Ax = 0, y¯(x) yp(x) satisfies y 00 + Py 0 + Qy = 0, 3. If some initial or boundary conditions are known, they can be − − 2 is in null space of A. is in null space of D + P(x)D + Q(x). imposed now to determine c1 and c2. Complete solution: Complete solution: Caution: If y1 and y2 are two solutions of the NHE, then x = x∗ + i ci (x0)i yp(x) + i ci yi (x) do not expect c1y1 + c2y2 to satisfy the equation. Methodology: Methodology: P P Implication of linearity or superposition: Find null space of A Find null space of D 2 + P(x)D + Q(x) With zero initial conditions, if y1 and y2 are responses i.e. basis members (x0)i . i.e. basis members yi (x). due to inputs R1(x) and R2(x), respectively, then the Find x∗ and compose. Find yp(x) and compose. response due to input c1R1 + c2R2 is c1y1 + c2y2.

Applied Mathematical Methods Second Order Linear Non-Homogeneous ODE’s 387, Applied Mathematical Methods Second Order Linear Non-Homogeneous ODE’s 388, Linear ODE’s and Their Solutions Linear ODE’s and Their Solutions Method of Undetermined CoefficientsMethod of Undetermined Coefficients Method of Undetermined CoefficientsMethod of Undetermined Coefficients Method of Variation of Parameters Method of Variation of Parameters Closure Closure Example: y 00 + ay 0 + by = R(x) 3x (a) y 00 6y 0 + 5y = e n − 3x I What kind of function to propose as yp(x) if R(x) = x ? (b) y 00 5y 0 + 6y = e − 3x I And what if R(x) = eλx ? (c) y 00 6y 0 + 9y = e − I n λx 3x If R(x) = x + e , i.e. in the form k1R1(x) + k2R2(x)? In each case, the first official proposal: yp = ke The principle of superposition (linearity) (a) y(x) = c ex + c e5x e3x /4 1 2 − 2x 3x 3x (b) y(x) = c1e + c2e + xe Table: Candidate solutions for linear non-homogeneous ODE’s 3x 3x 1 2 3x (c) y(x) = c1e + c2xe + 2 x e RHS function R(x) Candidate solution yp(x) Modification rule pn(x) qn(x) I λx eλx keλx If the candidate function (ke , k1 cos ωx + k2 sin ωx or k eλx cos ωx + k eλx sin ωx) is a solution of the corresponding cos ωx or sin ωx k1 cos ωx + k2 sin ωx 1 2 λx λx λx λx HE; with λ, iω or λ iω (respectively) satisfying the e cos ωx or e sin ωx k1e cos ωx + k2e sin ωx   λx λx auxiliary equation; then modify it by multiplying with x. pn(x)e qn(x)e I λx λx pn(x) cos ωx or pn(x) sin ωx qn(x) cos ωx + rn(x) sin ωx In the case of λ being a double root, i.e. both e and xe λx λx λx λx 2 λx pn(x)e cos ωx or pn(x)e sin ωx qn(x)e cos ωx + rn(x)e sin ωx being solutions of the HE, choose yp = kx e . Applied Mathematical Methods Second Order Linear Non-Homogeneous ODE’s 389, Applied Mathematical Methods Second Order Linear Non-Homogeneous ODE’s 390, Linear ODE’s and Their Solutions Linear ODE’s and Their Solutions Method of Variation of Parameters Method of Undetermined Coefficients Method of Variation of Parameters Method of Undetermined Coefficients Method of Variation of Parameters Method of Variation of Parameters Closure Closure From yp = u1y1 + u2y2, Solution of the HE: yp0 = u10 y1 + u1y10 + u20 y2 + u2y20 .

yh(x) = c1y1(x) + c2y2(x), Condition u10 y1 + u20 y2 = 0 gives

in which c1 and c2 are constant ‘parameters’. yp0 = u1y10 + u2y20 .

For solution of the NHE, Differentiating, how about ‘variable parameters’? yp00 = u10 y10 + u20 y20 + u1y100 + u2y200. Propose Substitution into the ODE: yp(x) = u1(x)y1(x) + u2(x)y2(x) u10 y10 +u20 y20 +u1y100+u2y200+P(x)(u1y10 +u2y20 )+Q(x)(u1y1+u2y2) = R(x) and force yp(x) to satisfy the ODE. Rearranging, A single second order ODE in u1(x) and u2(x). We need one more condition to fix them. u10 y10 +u20 y20 +u1(y100+P(x)y10 +Q(x)y1)+u2(y200+P(x)y20 +Q(x)y2) = R(x).

As y1 and y2 satisfy the associated HE, u10 y10 + u20 y20 = R(x)

Applied Mathematical Methods Second Order Linear Non-Homogeneous ODE’s 391, Applied Mathematical Methods Second Order Linear Non-Homogeneous ODE’s 392, Linear ODE’s and Their Solutions Linear ODE’s and Their Solutions Method of Variation of Parameters Method of Undetermined Coefficients Points to note Method of Undetermined Coefficients Method of Variation of Parameters Method of Variation of Parameters Closure Closure

y y u 0 1 2 10 = y y u R  10 20   20    I Function space perspective of linear ODE’s Since Wronskian is non-zero, this system has unique solution I Method of undetermined coefficients I y2R y1R Method of variation of parameters u0 = and u0 = . 1 − W 2 W Direct quadrature:

y (x)R(x) y (x)R(x) Necessary Exercises: 1,3,5,6 u (x) = 2 dx and u (x) = 1 dx 1 − W [y (x), y (x)] 2 W [y (x), y (x)] Z 1 2 Z 1 2

In contrast to the method of undetermined multipliers, variation of parameters is general. It is applicable for all continuous functions as P(x), Q(x) and R(x). Applied Mathematical Methods Higher Order Linear ODE’s 393, Applied Mathematical Methods Higher Order Linear ODE’s 394, Theory of Linear ODE’s Theory of Linear ODE’s Outline Homogeneous Equations with Constant Coefficients Theory of Linear ODE’s Homogeneous Equations with Constant Coefficients Non-Homogeneous Equations Non-Homogeneous Equations Euler-Cauchy Equation of Higher Order Euler-Cauchy Equation of Higher Order

(n) (n 1) (n 2) y +P1(x)y − +P2(x)y − + +Pn 1(x)y 0+Pn(x)y = R(x) · · · − General solution: y(x) = yh(x) + yp(x), where I Higher Order Linear ODE’s yp(x): a particular solution I Theory of Linear ODE’s yh(x): general solution of corresponding HE (n) (n 1) (n 2) Homogeneous Equations with Constant Coefficients y +P1(x)y − +P2(x)y − + +Pn 1(x)y 0+Pn(x)y = 0 Non-Homogeneous Equations · · · − For the HE, suppose we have n solutions y (x), y (x), , yn(x). Euler-Cauchy Equation of Higher Order 1 2 · · · Assemble the state vectors in matrix

y y yn 1 2 · · · y y y 10 20 · · · n0  y y y  Y(x) = 100 200 · · · n00 .  . . .. .   . . . .   (n 1) (n 1) (n 1)   y − y − yn −   1 2 · · ·  Wronskian:   W (y , y , , yn) = det[Y(x)] 1 2 · · ·

Applied Mathematical Methods Higher Order Linear ODE’s 395, Applied Mathematical Methods Higher Order Linear ODE’s 396, Theory of Linear ODE’s Theory of Linear ODE’s Theory of Linear ODE’s Homogeneous Equations with Constant Coefficients Homogeneous Equations with ConstantHomogeneousCoefficientsEquations with Constant Coefficients Non-Homogeneous Equations Non-Homogeneous Equations Euler-Cauchy Equation of Higher Order Euler-Cauchy Equation of Higher Order I If solutions y (x), y (x), , yn(x) of HE are linearly 1 2 · · · dependent, then for a non-zero k R n, (n) (n 1) (n 2) y + a1y − + a2y − + + an 1y 0 + any = 0 n n ∈ · · · − (j) λx ki yi (x) = 0 ki y (x) = 0 for j = 1, 2, 3, , (n 1) With trial solution y = e , the auxiliary equation: ⇒ i · · · − i=1 i=1 X X n n 1 n 2 [Y(x)]k = 0 [Y(x)] is singular, λ + a1λ − + a2λ − + + an 1λ + an = 0 ⇒ ⇒ · · · − W [y1(x), y2(x), , yn(x)] = 0. ⇒ · · · Construction of the basis: I If Wronskian is zero at x = x0, then Y(x0) is singular and a γx n 1. For every simple real root λ = γ, e is a solution. non-zero k Null[Y(x0)] gives i=1 ki yi (x) = 0, implying ∈ 2. For every simple pair of complex roots λ = µ iω, y1(x), y2(x), , yn(x) to be linearly dependent. µx µx  I · · · P e cos ωx and e sin ωx are linearly independent solutions. Zero Wronskian at some x = x0 implies zero Wronskian γx γx 2 γx everywhere. Non-zero Wronskian at some x = x1 ensures 3. For every real root λ = γ of multiplicity r; e , xe , x e , non-zero Wronskian everywhere and the corrseponding , xr 1eγx are all linearly independent solutions. · · · − solutions as linearly independent. 4. For every complex pair of roots λ = µ iω of multiplicity r; I  With n linearly independent solutions y1(x), y2(x), , yn(x) eµx cos ωx, eµx sin ωx, xeµx cos ωx, xeµx sin ωx, , n· · · r 1 µx r 1 µx · · · of the HE, we have its general solution yh(x) = i=1 ci yi (x), x − e cos ωx, x − e sin ωx are the required solutions. acting as the complementary function for the NHE. P Applied Mathematical Methods Higher Order Linear ODE’s 397, Applied Mathematical Methods Higher Order Linear ODE’s 398, Theory of Linear ODE’s Theory of Linear ODE’s Non-Homogeneous Equations Homogeneous Equations with Constant Coefficients Non-Homogeneous Equations Homogeneous Equations with Constant Coefficients Non-Homogeneous Equations Non-Homogeneous Equations Euler-Cauchy Equation of Higher Order Euler-Cauchy Equation of Higher Order Method of undetermined coefficients Since each yi (x) is a solution of the HE, (n) (n 1) (n 2) n y + a1y − + a2y − + + an 1y 0 + any = R(x) · · · − (n 1) ui0(x)yi − (x) = R(x). Extension of the second order case i=1 Method of variation of parameters X Assembling all conditions on u (x) together, n 0

yp(x) = ui (x)yi (x) [Y(x)]u0(x) = enR(x). i X=1 1 adj Y Since Y− = det(Y) , Imposed condition Derivative 1 R(x) n n u0(x) = [adj Y(x)]enR(x) = [last column of adj Y(x)]. i=1 ui0(x)yi (x) = 0 yp0 (x) = i=1 ui (x)yi0(x) det[Y(x)] W (x) n ⇒ n u (x)y (x) = 0 y (x) = ui (x)y (x) Pi=1 i0 i0 ⇒ p00 P i=1 i00 Using cofactors of elements from last row only,

P·n· · · · · (n· · ·2) ⇒ (n· · ·1) ·P· · · ·n· (n 1) Wi (x) i=1 ui0(x)yi − (x) = 0 yp − (x) = i=1 ui (x)yi − (x) u0(x) = R(x), ⇒ i W (x) (n) n (n 1) n (n) FinallyP , yp (x) = u0(x)y − (x) + P ui (x)y (x) i=1 i i i=1 i with Wi (x) = Wronskian evaluated with en in place of i-th column. n n (n 1)P (n) P (n 1) u (x) = Wi (x)R(x) dx u0(x)y − (x)+ ui (x) y + P y − + + Pnyi = R(x). i W (x) ⇒ i i i 1 i · · · i i X=1 X=1 h i R

Applied Mathematical Methods Higher Order Linear ODE’s 399, Applied Mathematical Methods Laplace Transforms 400, Theory of Linear ODE’s Introduction Points to note Homogeneous Equations with Constant Coefficients Outline Basic Properties and Results Non-Homogeneous Equations Application to Differential Equations Euler-Cauchy Equation of Higher Order Handling Discontinuities Convolution Advanced Issues

I Wronskian for a higher order ODE Laplace Transforms I General theory of linear ODE’s Introduction I Variation for parameters for n-th order ODE Basic Properties and Results Application to Differential Equations Handling Discontinuities Convolution Necessary Exercises: 1,3,4 Advanced Issues Applied Mathematical Methods Laplace Transforms 401, Applied Mathematical Methods Laplace Transforms 402, Introduction Introduction Introduction Basic Properties and Results Introduction Basic Properties and Results Application to Differential Equations Application to Differential Equations Handling Discontinuities Handling Discontinuities Convolution Another question: What if R(x) is not continuous?Convolution Classical perspective Advanced Issues Advanced Issues I When power is switched on or off, what happens? I Entire differential equation is known in advance. I If there is a sudden voltage fluctuation, what happens to the I Go for a complete solution first. equipment connected to the power line? I Afterwards, use the initial (or other) conditions. Or, does “anything” happen in the immediate future? A practical situation “Something” certainly happens. The IVP has a solution! I You have a plant Laplace transforms provide a tool to find the solution, in I intrinsic dynamic model as well as the starting conditions. spite of the discontinuity of R(x). I You may drive the plant with different kinds of inputs on Integral transform: different occasions. b Implication T [f (t)](s) = K(s, t)f (t)dt Za I Left-hand side of the ODE and the initial conditions are known a priori. s: frequency variable I Right-hand side, R(x), changes from task to task. K(s, t): kernel of the transform Note: T [f (t)] is a function of s, not t.

Applied Mathematical Methods Laplace Transforms 403, Applied Mathematical Methods Laplace Transforms 404, Introduction Introduction Introduction Basic Properties and Results Basic Properties and Results Basic Properties and Results Application to Differential Equations Application to Differential Equations Handling Discontinuities Handling Discontinuities st Convolution Linearity: Convolution With kernel function K(s, t) = e , and limitsAdvanceda =Issues0, b = , Advanced Issues − ∞ Laplace transform L af (t) + bg(t) = aL f (t) + bL g(t) { } { } { } b First shifting property or the frequency shifting rule: ∞ st st F (s) = L f (t) = e− f (t)dt = lim e− f (t)dt at { } b L e f (t) = F (s a) Z0 →∞ Z0 { } − When this integral exists, f (t) has its Laplace transform. Laplace transforms of some elementary functions: st ∞ st e− ∞ 1 Sufficient condition: L(1) = e− dt = = , 0 s s I Z  0 f (t) is piecewise continuous, and − st ∞ st e− ∞ 1 ∞ st 1 I ct it is of exponential order, i.e. f (t) < Me for some (finite) L(t) = e− tdt = t + e− dt = 2 , | | 0 s 0 s 0 s M and c. Z  −  Z n! L(tn) = (for positive integer n), sn+1 Inverse Laplace transform: Γ(a + 1) L(ta) = (for a R+) 1 a+1 f (t) = L− F (s) s ∈ { } 1 and L(eat ) = . s a − Applied Mathematical Methods Laplace Transforms 405, Applied Mathematical Methods Laplace Transforms 406, Introduction Introduction Basic Properties and Results Basic Properties and Results Application to Differential Equations Basic Properties and Results Application to Differential Equations Application to Differential Equations Handling Discontinuities Handling Discontinuities Convolution Example: Convolution s Advanced Issues ω Initial value problem of a linear constant coefficientAdvanced IssuesODE L(cos ωt) = , L(sin ωt) = ; s2 + ω2 s2 + ω2 s a y 00 + ay 0 + by = r(t), y(0) = K0, y 0(0) = K1 L(cosh at) = , L(sinh at) = ; s2 a2 s2 a2 Laplace transforms of both sides of the ODE: − − µt s µ µt ω 2 L(e cos ωt) = − , L(e sin ωt) = . s Y (s) sy(0) y 0(0) + a[sY (s) y(0)] + bY (s) = R(s) (s µ)2 + ω2 (s µ)2 + ω2 − − − − − (s2 + as + b)Y (s) = (s + a)K + K + R(s) Laplace transform of derivative: ⇒ 0 1

∞ st A differential equation in y(t) has been converted to an L f 0(t) = e− f 0(t)dt { } algebraic equation in Y (s). Z0 st ∞ st = e− f (t) ∞ + s e− f (t)dt = sL f (t) f (0) Transfer function: ratio of Laplace transform of output function 0 { } − Z0 y(t) to that of input function r(t), with zero initial conditions   Using this process recursively, Y (s) 1 (n) n (n 1) (n 2) (n 1) Q(s) = = 2 (in this case) L f (t) = s L f (t) s − f (0) s − f 0(0) f − (0). R(s) s + as + b { } { }− − −· · · − t Y (s) = [(s + a)K0 + K1]Q(s) + Q(s)R(s) For integral g(t) = 0 f (t)dt, g(0) = 0, and 1 1 L g 0(t) = sL g(t) g(0) = sL g(t) L g(t) = L f (t) . Solution of the given IVP: y(t) = L Y (s) { } { }R− { } ⇒ { } s { } − { }

Applied Mathematical Methods Laplace Transforms 407, Applied Mathematical Methods Laplace Transforms 408, Introduction Introduction Handling Discontinuities Basic Properties and Results Handling Discontinuities Basic Properties and Results Application to Differential Equations Application to Differential Equations Handling Discontinuities Handling Discontinuities Unit step function Convolution Define Convolution Advanced Issues Advanced Issues 0 if t < a 1/k if a t a + k u(t a) = fk (t a) = ≤ ≤ − 1 if t > a − 0 otherwise   1 1 Its Laplace transform: = u(t a) u(t a k) a as k − − k − − ∞ st ∞ st e− L u(t a) = e− u(t a)dt = 0 dt + e− dt = 1 u(t−a) u(t−a) f (t−a) δ (t−a) { − } 0 − 0 · a s 1 k 1 k Z Z Z k k For input f (t) with a time delay, 1 1 1 1 o a t o a a+k t o a a+k t o a t 1 0 if t < a 1 − k 1 f (t a)u(t a) = − u(t−a−k) − − f (t a) if t > a k  − (a) Unit step function (b) Composition (c) Function f (d) Dirac’s δ − function has its Laplace transform as k

∞ st L f (t a)u(t a) = e− f (t a)dt Figure: Step and impulse functions { − − } − Za and note that its integral ∞ s(a+τ) as a+k = e− f (τ)dτ = e− L f (t) . ∞ 1 0 { } Ik = fk (t a)dt = dt = 1. Z 0 − a k Second shifting property or the time shifting rule Z Z does not depend on k. Applied Mathematical Methods Laplace Transforms 409, Applied Mathematical Methods Laplace Transforms 410, Introduction Introduction Handling Discontinuities Basic Properties and Results Convolution Basic Properties and Results Application to Differential Equations Application to Differential Equations Handling Discontinuities A generalized product of two functions Handling Discontinuities In the limit, Convolution Convolution Advanced Issues t Advanced Issues h(t) = f (t) g(t) = f (τ)g(t τ) dτ δ(t a) = lim fk (t a) ∗ 0 − − k 0 − Z → Laplace transform of the convolution: if t = a ∞ or, δ(t a) = ∞ and δ(t a)dt = 1. t − 0 otherwise − ∞ st ∞ ∞ st  0 H(s) = e− f (τ)g(t τ)dτ dt = f (τ) e− g(t τ)dt dτ Z − − Z0 Z0 Z0 Zτ Unit impulse function or Dirac’s delta function t = τ t = τ

τ τ ¢£

1 ¢£ ¡ ¡ ¡ ¡ ¡

L δ(t a) = lim [L u(t a) L u(t a k) ] ¢£ k 0

{ − } k { − } − { − − } ¢£ → e as e (a+k)s ¢£ − − as ¢£ = lim − = e− k 0 ks o t o t →

Through step and impulse functions, Laplace transform (a) Original order (b) Changed order method can handle IVP’s with discontinuous inputs. Figure: Region of integration for L h(t) { }

Applied Mathematical Methods Laplace Transforms 411, Applied Mathematical Methods Laplace Transforms 412, Introduction Introduction Convolution Basic Properties and Results Points to note Basic Properties and Results Application to Differential Equations Application to Differential Equations Handling Discontinuities Handling Discontinuities Convolution Convolution Through substitution t = t τ, Advanced Issues Advanced Issues 0 −

∞ ∞ s(t +τ) H(s) = f (τ) e− 0 g(t0) dt0 dτ I A paradigm shift in solution of IVP’s 0 0 Z Z I Handling discontinuous input functions ∞ sτ ∞ st0 = f (τ)e− e− g(t0) dt0 dτ I Extension to ODE systems Z0 Z0  I The idea of integral transforms H(s) = F (s)G(s) Convolution theorem: Laplace transform of the convolution integral of two functions is given by the product of the Laplace Necessary Exercises: 1,2,4 transforms of the two functions. Utilities: I To invert Q(s)R(s), one can convolute y(t) = q(t) r(t). ∗ I In solving some integral equation. Applied Mathematical Methods ODE Systems 413, Applied Mathematical Methods ODE Systems 414, Fundamental Ideas Fundamental Ideas Outline Linear Homogeneous Systems with Constant Coefficients Fundamental Ideas Linear Homogeneous Systems with Constant Coefficients Linear Non-Homogeneous Systems Linear Non-Homogeneous Systems Nonlinear Systems Nonlinear Systems

y0 = f(t, y) Solution: a vector function y = h(t)

ODE Systems Autonomous system: y0 = f(y) Fundamental Ideas I Points in y-space where f(y) = 0: Linear Homogeneous Systems with Constant Coefficients equilibrium points or critical points Linear Non-Homogeneous Systems Nonlinear Systems System of linear ODE’s:

y0 = A(t)y + g(t)

I autonomous systems if A and g are constant I homogeneous systems if g(t) = 0 I homogeneous constant coefficient systems if A is constant and g(t) = 0

Applied Mathematical Methods ODE Systems 415, Applied Mathematical Methods ODE Systems 416, Fundamental Ideas Fundamental Ideas Fundamental Ideas Linear Homogeneous Systems with Constant Coefficients Linear Homogeneous Systems with ConstantLinear HomogeneousCoSystemsefficientswith Constant Coefficients Linear Non-Homogeneous Systems Linear Non-Homogeneous Systems Nonlinear Systems Nonlinear Systems For a homogeneous system, y0 = Ay

y0 = A(t)y Non-degenerate case: matrix A non-singular I Origin (y = 0) is the unique equilibrium point. λt λt Attempt y = xe y0 = λxe . I Wronskian: W (y , y , y , , yn) = y y y yn ⇒ 1 2 3 · · · | 1 2 3 · · · | Substitution: Axeλt = λxeλt Ax = λx ⇒ If A is diagonalizable, If Wronskian is non-zero, then λ t I n linearly independent solutions yi = xi e i corresponding to n I Fundamental matrix: (t) = [y y y yn], Y 1 2 3 · · · eigenpairs giving a basis. If A is not diagonalizable? λ t General solution: All xi e i together will not complete the basis. n Try y = xteµt ? Substitution leads to y(t) = ci yi (t) = [ (t)] c Y i µt µt µt µt X=1 xe + µxte = Axte xe = 0 x = 0. ⇒ ⇒ Absurd! Applied Mathematical Methods ODE Systems 417, Applied Mathematical Methods ODE Systems 418, Fundamental Ideas Fundamental Ideas Linear Homogeneous Systems with ConstantLinear HomogeneousCoSystemsefficientswith Constant Coefficients Linear Non-Homogeneous Systems Linear Homogeneous Systems with Constant Coefficients Linear Non-Homogeneous Systems Linear Non-Homogeneous Systems Try a linearly independent solution in the foNonlinearm r Systems Nonlinear Systems y0 = Ay + g(t) y = xteµt + ueµt . Complementary function: Linear independence here has two implications: in n function space AND in ordinary vector space! yh(t) = ci yi (t) = [ (t)]c Y i=1 Substitution: X Complete solution: xeµt + µxteµt + µueµt = Axteµt + Aueµt (A µI)u = x ⇒ − y(t) = yh(t) + yp(t) Solve for u, the generalized eigenvector of A. We need to develop one particular solution y . For Jordan blocks of larger sizes, p

µt µt µt 1 2 µt µt µt Method of undetermined coefficients y1 = xe , y2 = xte +u1e , y3 = xt e +u1te +u2e etc. 2 Based on g(t), select candidate function Gk (t) and propose

Jordan canonical form (JCF) of A provides a set of basis yp = uk Gk (t), functions to describe the complete solution of the ODE Xk system. vector coefficients (uk ) to be determined by substitution.

Applied Mathematical Methods ODE Systems 419, Applied Mathematical Methods ODE Systems 420, Fundamental Ideas Fundamental Ideas Linear Non-Homogeneous Systems Linear Homogeneous Systems with Constant Coefficients Linear Non-Homogeneous Systems Linear Homogeneous Systems with Constant Coefficients Linear Non-Homogeneous Systems Linear Non-Homogeneous Systems Nonlinear Systems Method of variation of parameters Nonlinear Systems Method of diagonalization If we can supply a basis (t) of the complementary function yh(t), 1 Y If A is a diagonalizable constant matrix, with X− AX = D, then we propose 1 yp(t) = [ (t)]u(t) changing variables to z = X− y, such that y = Xz, Y Substitution leads to 1 1 Xz0 = AXz+g(t) z0 = X− AXz+X− g(t) = Dz+h(t) (say). ⇒ 0u + u0 = A u + g. Y Y Y Single decoupled Leibnitz equations Since = A , Y0 Y 1 z0 = dk zk + hk (t), k = 1, 2, 3, , n; u0 = g, or, u0 = [ ]− g. k · · · Y Y leading to individual solutions Complete solution:

d t d t d t 1 z (t) = c e k + e k e k h (t)dt. y(t) = yh + yp = [ ]c + [ ] [ ]− gdt k k − k Y Y Y Z Z After assembling z(t), we reconstruct y = Xz. This method is completely general. Applied Mathematical Methods ODE Systems 421, Applied Mathematical Methods Stability of Dynamic Systems 422, Fundamental Ideas Second Order Linear Systems Points to note Linear Homogeneous Systems with Constant Coefficients Outline Nonlinear Dynamic Systems Linear Non-Homogeneous Systems Lyapunov Stability Analysis Nonlinear Systems

I Theory of ODE’s in terms of vector functions I Methods to find Stability of Dynamic Systems I complementary functions in the case of constant coefficients Second Order Linear Systems I particular solutions for all cases Nonlinear Dynamic Systems Lyapunov Stability Analysis

Necessary Exercises: 1

Applied Mathematical Methods Stability of Dynamic Systems 423, Applied Mathematical Methods Stability of Dynamic Systems 424, Second Order Linear Systems Second Order Linear Systems Second Order Linear Systems Nonlinear Dynamic Systems Second Order Linear Systems Nonlinear Dynamic Systems Lyapunov Stability Analysis Lyapunov Stability Analysis A system of two first order linear differential equations: Characteristic equation:

y10 = a11y1 + a12y2 λ2 pλ + q = 0, − y20 = a21y1 + a22y2 with p = (a11 + a22) = λ1 + λ2 and q = a11a22 a12a21 = λ1λ2 or, y0 = Ay − Phase: a pair of values of y1 and y2 Discriminant D = p2 4q and − Phase plane: plane of y1 and y2 Trajectory: a curve showing the evolution of the system for a p p 2 p √D λ1,2 = q = . particular initial value problem 2  2 − 2  2 r  Phase portrait: all trajectories together showing the complete Solution (for diagonalizable A): picture of the behaviour of the dynamic system λ1t λ2t y = c1x1e + c2x2e

Allowing only isolated equilibrium points, Solution for deficient A: I matrix A is non-singular: origin is the only equilibrium point. λt λt Eigenvalues of A: y = c1x1e + c2(tx1 + u)e λt λt λt λ2 (a + a )λ + (a a a a ) = 0 y0 = c1λx1e + c2(x1 + λu)e + λtc2x1e − 11 22 11 22 − 12 21 ⇒ Applied Mathematical Methods Stability of Dynamic Systems 425, Applied Mathematical Methods Stability of Dynamic Systems 426, Second Order Linear Systems Second Order Linear Systems Second Order Linear Systems Nonlinear Dynamic Systems Second Order Linear Systems Nonlinear Dynamic Systems Lyapunov Stability Analysis Lyapunov Stability Analysis Table: Critical points of linear systems

y2 y2 y2 Type Sub-type Eigenvalues Position in p-q chart Stability Saddle pt real, opposite signs q < 0 unstable Centre pure imaginary q > 0, p = 0 stable o y1 o y1 o y1 Spiral complex, both q > 0, p = 0 stable non-zero components D = p2 6 4q < 0 if p < 0, Node real, same sign q > 0, p−= 0, D 0 unstable (a) Saddle point (b) Centre (c) Spiral 6 ≥ y2 improper unequal in magnitude D > 0 if p > 0 y2 y2 proper equal, diagonalizable D = 0 degenerate equal, deficient D = 0

q spiral spiral o y1 o y1 y1 stable c unstable o e n t r node e q = 0 node p2 − 4

(d) Improper node (e) Proper node (f) Degenerate node o p

saddle point Figure: Neighbourhood of critical points unstable Figure: Zones of critical points in p-q chart

Applied Mathematical Methods Stability of Dynamic Systems 427, Applied Mathematical Methods Stability of Dynamic Systems 428, Second Order Linear Systems Second Order Linear Systems Nonlinear Dynamic Systems Nonlinear Dynamic Systems Lyapunov Stability Analysis Nonlinear Dynamic Systems Lyapunov Stability Analysis Lyapunov Stability Analysis Important terms Phase plane analysis Stability: If y0 is a critical point of the dynamic system I Determine all the critical points. y0 = f(y) and for every  > 0, δ > 0 such that I Linearize the ODE system around each of them as ∃ y(t ) y < δ y(t) y <  t > t , k 0 − 0k ⇒ k − 0k ∀ 0 y0 = J(y0)(y y0). − then y0 is a stable critical point. If, further, I With z = y y , analyze each neighbourhood from z = Jz. y(t) y0 as t , then y0 is said to be − 0 0 → → ∞ I Assemble outcomes of local phase plane analyses. asymptotically stable. Positive definite function: A function V (y), with V (0) = 0, is ‘Features’ of a dynamic system are typically captured by called positive definite if its critical points and their neighbourhoods. V (y) > 0 y = 0. ∀ 6 Limit cycles Lyapunov function: A positive definite function V (y), having I isolated closed trajectories (only in nonlinear systems) continuous ∂V , with a negative semi-definite rate of ∂yi change Systems with arbitrary dimension of state space? T V 0 = [ V (y)] f(y). ∇ Applied Mathematical Methods Stability of Dynamic Systems 429, Applied Mathematical Methods Stability of Dynamic Systems 430, Second Order Linear Systems Second Order Linear Systems Lyapunov Stability Analysis Nonlinear Dynamic Systems Points to note Nonlinear Dynamic Systems Lyapunov Stability Analysis Lyapunov Stability Analysis

Lyapunov’s stability criteria: I Analysis of second order systems Theorem: For a system y0 = f(y) with the origin as a critical point, if there exists a Lyapunov function V (y), I Classification of critical points then the system is stable at the origin, i.e. the origin is a I Nonlinear systems and local linearization stable critical point. I Phase plane analysis Further, if V 0(y) is negative definite, then it is Examples in physics, engineering, economics, asymptotically stable. biological and social systems A generalization of the notion of total energy: negativity of its rate I Lyapunov’s method of stability analysis correspond to trajectories tending to decrease this ‘energy’.

Note: Lyapunov’s method becomes particularly important when a Necessary Exercises: 1,2,3,4,5 linearized model allows no analysis or when its results are suspect.

Caution: It is a one-way criterion only!

Applied Mathematical Methods Series Solutions and Special Functions 431, Applied Mathematical Methods Series Solutions and Special Functions 432, Power Series Method Power Series Method Outline Frobenius’ Method Power Series Method Frobenius’ Method Special Functions Defined as Integrals Special Functions Defined as Integrals Special Functions Arising as Solutions of ODE’s Methods to solve an ODE in terms of elementaSpecialryFunctionsfunctions:Arising as Solutions of ODE’s I restricted in scope Theory allows study of the properties of solutions!

Series Solutions and Special Functions When elementary methods fail, I Power Series Method gain knowledge about solutions through properties, and I Frobenius’ Method for actual evaluation develop infinite series. Special Functions Defined as Integrals Power series: Special Functions Arising as Solutions of ODE’s ∞ n 2 3 4 5 y(x) = anx = a + a x + a x + a x + a x + a x + 0 1 2 3 4 5 · · · n X=0 or in powers of (x x ). − 0 A simple exercise: Try developing power series solutions in the above form and study their properties for differential equations

2 y 00 + y = 0 and 4x y 00 = y. Applied Mathematical Methods Series Solutions and Special Functions 433, Applied Mathematical Methods Series Solutions and Special Functions 434, Power Series Method Power Series Method Power Series Method Frobenius’ Method Power Series Method Frobenius’ Method Special Functions Defined as Integrals Special Functions Defined as Integrals Special Functions Arising as Solutions of ODE’s n Special Functions Arising as Solutions of ODE’s Differentiation of y(x) = n∞=0 anx as y 00 + P(x)y 0 + Q(x)y = 0 ∞ P ∞ y (x) = (n + 1)a xn and y (x) = (n + 2)(n + 1)a xn If P(x) and Q(x) are analytic at a point x = x0, 0 n+1 00 n+2 n=0 n=0 i.e. if they possess convergent series expansions in powers X X leads to of (x x ) with some radius of convergence R, − 0 n ∞ n ∞ n ∞ n P(x)y 0 = pnx (n + 1)an+1x = pn k (k + 1)ak+1x then the solution is analytic at x0, and a power series solution − n=0 "n=0 # n=0 k=0 2 3 y(x) = a0 + a1(x x0) + a2(x x0) + a3(x x0) + X X n X X − − − · · · ∞ n ∞ n ∞ n Q(x)y = qnx anx = qn k ak x is convergent at least for x x0 < R. − | − | n "n # n k X=0 X=0 X=0 X=0 For x0 = 0 (without loss of generality), suppose n n ∞ ∞ n n 2 3 (n + 2)(n + 1)an+2 + pn k (k + 1)ak+1 + qn k ak x = 0 P(x) = pnx = p0 + p1x + p2x + p3x + , ⇒ − − · · · n=0 " k=0 k=0 # n=0 X X X X Recursion formula: ∞ n 2 3 n Q(x) = qnx = q + q x + q x + q x + , 0 1 2 3 · · · 1 n=0 an+2 = [(k + 1)pn k ak+1 + qn k ak ] X −(n + 2)(n + 1) − − n k=0 and assume y(x) = n∞=0 anx . X P

Applied Mathematical Methods Series Solutions and Special Functions 435, Applied Mathematical Methods Series Solutions and Special Functions 436, Power Series Method Power Series Method Frobenius’ Method Frobenius’ Method Frobenius’ Method Frobenius’ Method Special Functions Defined as Integrals Special Functions Defined as Integrals Special Functions Arising as Solutions of ODE’s Special Functions Arising as Solutions of ODE’s For the ODE y 00 + P(x)y 0 + Q(x)y = 0, a point x = x0 is Working steps: ordinary point if P(x) and Q(x) are analytic at x = x0: power r n 1. Assume the solution in the form y(x) = x n∞=0 anx . series solution is analytic 2. Differentiate to get the series expansions forPy 0(x) and y 00(x). singular point if any of the two is non-analytic (singular) at x = x0 3. Substitute these series for y(x), y 0(x) and y 00(x) into the I regular singularity: (x x0)P(x) and r r+1 r+2 2 − given ODE and collect coefficients of x , x , x etc. (x x0) Q(x) are analytic at the point − 4. Equate the coefficient of x r to zero to obtain an equation in I irregular singularity the index r, called the indicial equation as

The case of regular singularity r(r 1) + b r + c = 0; − 0 0 b(x) c(x) allowing a to become arbitrary. For x0 = 0, with P(x) = x and Q(x) = x2 , 0 5. For each solution r, equate other coefficients to obtain a1, a2, 2 x y 00 + xb(x)y 0 + c(x)y = 0 a3 etc in terms of a0. in which b(x) and c(x) are analytic at the origin. Note: The need is to develop two solutions. Applied Mathematical Methods Series Solutions and Special Functions 437, Applied Mathematical Methods Series Solutions and Special Functions 438, Power Series Method Power Series Method Special Functions Defined as IntegralsFrobenius’ Method Special Functions Arising as SolutionsFrobofenius’ODE’sMethod Special Functions Defined as Integrals Special Functions Defined as Integrals Special Functions Arising as Solutions of ODE’s Special Functions Arising as Solutions of ODE’s In the study of some important problems in physics, x n 1 some variable-coefficient ODE’s appear recurrently, Gamma function: Γ(n) = 0∞ e− x − dx, convergent for n > 0. Recurrence relation Γ(1) = 1, Γ(n + 1) = nΓ(n) R defying analytical solution! allows extension of the definition for the entire real Series solutions properties and connections line except for zero and negative integers. ⇒ further problems further solutions Γ(n + 1) = n! for non-negative integers. ⇒ ⇒ ⇒ · · · (A generalization of the factorial function.) 1 m 1 n 1 Table: Special functions of mathematical physics Beta function: B(m, n) = 0 x − (1 x) − dx = π/2 − Name of the ODE Form of the ODE Resulting functions 2m 1 2n 1 2 2 sin − θ cos − θ dθ; m, n > 0. Legendre’s equation (1 x )y00 2xy0 + k(k + 1)y = 0 Legendre functions 0 R − − m n Legendre polynomials Γ( )Γ( ) 2 B(m, n) = B(n, m); B(m, n) = Airy’s equation y 00 k xy = 0 Airy functions R Γ(m+n)  2 2 Chebyshev’s equation (1 x )y00 xy0 + k y = 0 Chebyshev polynomials 2 − − 2 x t Hermite’s equation y 00 2xy0 + 2ky = 0 Hermite functions − Error function: erf (x) = √π 0 e− dt. Hermite polynomials Bessel’s equation x 2y + xy + (x2 k2)y = 0 Bessel functions (Area under the normal or Gaussian distribution) 00 0 − R Neumann functions x sin t Hankel functions Sine integral function: Si (x) = 0 t dt. Gauss’s hypergeometric x(1 x)y 00 + [c (a + b + 1)x]y0 aby = 0 Hypergeometric function equation − − − Laguerre’s equation xy + (1 x)y + ky = 0 Laguerre polynomials R 00 − 0

Applied Mathematical Methods Series Solutions and Special Functions 439, Applied Mathematical Methods Series Solutions and Special Functions 440, Power Series Method Power Series Method Special Functions Arising as SolutionsFrobofenius’ODE’sMethod Special Functions Arising as SolutionsFrobofenius’ODE’sMethod Special Functions Defined as Integrals Special Functions Defined as Integrals Special Functions Arising as Solutions of ODE’s Special Functions Arising as Solutions of ODE’s Legendre’s equation Legendre functions 2 (1 x )y 00 2xy 0 + k(k + 1)y = 0 − − k(k + 1) 2 k(k 2)(k + 1)(k + 3) 4 P(x) = 2x and Q(x) = k(k+1) are analytic at x = 0 with y1(x) = 1 x + − x − 1 x2 1 x2 − 2! 4! − · · · radius of convergence− R = 1. − (k 1)(k + 2) (k 1)(k 3)(k + 2)(k + 4) y (x) = x − x3 + − − x5 x = 0 is an ordinary point and a power series solution 2 − 3! 5! − · · · n y(x) = n∞=0 anx is convergent at least for x < 1. | | Special significance: non-negative integral values of k Apply powerPseries method: For each k = 0, 1, 2, 3, , · · · k(k + 1) one of the series terminates at the term containing x k . a = a , 2 − 2! 0 (k + 2)(k 1) Polynomial solution: valid for the entire real line! a = − a 3 − 3! 1 Recurrence relation in reverse: (k n)(k + n + 1) and an+2 = − an for n 2. k(k 1) − (n + 2)(n + 1) ≥ ak 2 = − ak − −2(2k 1) − Solution: y(x) = a0y1(x) + a1y2(x) Applied Mathematical Methods Series Solutions and Special Functions 441, Applied Mathematical Methods Series Solutions and Special Functions 442, Power Series Method Power Series Method Special Functions Arising as SolutionsFrobofenius’ODE’sMethod Special Functions Arising as SolutionsFrobofenius’ODE’sMethod Special Functions Defined as Integrals Special Functions Defined as Integrals Special Functions Arising as Solutions of ODE’s Special Functions Arising as Solutions of ODE’s Legendre polynomial 1 P (x) (2k 1)(2k 3) 3 1 0 Choosing ak = − k!− ··· · , 0.8 P (x) 1 (2k 1)(2k 3) 3 1 0.6 P 3 (x) Pk (x) = − − · · · · 0.4 P (x) k! 2 0.2 k k(k 1) k 2 k(k 1)(k 2)(k 3) k 4 x − x − + − − − x − . 0 n × − 2(2k 1) 2 4(2k 1)(2k 3) − · · · P (x)  − · − −  −0.2 k This choice of ak ensures Pk (1) = 1 and implies Pk ( 1) = ( 1) . − − −0.4 Initial Legendre polynomials: −0.6 P 5 (x) P 4 (x) P0(x) = 1, −0.8

−1 P1(x) = x, −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 x 1 P (x) = (3x2 1), 2 2 − Figure: Legendre polynomials 1 3 P3(x) = (5x 3x), 2 − All roots of a Legendre polynomial are real and they lie in [ 1, 1]. 1 4 2 − P4(x) = (35x 30x + 3) etc. 8 − Orthogonality?

Applied Mathematical Methods Series Solutions and Special Functions 443, Applied Mathematical Methods Series Solutions and Special Functions 444, Power Series Method Power Series Method Special Functions Arising as SolutionsFrobofenius’ODE’sMethod Special Functions Arising as SolutionsFrobofenius’ODE’sMethod Special Functions Defined as Integrals Special Functions Defined as Integrals Bessel’s equation Special Functions Arising as Solutions of ODE’s Special Functions Arising as Solutions of ODE’s Bessel functions: 2 2 2 1 x y 00 + xy 0 + (x k )y = 0 Selecting a = and using n = 2m, − 0 2k Γ(k+1) x = 0 is a regular singular point. ( 1)m Frobenius’ method: carrying out the early steps, a = − . m 2k+2mm!Γ(k + m + 1)

2 2 r 2 2 r+1 ∞ 2 2 r+n (r k )a0x +[(r+1) k ]a1x + [an 2+ r k +n(n+2r) an]x = 0 Bessel function of the first kind of order k: − − − { − } n=2 X m x k+2m ∞ xk+2m ∞ ( 1) 2 2 m 2 Indicial equation: r k = 0 r = k Jk (x) = ( 1) k m = − − ⇒  − 2 +2 m!Γ(k + m + 1) m!Γ(k + m + 1) With r = k, (r + 1)2 k2 = 0 a = 0 and m=0 m=0
− 6 ⇒ 1 X X an 2 an = − for n 2. When k is not an integer, J k (x) completes the basis. −n(n + 2r) ≥ − k Odd coefficients are zero and For integer k, J k (x) = ( 1) Jk (x), linearly dependent! − − Reduction of order can be used to find another solution. a0 a0 a2 = , a4 = , etc. Bessel function of the second kind or Neumann function −2(2k + 2) 2 4(2k + 2)(2k + 4) · Applied Mathematical Methods Series Solutions and Special Functions 445, Applied Mathematical Methods Sturm-Liouville Theory 446, Power Series Method Preliminary Ideas Points to note Frobenius’ Method Outline Sturm-Liouville Problems Special Functions Defined as Integrals Eigenfunction Expansions Special Functions Arising as Solutions of ODE’s

I Solution in power series I Ordinary points and singularities Sturm-Liouville Theory I Definition of special functions Preliminary Ideas I Legendre polynomials Sturm-Liouville Problems I Bessel functions Eigenfunction Expansions

Necessary Exercises: 2,3,4,5

Applied Mathematical Methods Sturm-Liouville Theory 447, Applied Mathematical Methods Sturm-Liouville Theory 448, Preliminary Ideas Preliminary Ideas Preliminary Ideas Sturm-Liouville Problems Preliminary Ideas Sturm-Liouville Problems Eigenfunction Expansions Eigenfunction Expansions A simple boundary value problem: Boundary value problems as eigenvalue problems Explore the possible solutions of the BVP

y 00 + 2y = 0, y(0) = 0, y(π) = 0 y 00 + ky = 0, y(0) = 0, y(π) = 0. General solution of the ODE: I With k 0, no hope for a non-trivial solution. Consider y(x) = a sin(x√2) + b cos(x√2) ≤ k = ν2 > 0. I Condition y(0) = 0 b = 0. Hence, y(x) = a sin(x√2). Solutions: y = a sin(νx), only for specific values of ν (or k): ⇒ ν = 0, 1, 2, 3, ; i.e. k = 0, 1, 4, 9, . Then, y(π) = 0 a = 0. Only solution is y(x) = 0.    · · · · · · ⇒ Question: Now, consider the BVP I For what values of k (eigenvalues), does the given BVP possess non-trivial solutions, and y 00 + 4y = 0, y(0) = 0, y(π) = 0. I what are the corresponding solutions (eigenfunctions), up to The same steps give y(x) = a sin(2x), with arbitrary value of a. arbitrary scalar multiples? Infinite number of non-trivial solutions! Analogous to the algebraic eigenvalue problem Av = λv! Analogy of a Hermitian matrix: self-adjoint differential operator. Applied Mathematical Methods Sturm-Liouville Theory 449, Applied Mathematical Methods Sturm-Liouville Theory 450, Preliminary Ideas Preliminary Ideas Preliminary Ideas Sturm-Liouville Problems Preliminary Ideas Sturm-Liouville Problems Eigenfunction Expansions Eigenfunction Expansions Consider the ODE y 00 + P(x)y 0 + Q(x)y = 0. The adjoint ODE Question: I The adjoint of the ODE y 00 + P(x)y 0 + Q(x)y = 0 is Is it possible to find functions F (x) and G(x) such that F 00 + P1F 0 + Q1F = 0, F (x)y 00 + F (x)P(x)y 0 + F (x)Q(x)y where P = P and Q = Q P . 1 − 1 − 0 I gets reduced to the derivative of F (x)y 0 + G(x)y? Then, the adjoint of F 00 + P1F 0 + Q1F = 0 is

Comparing with φ00 + P2φ0 + Q2φ = 0,

d where P2 = P1 = P and − [F (x)y 0 + G(x)y] = F (x)y 00 + [F 0(x) + G(x)]y 0 + G 0(x)y, Q = Q P = Q P ( P ) = Q. dx 2 1 − 10 − 0 − − 0 The adjoint of the adjoint of a second order linear F (x) + G(x) = F (x)P(x) and G (x) = F (x)Q(x). 0 0 homogeneous equation is the original equation itself. Elimination of G(x): I When is an ODE its own adjoint? I y + P(x)y + Q(x)y = 0 is self-adjoint only in the trivial case F 00(x) P(x)F 0(x) + [Q(x) P 0(x)]F (x) = 0 00 0 − − of P(x) = 0. I This is the adjoint of the original ODE. What about F (x)y 00 + F (x)P(x)y 0 + F (x)Q(x)y = 0?

Applied Mathematical Methods Sturm-Liouville Theory 451, Applied Mathematical Methods Sturm-Liouville Theory 452, Preliminary Ideas Preliminary Ideas Preliminary Ideas Sturm-Liouville Problems Preliminary Ideas Sturm-Liouville Problems Eigenfunction Expansions Eigenfunction Expansions Second order self-adjoint ODE Casting a given ODE into the self-adjoint form:

Question: What is the adjoint of Fy 00 + FPy 0 + FQy = 0? Equation y 00 + P(x)y 0 + Q(x)y = 0 is converted to the Rephrased question: What is the ODE that φ(x) has to satisfy if self-adjoint form through the multiplication of F (x) = eR P(x)dx . d φFy 00 + φFPy 0 + φFQy = φFy 0 + ξ(x)y ? dx General form of self-adjoint equations:   Comparing terms, d [F (x)y 0] + R(x)y = 0 dx d (φF ) + ξ(x) = φFP and ξ0(x) = φFQ. dx Working rules: I 2 To determine whether a given ODE is in the self-adjoint form, Eliminating ξ(x), we have d (φF ) + φFQ = d (φFP). dx2 dx check whether the coefficient of y 0 is the derivative of the coefficient of y 00. F φ00 + 2F 0φ0 + F 00φ + FQφ = FPφ0 + (FP)0φ I To convert an ODE into the self-adjoint form, first obtain the F φ00 + (2F 0 FP)φ0 + F 00 (FP)0 + FQ φ = 0 equation in normal form by dividing with the coefficient of y 00. ⇒ − − If the coefficient of y 0 now is P(x), then next multiply the   This is the same as the original ODE, when F 0(x) = F (x)P(x) resulting equation with eR Pdx . Applied Mathematical Methods Sturm-Liouville Theory 453, Applied Mathematical Methods Sturm-Liouville Theory 454, Preliminary Ideas Preliminary Ideas Sturm-Liouville Problems Sturm-Liouville Problems Sturm-Liouville Problems Sturm-Liouville Problems Eigenfunction Expansions Eigenfunction Expansions Orthogonality of eigenfunctions Sturm-Liouville equation Theorem: If ym(x) and yn(x) are eigenfunctions [r(x)y 0]0 + [q(x) + λp(x)]y = 0, (solutions) of a Sturm-Liouville problem corresponding to distinct eigenvalues λm and λn respectively, then where p, q, r and r 0 are continuous on [a, b], with p(x) > 0 on [a, b] and r(x) > 0 on (a, b). b (ym, yn) p(x)ym(x)yn(x)dx = 0, ≡ With different boundary conditions, Za Regular S-L problem: i.e. they are orthogonal with respect to the weight a1y(a) + a2y 0(a) = 0 and b1y(b) + b2y 0(b) = 0, function p(x). T T vectors [a1 a2] and [b1 b2] being non-zero. From the hypothesis, Periodic S-L problem: With r(a) = r(b), y(a) = y(b) and y (a) = y (b). (rym0 )0 + (q + λmp)ym = 0 (q + λmp)ymyn = (rym0 )0yn 0 0 ⇒ − Singular S-L problem: If r(a) = 0, no boundary condition is (ryn0 )0 + (q + λnp)yn = 0 (q + λnp)ymyn = (ryn0 )0ym ⇒ − needed at x = a. If r(b) = 0, no boundary condition Subtracting, is needed at x = b. (λm λn)pymyn = (ry 0 )0ym + (ry 0 )y 0 (ry 0 )y 0 (ry 0 )0yn (We just look for bounded solutions over [a, b].) − n n m − m n − m = r(ymy 0 yny 0 ) 0 . n − m  

Applied Mathematical Methods Sturm-Liouville Theory 455, Applied Mathematical Methods Sturm-Liouville Theory 456, Preliminary Ideas Preliminary Ideas Sturm-Liouville Problems Sturm-Liouville Problems Sturm-Liouville Problems Sturm-Liouville Problems Eigenfunction Expansions Eigenfunction Expansions Integrating both sides, Example: Legendre polynomials over [ 1, 1] − b Legendre’s equation (λm λn) p(x)ym(x)yn(x)dx d 2 − a [(1 x )y 0] + k(k + 1)y = 0 Z dx − = r(b)[ym(b)yn0 (b) yn(b)ym0 (b)] r(a)[ym(a)yn0 (a) yn(a)ym0 (a)]. − − − is self-adjoint and defines a singular Sturm Liouville problem over I In a regular S-L problem, from the boundary condition at [ 1, 1] with p(x) = 1, q(x) = 0, r(x) = 1 x 2 and λ = k(k + 1). x = a, the homogeneous system − − ym(a) ym0 (a) a1 0 1 = has non-trivial solutions. 2 1 yn(a) yn0 (a) a2 0 (m n)(m+n+1) Pm(x)Pn(x)dx = [(1 x )(PmPn0 PnPm0 )] 1 = 0       − 1 − − − Therefore, ym(a)y 0 (a) yn(a)y 0 (a) = 0. Z n − m − Similarly, ym(b)y (b) yn(b)y (b) = 0. From orthogonal decompositions 1 = P0(x), x = P1(x), n0 − m0 I In a singular S-L problem, zero value of r(x) at a boundary 1 1 2 1 x2 = (3x2 1) + = P (x) + P (x), makes the corresponding term vanish even without a BC. 3 − 3 3 2 3 0 I In a periodic S-L problem, the two terms cancel out together. 3 1 3 3 2 3 x = (5x 3x) + x = P3(x) + P1(x), Since λm = λn, in all cases, 5 − 5 5 5 6 b 4 8 4 1 x = P4(x) + P2(x) + P0(x) etc; p(x)ym(x)yn(x)dx = 0. 35 7 5 a Z Pk (x) is orthogonal to all polynomials of degree less than k. Applied Mathematical Methods Sturm-Liouville Theory 457, Applied Mathematical Methods Sturm-Liouville Theory 458, Preliminary Ideas Preliminary Ideas Sturm-Liouville Problems Sturm-Liouville Problems Eigenfunction Expansions Sturm-Liouville Problems Eigenfunction Expansions Eigenfunction Expansions Real eigenvalues Eigenfunctions of Sturm-Liouville problems: Eigenvalues of a Sturm-Liouville problem are real. convenient and powerful instruments to represent and Let eigenvalue λ = µ + iν and eigenfunction y(x) = u(x) + iv(x). manipulate fairly general classes of functions Substitution leads to

[r(u + iv )] + [q + (µ + iν)p](u + iv) = 0. y0, y1, y2, y3, : a family of continuous functions over [a, b], 0 0 0 { · · · } mutually orthogonal with respect to p(x). Separation of real and imaginary parts: 2 Representation of a function f (x) on [a, b]: [ru0]0 + (q + µp)u νpv = 0 νpv = [ru0]0v + (q + µp)uv − ⇒ 2 [rv 0]0 + (q + µp)v + νpu = 0 νpu = [rv 0]0u (q + µp)uv ∞ ⇒ − − f (x) = amym(x) = a y (x) + a y (x) + a y (x) + a y (x) + Adding together, 0 0 1 1 2 2 3 3 · · · m=0 2 2 X νp(u + v ) = [ru0]0v + [ru0]v 0 [rv 0]u0 [rv 0]0u = r(uv 0 vu0) 0 − − − − Generalized Fourier series Integration and application of boundary conditions leads to   Analogous to the representation of a vector as a linear combination b of a set of mutually orthogonal vectors. ν p(x)[u2(x) + v 2(x)]dx = 0. a Z Question: How to determine the coefficients (an)? ν = 0 and λ = µ

Applied Mathematical Methods Sturm-Liouville Theory 459, Applied Mathematical Methods Sturm-Liouville Theory 460, Preliminary Ideas Preliminary Ideas Eigenfunction Expansions Sturm-Liouville Problems Eigenfunction Expansions Sturm-Liouville Problems Eigenfunction Expansions Eigenfunction Expansions Inner product: In terms of a finite number of members of the family φk (x) , { } b N (f , yn) = p(x)f (x)yn(x)dx ΦN (x) = αmφm(x) = α0φ0(x)+α1φ1(x)+α2φ2(x)+ +αN φN (x). Za · · · b m=0 ∞ ∞ 2 X = [amp(x)ym(x)yn(x)]dx = am(ym, yn) = an yn Error a k k Z m=0 m=0 2 X X b N where E = f Φ 2 = p(x) f (x) α φ (x) dx b N m m k − k a − 2 Z " m=0 # yn = (yn, yn) = p(x)yn (x)dx X k k s a Z Error is minimized when p (f ,yn) Fourier coefficients: a = b N n yn 2 k k ∂E Normalized eigenfunctions: = 2p(x) f (x) αmφm(x) [ φn(x)]dx = 0 ∂αn − − Za " m=0 # ym(x) X φm(x) = b b ym(x) 2 k k αnp(x)φ (x)dx = p(x)f (x)φn(x)dx. ⇒ n Generalized Fourier series (in orthonormal basis): Za Za α = c ∞ n n f (x) = cmφm(x) = c φ (x)+c φ (x)+c φ (x)+c φ (x)+ best approximation in the mean or least square approximation 0 0 1 1 2 2 3 3 · · · m X=0 Applied Mathematical Methods Sturm-Liouville Theory 461, Applied Mathematical Methods Sturm-Liouville Theory 462, Preliminary Ideas Preliminary Ideas Eigenfunction Expansions Sturm-Liouville Problems Eigenfunction Expansions Sturm-Liouville Problems Eigenfunction Expansions Eigenfunction Expansions Using the Fourier coefficients, error Question: Does it converge to f ?

N N N N b 2 2 2 2 2 E = (f , f ) 2 cn(f , φn)+ cn (φn, φn) = f 2 cn + cn lim p(x)[sk (x) f (x)] dx = 0? − k k − k − n n n n →∞ a X=0 X=0 X=0 X=0 Z N Answer: Depends on the basis used. E = f 2 c2 0. Convergence in the mean or mean-square convergence: k k − n ≥ n=0 An orthonormal set of functions φk (x) on an interval X { } Bessel’s inequality: a x b is said to be complete in a class of functions, ≤ ≤ N b or to form a basis for it, if the corresponding generalized c2 f 2 = p(x)f 2(x)dx Fourier series for a function converges in the mean to the n ≤ k k n a function, for every function belonging to that class. X=0 Z 2 2 Partial sum Parseval’s identity: ∞ c = f k n=0 n k k Eigenfunction expansion: generalized Fourier series in terms of sk (x) = amφm(x) P m=0 eigenfunctions of a Sturm-Liouville problem X I Question: Does the sequence of sk converge? convergent for continuous functions with piecewise continuous { } Answer: The bound in Bessel’s inequality ensures convergence. derivatives, i.e. they form a basis for this class.

Applied Mathematical Methods Sturm-Liouville Theory 463, Applied Mathematical Methods Fourier Series and Integrals 464, Preliminary Ideas Basic Theory of Fourier Series Points to note Sturm-Liouville Problems Outline Extensions in Application Eigenfunction Expansions Fourier Integrals

I Eigenvalue problems in ODE’s I Self-adjoint differential operators Fourier Series and Integrals I Sturm-Liouville problems Basic Theory of Fourier Series I Orthogonal eigenfunctions Extensions in Application I Eigenfunction expansions Fourier Integrals

Necessary Exercises: 1,2,4,5 Applied Mathematical Methods Fourier Series and Integrals 465, Applied Mathematical Methods Fourier Series and Integrals 466, Basic Theory of Fourier Series Basic Theory of Fourier Series Basic Theory of Fourier Series Extensions in Application Basic Theory of Fourier Series Extensions in Application Fourier Integrals Fourier Integrals With q(x) = 0 and p(x) = r(x) = 1, periodic S-L problem: Dirichlet’s conditions:

y 00 + λy = 0, y( L) = y(L), y 0( L) = y 0(L) If f (x) and its derivative are piecewise continuous on − − [ L, L] and are periodic with a period 2L, then the series πx πx 2πx 2πx − f (x+)+f (x ) Eigenfunctions 1, cos L , sin L , cos L , sin L , converges to the mean − of one-sided limits, at · · · 2 constitute an orthogonal basis for representing functions. all points. For a periodic function f (x) of period 2L, we propose Fourier series ∞ nπx nπx Note: The interval of integration can be [x , x + 2L] for any x . f (x) = a + a cos + b sin 0 0 0 0 n L n L n X=1   I It is valid to integrate the Fourier series term by term. and determine the Fourier coefficients from Euler formulae I The Fourier series uniformly converges to f (x) over an L interval on which f (x) is continuous. At a jump discontinuity, 1 f (x+)+f (x ) a0 = f (x)dx, convergence to − is not uniform. Mismatch peak 2L L 2 Z− shifts with inclusion of more terms (Gibb’s phenomenon). 1 L mπx 1 L mπx am = f (x) cos dx and bm = f (x) sin dx. I Term-by-term differentiation of the Fourier series at a point L L L L L L Z− Z− requires f (x) to be smooth at that point. Question: Does the series converge?

Applied Mathematical Methods Fourier Series and Integrals 467, Applied Mathematical Methods Fourier Series and Integrals 468, Basic Theory of Fourier Series Basic Theory of Fourier Series Basic Theory of Fourier Series Extensions in Application Extensions in Application Extensions in Application Fourier Integrals Fourier Integrals Multiplying the Fourier series with f (x), Original spirit of Fouries series I representation of periodic functions over ( , ). 2 ∞ nπx nπx −∞ ∞ f (x) = a0f (x) + anf (x) cos + bnf (x) sin Question: What about a function f (x) defined only on [ L, L]? L L n − X=1 h i Answer: Extend the function as Parseval’s identity: F (x) = f (x) for L x L, and F (x + 2L) = F (x). L − ≤ ≤ 1 ∞ 1 a2 + (a2 + b2) = f 2(x)dx Fourier series of F (x) acts as the Fourier series representation of ⇒ 0 2 n n 2L n=1 L f (x) in its own domain. X Z− The Fourier series representation is complete. In Euler formulae, notice that bm = 0 for an even function. I A periodic function f (x) is composed of its mean value and The Fourier series of an even function is a Fourier several sinusoidal components, or harmonics. cosine series I Fourier coefficients are corresponding amplitudes. ∞ nπx I Parseval’s identity is simply a statement on energy balance! f (x) = a + a cos , 0 n L n Bessel’s inequality X=1 N 1 1 where a = 1 L f (x)dx and a = 2 L f (x) cos nπx dx. a2 + (a2 + b2) f (x) 2 0 L 0 n L 0 L 0 2 n n ≤ 2Lk k n X=1 Similarly, for an oddRfunction, Fourier sine Rseries. Applied Mathematical Methods Fourier Series and Integrals 469, Applied Mathematical Methods Fourier Series and Integrals 470, Basic Theory of Fourier Series Basic Theory of Fourier Series Extensions in Application Extensions in Application Extensions in Application Extensions in Application Fourier Integrals Fourier Integrals Over [0, L], sometimes we need a series of sine terms only, or Half-range expansions cosine terms only! I For Fourier cosine series of a function f (x) over [0, L], even

f(x) fc (x) periodic extension: f (x) for 0 x L, fc (x) = ≤ ≤ and fc (x+2L) = fc (x) f ( x) for L x < 0, O L x −3L −2L −L O L 2L 3L x  − − ≤ I For Fourier sine series of a function f (x) over [0, L], odd

(a) Function over ( 0,L ) (b) Even periodic extension periodic extension: f (x) for 0 x L, fs (x) = ≤ ≤ and fs (x+2L) = fs (x) fs (x) f ( x) for L x < 0,  − − − ≤ To develop the Fourier series of a function, which is available as a set of tabulated values or a black-box library routine, −3L −2L −L O L 2L 3L x integrals in the Euler formulae are evaluated numerically. Important: Fourier series representation is richer and more (c) Odd periodic extension powerful compared to interpolatory or least square approximation Figure: Periodic extensions for cosine and sine series in many contexts.

Applied Mathematical Methods Fourier Series and Integrals 471, Applied Mathematical Methods Fourier Series and Integrals 472, Basic Theory of Fourier Series Basic Theory of Fourier Series Fourier Integrals Extensions in Application Fourier Integrals Extensions in Application Fourier Integrals Fourier Integrals Question: How to apply the idea of Fourier series to a In the limit (if it exists), as L , ∆p 0, → ∞ → non-periodic function over an infinite domain? 1 ∞ ∞ ∞ Answer: Magnify a single period to an infinite length. f (x) = cos px f (v) cos pv dv + sin px f (v) sin pv dv dp. π 0 Z  Z−∞ Z−∞  Fourier series of function fL(x) of period 2L: Fourier integral of f (x): ∞ fL(x) = a0 + (an cos pnx + bn sin pnx), ∞ n=1 f (x) = [A(p) cos px + B(p) sin px]dp, X 0 nπ Z where pn = L is the frequency of the n-th harmonic. where amplitude functions Inserting the expressions for the Fourier coefficients, 1 1 A(p) = ∞ f (v) cos pv dv and B(p) = ∞ f (v) sin pv dv L π π 1 Z−∞ Z−∞ fL(x) = fL(x)dx are defined for a continuous frequency variable p. 2L L Z− 1 ∞ L L In phase angle form, + cos p x f (v) cos p v dv + sin p x f (v) sin p v dv ∆p, π n L n n L n n L L 1 ∞ ∞ X=1  Z− Z−  f (x) = f (v) cos p(x v)dv dp. π π 0 − where ∆p = pn pn = . Z Z−∞ +1 − L Applied Mathematical Methods Fourier Series and Integrals 473, Applied Mathematical Methods Fourier Series and Integrals 474, Basic Theory of Fourier Series Basic Theory of Fourier Series Fourier Integrals Extensions in Application Points to note Extensions in Application Fourier Integrals Fourier Integrals iθ iθ e +e− Using cos θ = 2 in the phase angle form,

1 ∞ ∞ ip(x v) ip(x v) f (x) = f (v)[e − + e− − ]dv dp. 2π 0 I Z Z−∞ Fourier series arising out of a Sturm-Liouville problem With substitution p = q, I A versatile tool for function representation − I Fourier integral as the limiting case of Fourier series 0 ∞ ∞ ip(x v) ∞ iq(x v) f (v)e− − dv dp = f (v)e − dv dq. 0 Z Z−∞ Z−∞ Z−∞ Complex form of Fourier integral Necessary Exercises: 1,3,6,8 1 ∞ ∞ ip(x v) ∞ ipx f (x) = f (v)e − dv dp = C(p)e dp, 2π Z−∞ Z−∞ Z−∞ in which the complex Fourier integral coefficient is

1 ∞ ipv C(p) = f (v)e− dv. 2π Z−∞

Applied Mathematical Methods Fourier Transforms 475, Applied Mathematical Methods Fourier Transforms 476, Definition and Fundamental Properties Definition and Fundamental Properties Outline Important Results on Fourier Transforms Definition and Fundamental PropertiesImportant Results on Fourier Transforms Discrete Fourier Transform Discrete Fourier Transform Complex form of the Fourier integral:

1 ∞ 1 ∞ iwv iwt f (t) = f (v)e− dv e dw √2π √2π Z−∞  Z−∞  Fourier Transforms Composition of an infinite number of functions in the iwt Definition and Fundamental Properties form e , over a continuous distribution of frequency w. √2π Important Results on Fourier Transforms Discrete Fourier Transform Fourier transform: Amplitude of a frequency component:

1 ∞ iwt (f ) fˆ(w) = f (t)e− dt F ≡ √2π Z−∞ Function of the frequency variable. Inverse Fourier transform

1 1 ∞ iwt − (fˆ) f (t) = fˆ(w)e dw F ≡ √2π Z−∞ recovers the original function. Applied Mathematical Methods Fourier Transforms 477, Applied Mathematical Methods Fourier Transforms 478, Definition and Fundamental Properties Definition and Fundamental Properties Definition and Fundamental PropertiesImportant Results on Fourier Transforms Important Results on Fourier TransformsImportant Results on Fourier Transforms Discrete Fourier Transform Discrete Fourier Transform Example: Fourier transform of f (t) = 1? Fourier transform of the derivative of a function: Let us find out the inverse Fourier transform of fˆ(w) = kδ(w). If f (t) is continuous in every interval and f 0(t) is piecewise 1 ˆ 1 ∞ iwt k continuous, ∞ f (t) dt converges and f (t) approaches zero as f (t) = − (f ) = kδ(w)e dw = | | F √2π √2π t , −∞then Z−∞ → ∞ R √ 1 ∞ iwt (1) = 2πδ(w) f 0(t) = f 0(t)e− dt F F{ } √2π Linearity of Fourier transforms: Z−∞ 1 iwt 1 ∞ iwt = f (t)e− ∞ ( iw)f (t)e− dt αf1(t) + βf2(t) = αfˆ1(w) + βfˆ2(w) √2π −∞ − √2π − F{ } Z−∞ ˆ   Scaling: = iwf (w).

1 w 1 w Alternatively, differentiating the inverse Fourier transform, f (at) = fˆ and − fˆ = a f (at) F{ } a a F a | | | | d d 1   n  o [f (t)] = ∞ fˆ(w)eiwt dw Shifting rules: dt dt √2π  Z−∞  iwt0 f (t t0) = e− f (t) 1 ∞ ∂ iwt 1 = fˆ(w)e dw = − iwfˆ(w) . 1F{ − } iw t F{1 } − fˆ(w w ) = e 0 − fˆ(w) √2π ∂t F { } F { − 0 } F { } Z−∞ h i

Applied Mathematical Methods Fourier Transforms 479, Applied Mathematical Methods Fourier Transforms 480, Definition and Fundamental Properties Definition and Fundamental Properties Important Results on Fourier TransformsImportant Results on Fourier Transforms Important Results on Fourier TransformsImportant Results on Fourier Transforms Discrete Fourier Transform Discrete Fourier Transform Under appropriate premises, Convolution of two functions: 2 2 f 00(t) = (iw) fˆ(w) = w fˆ(w). ∞ F{ } − h(t) = f (t) g(t) = f (τ)g(t τ)dτ ∗ − In general, f (n)(t) = (iw)nfˆ(w). Z−∞ F{ } Fourier transform of an integral: hˆ(w) = h(t) If f (t) is piecewise continuous on every interval, F{ } 1 ∞ ∞ iwt ∞ f (t) dt converges and fˆ(0) = 0, then = f (τ)g(t τ)e− dτ dt −∞ | | √2π − Z−∞ Z−∞ R t 1 1 ∞ iwτ ∞ iw(t τ) f (τ)dτ = fˆ(w). = f (τ)e− g(t τ)e− − dt dτ F iw √2π − Z−∞  Z−∞ Z−∞  ∞ iwτ 1 ∞ iwt = f (τ)e− g(t0)e− 0 dt0 dτ Derivative of a Fourier transform (with respect to the frequency √  2π  variable): Z−∞ Z−∞ d n tnf (t) = i n fˆ(w), Convolution theorem for Fourier transforms: F{ } dw n ˆ ˆ n h(w) = √2πf (w)gˆ(w) if f (t) is piecewise continuous and ∞ t f (t) dt converges. −∞ | | R Applied Mathematical Methods Fourier Transforms 481, Applied Mathematical Methods Fourier Transforms 482, Definition and Fundamental Properties Definition and Fundamental Properties Important Results on Fourier TransformsImportant Results on Fourier Transforms Discrete Fourier Transform Important Results on Fourier Transforms Discrete Fourier Transform Discrete Fourier Transform Conjugate of the Fourier transform: Consider a signal f (t) from actual measurement or sampling. 1 ∞ iwt We want to analyze its amplitude spectrum (versus frequency). fˆ∗(w) = f ∗(t)e dt √2π For the FT, how to evaluate the integral over ( , )? Z−∞ −∞ ∞ Inner product of fˆ(w) and gˆ(w): Windowing: Sample the signal f (t) over a finite interval.

∞ ∞ 1 ∞ iwt fˆ∗(w)gˆ(w)dw = f ∗(t)e dt gˆ(w)dw A window function: √2π Z−∞ Z−∞ Z−∞ 1 for a t b ∞ 1 ∞ iwt g(t) = ≤ ≤ = f ∗(t) gˆ(w)e dw dt 0 otherwise √2π  Z−∞  Z−∞  ∞ Actual processing takes place on the windowed function f (t)g(t). = f ∗(t)g(t)dt. Z−∞ Next question: Do we need to evaluate the amplitude for all Parseval’s identity: For g(t) = f (t) in the above, w ( , )? ∈ −∞ ∞ ∞ fˆ(w) 2dw = ∞ f (t) 2dt, Most useful signals are particularly rich only in their own k k k k characteristic frequency bands. Z−∞ Z−∞ equating the total energy content of the frequency spectrum of a Decide on an expected frequency band, say [ wc , wc ]. wave or a signal to the total energy flow over time. −

Applied Mathematical Methods Fourier Transforms 483, Applied Mathematical Methods Fourier Transforms 484, Definition and Fundamental Properties Definition and Fundamental Properties Discrete Fourier Transform Important Results on Fourier Transforms Discrete Fourier Transform Important Results on Fourier Transforms Discrete Fourier Transform Discrete Fourier Transform Time step for sampling? With discrete data at tk = k∆ for k = 0, 1, 2, 3, , N 1, · · · − With N sampling over [a, b), ˆ ∆ k f(w) = mj f(t), wc ∆ π, √2π ≤ h i

iwj ∆ k data being collected at t = a, a + ∆, a + 2∆, , a + (N 1)∆, where mj = e− and m is an N N matrix. · · · − j × with N∆ = b a. − A similar discrete versionhof inversei Fourier transform. Nyquist critical frequency Reconstruction: a trigonometric interpolation of sampled data. Note the duality. I Structure of Fourier and inverse Fourier transforms reduces the I Decision of sampling rate ∆ determines the band of frequency problem with a system of linear equations [ (N 3) operations] content that can be accommodated. O to that of a matrix-vector multiplication [ (N 2) operations]. I Decision of the interval [a, b) dictates how finely the O I k frequency spectrum can be developed. Structure of matrix mj , with patterns of redundancies, opens up a trick to reduceh i it further to (N log N) operations. Shannon’s sampling theorem O A band-limited signal can be reconstructed from a finite Cooley-Tuckey algorithm: number of samples. fast Fourier transform (FFT) Applied Mathematical Methods Fourier Transforms 485, Applied Mathematical Methods Fourier Transforms 486, Definition and Fundamental Properties Definition and Fundamental Properties Discrete Fourier Transform Important Results on Fourier Transforms Points to note Important Results on Fourier Transforms Discrete Fourier Transform Discrete Fourier Transform DFT representation reliable only if the incoming signal is really band-limited in the interval [ wc , wc ]. − Frequencies beyond [ wc , wc ] distort the spectrum near w = wc −  by folding back. I Fourier transform as amplitude function in Fourier integral Aliasing I Basic operational tools in Fourier and inverse Fourier Detection: a posteriori transforms Bandpass filtering: If we expect a signal having components only I Conceptual notions of discrete Fourier transform (DFT) in certain frequency bands and want to get rid of unwanted noise frequencies,

for every band [w1, w2] of our interest, we define window ˆ function φ(w) with intervals [ w2, w1] and [w1, w2]. Necessary Exercises: 1,3,6 − − Windowed Fourier transform φˆ(w)fˆ(w) filters out frequency components outside this band. For recovery, convolve raw signal f (t) with IFT φ(t) of φˆ(w).

Applied Mathematical Methods Minimax Approximation* 487, Applied Mathematical Methods Minimax Approximation* 488, Approximation with Chebyshev polynomials Approximation with Chebyshev polynomials Outline Minimax Polynomial Approximation Approximation with Chebyshev polynomialsMinimax Polynomial Approximation Chebyshev polynomials: Polynomial solutions of the singular Sturm-Liouville problem

2 2 2 2 0 n (1 x )y 00 xy 0 + n y = 0 or 1 x y 0 + y = 0 − − − √1 x2 Minimax Approximation* hp i − over 1 x 1, with Tn(1) = 1 for all n. Approximation with Chebyshev polynomials − ≤ ≤ Minimax Polynomial Approximation Closed-form expressions:

1 Tn(x) = cos(n cos− x),

or,

T (x) = 1, T (x) = x, T (x) = 2x2 1, T (x) = 4x3 3x, ; 0 1 2 − 3 − · · · with the three-term recurrence relation

Tk+1(x) = 2xTk (x) Tk 1(x). − − Applied Mathematical Methods Minimax Approximation* 489, Applied Mathematical Methods Minimax Approximation* 490, Approximation with Chebyshev polynomials Approximation with Chebyshev polynomials Approximation with Chebyshev polynomialsMinimax Polynomial Approximation Approximation with Chebyshev polynomialsMinimax Polynomial Approximation Immediate observations I 1 Coefficients in a Chebyshev polynomial are integers. In P 8 (x) 1 T (x) n 0.8 8 particular, the leading coefficient of T (x) is 2 1. 0.8 n − 0.6 I 0.6 For even n, Tn(x) is an even function, while for odd n it is an 0.4 0.4 odd function. 0.2 0.2 0 3 y 0 I n T (x) T (1) = 1, T ( 1) = ( 1) and T (x) 1 for 1 x 1. −0.2 n n n −0.2 I − − | | ≤ − ≤ ≤ −0.4 Zeros of a Chebyshev polynomial T (x) are real and lie inside −0.4 n −0.6 k (2 1)π −0.8 −0.6 the interval [ 1, 1] at locations x = cos − for extrema 2n zeroes − −1 −0.8 k = 1, 2, 3, , n. −1.5 −1 −0.5 0 0.5 1 1.5 −1 · · · −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 These locations are also called Chebyshev accuracy points. x x Further, zeros of Tn(x) are interlaced by those of Tn+1(x). Figure: Extrema and zeros of T3(x) Figure: Contrast: P8(x) and T8(x) I Extrema of Tn(x) are of magnitude equal to unity, alternate in sign and occur at x = cos kπ for k = 0, 1, 2, 3, , n. n Being cosines and polynomials at the same time, Chebyshev I · · · Orthogonality and norms: polynomials possess a wide variety of interesting properties! 1 0 if m = n, Tm(x)Tn(x) 6 dx = π if m = n = 0, and 2 2 Most striking property: 1 √1 x  6 Z− −  π if m = n = 0. equal-ripple oscillations, leading to minimax property 

Applied Mathematical Methods Minimax Approximation* 491, Applied Mathematical Methods Minimax Approximation* 492, Approximation with Chebyshev polynomials Approximation with Chebyshev polynomials Approximation with Chebyshev polynomialsMinimax Polynomial Approximation Approximation with Chebyshev polynomialsMinimax Polynomial Approximation Minimax property Chebyshev series Theorem: Among all polynomials pn(x) of degree n > 0 1 n with the leading coefficient equal to unity, 2 − Tn(x) f (x) = a0T0(x) + a1T1(x) + a2T2(x) + a3T3(x) + deviates least from zero in [ 1, 1]. That is, · · · − with coefficients 1 n 1 n max pn(x) max 2 − Tn(x) = 2 − . 1 x 1 | | ≥ 1 x 1 | | 1 1 − ≤ ≤ − ≤ ≤ 1 f (x)T0(x) 2 f (x)Tn(x) a0 = dx and an = dx for n = 1, 2, 3, 2 2 π 1 √1 x π 1 √1 x · · · If there exists a monic polynomial pn(x) of degree n such that Z− − Z− − 1 n max pn(x) < 2 − , n 1 x 1 | | A truncated series k=0 ak Tk (x): − ≤ ≤ 1 n then at (n + 1) locations of alternating extrema of 2 − Tn(x), the Chebyshev economizationP polynomial 1 n Leading error term an+1Tn+1(x) deviates least from zero over qn(x) = 2 − Tn(x) pn(x) − [ 1, 1] and is qualitatively similar to the error function. 1 n − will have the same sign as 2 − Tn(x). With alternating signs at (n + 1) locations in sequence, qn(x) will Question: How to develop a Chebyshev series approximation? have n intervening zeros, even though it is a polynomial of degree Find out so many Chebyshev polynomials and evaluate coefficients? at most (n 1): CONTRADICTION! − Applied Mathematical Methods Minimax Approximation* 493, Applied Mathematical Methods Minimax Approximation* 494, Approximation with Chebyshev polynomials Approximation with Chebyshev polynomials Approximation with Chebyshev polynomialsMinimax Polynomial Approximation Minimax Polynomial Approximation Minimax Polynomial Approximation

For approximating f (t) over [a, b], scale the variable as Situations in which minimax approximation is desirable: a+b b a I Develop the approximation once and keep it for use in future. t = + − x, with x [ 1, 1]. 2 2 ∈ − n Requirement: Uniform quality control over the entire domain Remark: The economized series k=0 ak Tk (x) gives minimax deviation of the leading error term an Tn (x). Minimax approximation: P +1 +1 Assuming an+1Tn+1(x) to be the error, at the zeros of Tn+1(x), deviation limited by the constant amplitude of ripple the error will be ‘officially’ zero, i.e. Chebyshev’s minimax theorem n Theorem: Of all polynomials of degree up to n, p(x) is ak Tk (xj ) = f (t(xj )), the minimax polynomial approximation of f (x), i.e. it Xk=0 minimizes

where x , x , x , , xn are the roots of Tn (x). max f (x) p(x) , 0 1 2 · · · +1 | − | Recall: Values of an n-th degree polynomial at n + 1 if and only if there are n + 2 points xi such that points uniquely fix the entire polynomial. a x1 < x2 < x3 < < xn+2 b, Interpolation of these n + 1 values leads to the same polynomial! ≤ · · · ≤ where the difference f (x) p(x) takes its extreme values Chebyshev-Lagrange approximation − of the same magnitude and alternating signs.

Applied Mathematical Methods Minimax Approximation* 495, Applied Mathematical Methods Minimax Approximation* 496, Approximation with Chebyshev polynomials Approximation with Chebyshev polynomials Minimax Polynomial Approximation Minimax Polynomial Approximation Points to note Minimax Polynomial Approximation Utilize any gap to reduce the deviation at the other extrema with values at the bound. y d I Unique features of Chebyshev polynomials f(x) − p(x) I The equal-ripple and minimax properties ∆p(x) I ε/2 Chebyshev series and Chebyshev-Lagrange approximation l b O a m w n x I −ε/2 Fundamental ideas of general minimax approximation

ε −d Necessary Exercises: 2,3,4 Figure: Schematic of an approximation that is not minimax

Construction of the minimax polynomial: Remez algorithm

Note: In the light of this theorem and algorithm, examine how Tn+1(x) is qualitatively similar to the complete error function! Applied Mathematical Methods Partial Differential Equations 497, Applied Mathematical Methods Partial Differential Equations 498, Introduction Introduction Outline Hyperbolic Equations Introduction Hyperbolic Equations Parabolic Equations Parabolic Equations Elliptic Equations Elliptic Equations Two-Dimensional Wave Equation Quasi-linear second order PDE’s Two-Dimensional Wave Equation ∂2u ∂2u ∂2u a + 2b + c = F (x, y, u, u , u ) ∂x2 ∂x∂y ∂y 2 x y hyperbolic if b2 ac > 0, modelling phenomena which evolve in Partial Differential Equations − Introduction time perpetually and do not approach a steady state parabolic if b2 ac = 0, modelling phenomena which evolve in Hyperbolic Equations − Parabolic Equations time in a transient manner, approaching steady state Elliptic Equations elliptic if b2 ac < 0, modelling steady-state configurations, − Two-Dimensional Wave Equation without evolution in time If F (x, y, u, ux , uy ) = 0, second order linear homogeneous differential equation Principle of superposition: A linear combination of different solutions is also a solution. Solutions are often in the form of infinite series. I Solution techniques in PDE’s typically attack the boundary value problem directly.

Applied Mathematical Methods Partial Differential Equations 499, Applied Mathematical Methods Partial Differential Equations 500, Introduction Introduction Introduction Hyperbolic Equations Introduction Hyperbolic Equations Parabolic Equations Parabolic Equations Elliptic Equations Elliptic Equations Two-Dimensional Wave Equation Method of separation of variables Two-Dimensional Wave Equation Initial and boundary conditions For u(x, y), propose a solution in the form Time and space variables are qualitatively different. u(x, y) = X (x)Y (y) I Conditions in time: typically initial conditions. For second order PDE’s, u and ut over the entire space and substitute domain: Cauchy conditions I Time is a single variable and is decoupled from the space ux = X 0Y , uy = XY 0, uxx = X 00Y , uxy = X 0Y 0, uyy = XY 00 variables. to cast the equation into the form I Conditions in space: typically boundary conditions. For u(t, x, y), boundary conditions over the entire curve in the φ(x, X , X 0, X 00) = ψ(y, Y , Y 0, Y 00). x-y plane that encloses the domain. For second order PDE’s, I Dirichlet condition: value of the function If the manoeuvre succeeds then, x and y being independent I Neumann condition: derivative normal to the boundary variables, it implies I Mixed (Robin) condition φ(x, X , X 0, X 00) = ψ(y, Y , Y 0, Y 00) = k.

Dirichlet, Neumann and Cauchy problems Nature of the separation constant k is decided based on the context, resulting ODE’s are solved in consistency with the boundary conditions and assembled to construct u(x, y). Applied Mathematical Methods Partial Differential Equations 501, Applied Mathematical Methods Partial Differential Equations 502, Introduction Introduction Hyperbolic Equations Hyperbolic Equations Hyperbolic Equations Hyperbolic Equations Parabolic Equations Parabolic Equations Elliptic Equations Elliptic Equations Transverse vibrations of a string Two-Dimensional Wave Equation 2 T Two-Dimensional Wave Equation Under the assumptions, denoting c = ρ , Q T u θ+δθ 2 P ∂ u 2 ∂u ∂u δx δx = c . θ ∂t2 ∂x − ∂x T " Q P #

Q P In the limit, as δx 0, PDE of transverse vib ration: → 2 2 ∂ u 2 ∂ u O δx L x = c ∂t2 ∂x2 Figure: Transverse vibration of a stretched string one-dimensional wave equation Small deflection and slope: cos θ 1, sin θ θ tan θ ≈ ≈ ≈ Boundary conditions (in this case): u(0, t) = u(L, t) = 0 Horizontal (longitudinal) forces on PQ balance. Initial configuration and initial velocity: From Newton’s second law, vertical (transverse) deflection u(x, t): u(x, 0) = f (x) and ut (x, 0) = g(x) ∂2u T sin(θ + δθ) T sin θ = ρδx Cauchy problem: Determine u(x, t) for 0 x L, t 0. − ∂t2 ≤ ≤ ≥

Applied Mathematical Methods Partial Differential Equations 503, Applied Mathematical Methods Partial Differential Equations 504, Introduction Introduction Hyperbolic Equations Hyperbolic Equations Hyperbolic Equations Hyperbolic Equations Parabolic Equations Parabolic Equations Elliptic Equations Elliptic Equations Solution by separation of variables Two-Dimensional Wave Equation Two-Dimensional Wave Equation Corresponding solution:

2 utt = c uxx , u(0, t) = u(L, t) = 0, u(x, 0) = f (x), ut (x, 0) = g(x) Tn(t) = An cos λnt + Bn sin λnt

Assuming Then, for n = 1, 2, 3, , u(x, t) = X (x)T (t), · · · nπx and substituting u = XT and u = X T , variables are un(x, t) = Xn(x)Tn(t) = (An cos λnt + Bn sin λnt) sin tt 00 xx 00 L separated as T X 00 = 00 = p2. satisfies the PDE and the boundary conditions. c2T X − Since the PDE and the BC’s are homogeneous, by superposition, The PDE splits into two ODE’s 2 2 2 X 00 + p X = 0 and T 00 + c p T = 0. ∞ nπx u(x, t) = [An cos λnt + Bn sin λnt] sin . 2 nπ L Eigenvalues of BVP X 00 + p X = 0, X (0) = X (L) = 0 are p = L n=1 and eigenfunctions X

nπx Question: How to determine coefficients An and Bn? Xn(x) = sin px = sin for n = 1, 2, 3, . L · · · 2 cnπ Answer: By imposing the initial conditions. Second ODE: T 00 + λnT = 0, with λn = L Applied Mathematical Methods Partial Differential Equations 505, Applied Mathematical Methods Partial Differential Equations 506, Introduction Introduction Hyperbolic Equations Hyperbolic Equations Hyperbolic Equations Hyperbolic Equations Parabolic Equations Parabolic Equations Elliptic Equations Elliptic Equations Initial conditions: Fourier sine series of f (x)Twando-Dimensionalg(x) Wave Equation Two-Dimensional Wave Equation D’Alembert’s solution of the wave equation ∞ nπx u(x, 0) = f (x) = An sin L Method of characteristics n=1 X Canonical form ∞ nπx u (x, 0) = g(x) = λ B sin t n n L n By coordinate transformation from (x, y) to (ξ, η), with X=1 Hence, coefficients: U(ξ, η) = u[x(ξ, η), y(ξ, η)], hyperbolic equation: U = Φ 2 L nπx 2 L nπx ξη A = f (x) sin dx and B = g(x) sin dx n L L n cnπ L parabolic equation: Uξξ = Φ Z0 Z0 Related problems: elliptic equation: Uξξ + Uηη = Φ I Different boundary conditions: other kinds of series in which Φ(ξ, η, U, Uξ, Uη) is free from second derivatives. I Long wire: infinite domain, continuous frequencies and For a hyperbolic equation, entire domain becomes a network of ξ-η solution from Fourier integrals coordinate curves, known as characteristic curves, Alternative: Reduce the problem using Fourier transforms. 2 2 along which decoupled solutions can be tracked! I General wave equation in 3-d: utt = c u 2 ∇ I Membrane equation: utt = c (uxx + uyy )

Applied Mathematical Methods Partial Differential Equations 507, Applied Mathematical Methods Partial Differential Equations 508, Introduction Introduction Hyperbolic Equations Hyperbolic Equations Hyperbolic Equations Hyperbolic Equations Parabolic Equations Parabolic Equations Elliptic Equations Elliptic Equations For a hyperbolic equation in the form Two-Dimensional Wave Equation Substitution of derivatives Two-Dimensional Wave Equation ∂2u ∂2u ∂2u a + 2b + c = F (x, y, u, ux , uy ), ∂x2 ∂x∂y ∂y 2 ux = Uξξx + Uηηx = Uξ + Uη uxx = Uξξ + 2Uξη + Uηη ⇒ 2 2 2 2 ut = U ξt + U ηt = cU + cU utt = c U 2c U + c U roots of am + 2bm + c are ξ η − ξ η ⇒ ξξ − ξη ηη b √b2 ac 2 m = −  − , into the PDE utt = c uxx gives 1,2 a real and distinct. c2(U 2U + U ) = c2(U + 2U + U ). ξξ − ξη ηη ξξ ξη ηη Coordinate transformation Canonical form: Uξη = 0 ξ = y + m1x, η = y + m2x Integration: leads to Uξη = Φ(ξ, η, U, Uξ, Uη). For the BVP Uξ = Uξηdη + ψ(ξ) = ψ(ξ) 2 Z utt = c uxx , u(0, t) = u(L, t) = 0, u(x, 0) = f (x), ut (x, 0) = g(x), U(ξ, η) = ψ(ξ)dξ + f (η) = f (ξ) + f (η) canonical coordinate transformation: ⇒ 2 1 2 1 1 Z ξ = x ct, η = x + ct, with x = (ξ + η), t = (η ξ). D’Alembert’s solution: u(x, t) = f (x ct) + f (x + ct) − 2 2c − 1 − 2 Applied Mathematical Methods Partial Differential Equations 509, Applied Mathematical Methods Partial Differential Equations 510, Introduction Introduction Hyperbolic Equations Hyperbolic Equations Parabolic Equations Hyperbolic Equations Parabolic Equations Parabolic Equations Elliptic Equations Elliptic Equations Two-Dimensional Wave Equation Heat conduction equation or diffusion equationTwo-Dimensional: Wave Equation Physical insight from D’Alembert’s solution: ∂u 2 2 f (x ct): a progressive wave in forward direction with speed c = c u 1 − ∂t ∇ Reflection at boundary: One-dimensional heat (diffusion) equation: in a manner depending upon the boundary condition 2 ut = c uxx Reflected wave f2(x + ct): another progressive wave, this one in Heat conduction in a finite bar: For a thin bar of length L with backward direction with speed c end-points at zero temperature, Superposition of two waves: complete solution (response) 2 ut = c uxx , u(0, t) = u(L, t) = 0, u(x, 0) = f (x). cnπ Note: Components of the earlier solution: with λn = L , Assumption u(x, t) = X (x)T (t) leads to

nπx 1 nπ nπ 2 T 0 X 00 2 cos λnt sin = sin (x ct) + sin (x + ct) XT 0 = c X 00T = = p , L 2 L − L ⇒ c2T X − nπx 1 h nπ nπ i sin λnt sin = cos (x ct) cos (x + ct) giving rise to two ODE’s as L 2 L − − L 2 2 2 h i X 00 + p X = 0 and T 0 + c p T = 0.

Applied Mathematical Methods Partial Differential Equations 511, Applied Mathematical Methods Partial Differential Equations 512, Introduction Introduction Parabolic Equations Hyperbolic Equations Parabolic Equations Hyperbolic Equations Parabolic Equations Parabolic Equations 2 Elliptic Equations Elliptic Equations BVP in the space coordinate X 00 + p X = Tw0,o-DimensionalX (0) =WaveX (EquationL) = 0 Non-homogeneous boundary conditions:Two-Dimensional Wave Equation has solutions 2 nπx ut = c uxx , u(0, t) = u1, u(L, t) = u2, u(x, 0) = f (x). Xn(x) = sin . L For u1 = u2, with u(x, t) = X (x)T (t), BC’s do not separate! cnπ 6 With λn = L , the ODE in T (t) has the corresponding solutions Assume

λ2 t u(x, t) = U(x, t) + uss (x), Tn(t) = Ane− n . where component uss (x), steady-state temperature (distribution), By superposition, does not enter the differential equation. u2 u1 ∞ nπx λ2 t uss00 (x) = 0, uss (0) = u1, uss (L) = u2 uss (x) = u1 + − x u(x, t) = An sin e− n , ⇒ L L n X=1 Substituting into the BVP, 2 coefficients being determined from initial condition as Ut = c Uxx , U(0, t) = U(L, t) = 0, U(x, 0) = f (x) uss (x). − ∞ nπx Final solution: u(x, 0) = f (x) = An sin , L ∞ nπx 2 n=1 λnt u(x, t) = B sin e− + u (x), X n L ss n a Fourier sine series. X=1 As t , u(x, t) 0 (steady state) Bn being coefficients of Fourier sine series of f (x) uss (x). → ∞ → − Applied Mathematical Methods Partial Differential Equations 513, Applied Mathematical Methods Partial Differential Equations 514, Introduction Introduction Parabolic Equations Hyperbolic Equations Parabolic Equations Hyperbolic Equations Parabolic Equations Parabolic Equations Elliptic Equations Elliptic Equations Heat conduction in an infinite wire Two-Dimensional Wave Equation Solution using Fourier transforms Two-Dimensional Wave Equation

2 2 ut = c uxx , u(x, 0) = f (x) ut = c uxx , u(x, 0) = f (x) nπ Using derivative formula of Fourier transforms, In place of L , now we have continuous frequency p. Solution as superposition of all frequencies: 2 2 ∂uˆ 2 2 (ut ) = c (iw) (u) = c w uˆ, F F ⇒ ∂t − ∞ ∞ c2 p2t u(x, t) = up(x, t)dp = [A(p) cos px+B(p) sin px]e− dp since variables x and t are independent. Z0 Z0 Initial value problem in uˆ(w, t): ∂uˆ Initial condition = c2w 2uˆ, uˆ(0) = fˆ(w) ∂t − ∞ c2 w 2t u(x, 0) = f (x) = [A(p) cos px + B(p) sin px]dp Solution: uˆ(w, t) = fˆ(w)e− Z0 Inverse Fourier transform gives solution of the original problem as

gives the Fourier integral of f (x) and amplitude functions 1 1 ∞ c2 w 2t iwx u(x, t) = − uˆ(w, t) = fˆ(w)e− e dw F { } √2π 1 ∞ 1 ∞ Z−∞ A(p) = f (v) cos pv dv and B(p) = f (v) sin pv dv. 1 ∞ ∞ c2w 2t π π u(x, t) = f (v) cos(wx wv)e− dw dv. Z−∞ Z−∞ ⇒ π 0 − Z−∞ Z

Applied Mathematical Methods Partial Differential Equations 515, Applied Mathematical Methods Partial Differential Equations 516, Introduction Introduction Elliptic Equations Hyperbolic Equations Elliptic Equations Hyperbolic Equations Parabolic Equations Parabolic Equations Elliptic Equations 2 Elliptic Equations nπx Heat flow in a plate: two-dimensional heat equationTwo-Dimensional Wave Equation From BVP X 00 + p X = 0, X (0) = X (a) =Tw0,o-DimensionalXn(xW) ave=Equationsin a 2 ∂u ∂2u ∂2u Corresponding solution of Y 00 p Y = 0: = c2 + − ∂t ∂x2 ∂y 2 nπy nπy   Yn(y) = An cosh + Bn sinh Steady-state temperature distribution: a a 2 2 ∂ u ∂ u Condition Y (0) = 0 An = 0, and + = 0 ⇒ ∂x2 ∂y 2 nπx nπy un(x, y) = Bn sin sinh Laplace’s equation a a Steady-state heat flow in a rectangular plate: The complete solution:

uxx + uyy = 0, u(0, y) = u(a, y) = u(x, 0) = 0, u(x, b) = f (x); ∞ nπx nπy u(x, y) = B sin sinh a Dirichlet problem over the domain 0 x a, 0 y b. n a a ≤ ≤ ≤ ≤ n Proposal u(x, y) = X (x)Y (y) leads to X=1

X 00 Y 00 2 The last boundary condition u(x, b) = f (x) fixes the coefficients X 00Y + XY 00 = 0 = = p . ⇒ X − Y − from the Fourier sine series of f (x). Separated ODE’s: Note: In the example, BC’s on three sides were homogeneous. 2 2 X 00 + p X = 0 and Y 00 p Y = 0 How did it help? What if there are more non-homogeneous BC’s? − Applied Mathematical Methods Partial Differential Equations 517, Applied Mathematical Methods Partial Differential Equations 518, Introduction Introduction Elliptic Equations Hyperbolic Equations Two-Dimensional Wave Equation Hyperbolic Equations Parabolic Equations Parabolic Equations Elliptic Equations Elliptic Equations Steady-state heat flow with internal heatTwo-DimensionalgenerationWave Equation Transverse vibration of a rectangular membTwo-Dimensionalrane: Wave Equation ∂2u ∂2u ∂2u 2u = φ(x, y) = c2 + ∇ ∂t2 ∂x2 ∂y 2   Poisson’s equation A Cauchy problem of the membrane: Separation of variables impossible! u = c2(u + u ); u(x, y, 0) = f (x, y), u (x, y, 0) = g(x, y); Consider function u(x, y) as tt xx yy t u(0, y, t) = u(a, y, t) = u(x, 0, t) = u(x, b, t) = 0.

u(x, y) = uh(x, y) + up(x, y) Separate the time variable from the space variables: Sequence of steps Fxx + Fyy T 00 I one particular solution u (x, y) that may or may not satisfy u(x, y, t) = F (x, y)T (t) = = λ2 p ⇒ F c2T − some or all of the boundary conditions I solution of the corresponding homogeneous equation, namely Helmholtz equation: u + u = 0 for u (x, y) xx yy h 2 Fxx + Fyy + λ F = 0 I such that u = uh + up satisfies all the boundary conditions

Applied Mathematical Methods Partial Differential Equations 519, Applied Mathematical Methods Partial Differential Equations 520, Introduction Introduction Two-Dimensional Wave Equation Hyperbolic Equations Two-Dimensional Wave Equation Hyperbolic Equations Parabolic Equations Parabolic Equations Elliptic Equations Elliptic Equations Assuming F (x, y) = X (x)Y (y), Two-Dimensional Wave Equation Composing Xm(x), Yn(y) and Tmn(t) and supTwo-Dimensionalerposing,Wave Equation

2 ∞ ∞ X 00 Y 00 + λ Y 2 mπx nπy = = µ u(x, y, t) = [Amn cos cλmnt+Bmn sin cλmnt] sin sin , a b X − Y − m=1 n=1 2 2 X X X 00 + µ X = 0 and Y 00 + ν Y = 0, coefficients being determined from the double Fourier series ⇒ such that λ = µ2 + ν2. ∞ ∞ mπx nπy f (x, y) = A sin sin mn a b With BC’s X (0)p= X (a) = 0 and Y (0) = Y (b) = 0, m=1 n=1 X X mπx nπy ∞ ∞ mπx nπy X (x) = sin and Y (y) = sin . and g(x, y) = cλmnBmn sin sin . m a n b a b m=1 n=1 Corresponding values of λ are X X

mπ 2 nπ 2 BVP’s modelled in polar coordinates λmn = + For domains of circular symmetry, important in many practical a b r    systems, the BVP is conveniently modelled in polar coordinates, with solutions of T + c2λ2T = 0 as 00 the separation of variables quite often producing I Tmn(t) = Amn cos cλmnt + Bmn sin cλmnt. Bessel’s equation, in cylindrical coordinates, and I Legendre’s equation, in spherical coordinates Applied Mathematical Methods Partial Differential Equations 521, Applied Mathematical Methods Analytic Functions 522, Introduction Analyticity of Complex Functions Points to note Hyperbolic Equations Outline Conformal Mapping Parabolic Equations Potential Theory Elliptic Equations Two-Dimensional Wave Equation

I PDE’s in physically relevant contexts I Initial and boundary conditions I Separation of variables Analytic Functions I Examples of boundary value problems with hyperbolic, Analyticity of Complex Functions parabolic and elliptic equations Conformal Mapping I Modelling, solution and interpretation Potential Theory I Cascaded application of separation of variables for problems with more than two independent variables

Necessary Exercises: 1,2,4,7,9,10

Applied Mathematical Methods Analytic Functions 523, Applied Mathematical Methods Analytic Functions 524, Analyticity of Complex Functions Analyticity of Complex Functions Analyticity of Complex Functions Conformal Mapping Analyticity of Complex Functions Conformal Mapping Potential Theory Potential Theory Function f of a complex variable z Derivative of a complex function: gives a rule to associate a unique complex number f (z) f (z0) f (z0 + δz) f (z0) w = u + iv to every z = x + iy in a set. f 0(z0) = lim − = lim − z z0 z z δz 0 δz → − 0 → Limit: If f (z) is defined in a neighbourhood of z (except possibly 0 When this limit exists, function f (z) is said to be differentiable. at z0 itself) and l C such that  > 0, δ > 0 such that ∃ ∈ ∀ ∃ Extremely restrictive definition! 0 < z z0 < δ f (z) l < , | − | ⇒ | − | Analytic function then A function f (z) is called analytic in a domain D if it is l = lim f (z). z z defined and differentiable at all points in D. → 0 Crucial difference from real functions: z can approach z0 in all Points to be settled later: possible manners in the complex plane. I Derivative of an analytic function is also analytic. Definition of the limit is more restrictive. I An analytic function possesses derivatives of all orders.

Continuity: limz z f (z) = f (z0) A great qualitative difference between functions of a real variable → 0 Continuity in a domain D: continuity at every point in D and those of a complex variable! Applied Mathematical Methods Analytic Functions 525, Applied Mathematical Methods Analytic Functions 526, Analyticity of Complex Functions Analyticity of Complex Functions Analyticity of Complex Functions Conformal Mapping Analyticity of Complex Functions Conformal Mapping Potential Theory Potential Theory Cauchy-Riemann conditions Cauchy-Riemann equations or conditions If f (z) = u(x, y) + iv(x, y) is analytic then ∂u = ∂v and ∂u = ∂v ∂x ∂y ∂y − ∂x δu + iδv are necessary for analyticity. f 0(z) = lim δx,δy 0 δx + iδy → Question: Do the C-R conditions imply analyticity? along all paths of approach for δz = δx + iδy 0 or δx, δy 0. Consider u(x, y) and v(x, y) having continuous first order partial → → derivatives that satisfy the Cauchy-Riemann conditions. y y 3 δ δ By mean value theorem, 2 z = i y z 0 ∂u ∂u δ δ z0 z = x δu = u(x + δx, y + δy) u(x, y) = δx (x1, y1) + δy (x1, y1) 4 1 − ∂x ∂y O x 5 O x with x = x + ξδx, y = y + ξδy for some ξ [0, 1]; and 1 1 ∈ ∂v ∂v δv = v(x + δx, y + δy) v(x, y) = δx (x , y ) + δy (x , y ) − ∂x 2 2 ∂y 2 2 Figure: Paths approaching z0 Figure: Paths in C-R equations with x = x + ηδx, y = y + ηδy for some η [0, 1]. 2 2 ∈ Two expressions for the derivative: Then, ∂u ∂v ∂v ∂u ∂u ∂v ∂v ∂u δf = δx (x , y ) + iδy (x , y ) +i δx (x , y ) iδy (x , y ) . f 0(z) = + i = i ∂x 1 1 ∂y 2 2 ∂x 2 2 − ∂y 1 1 ∂x ∂x ∂y − ∂y    

Applied Mathematical Methods Analytic Functions 527, Applied Mathematical Methods Analytic Functions 528, Analyticity of Complex Functions Analyticity of Complex Functions Analyticity of Complex Functions Conformal Mapping Analyticity of Complex Functions Conformal Mapping Potential Theory Potential Theory Using C-R conditions ∂v = ∂u and ∂u = ∂v , ∂y ∂x ∂y − ∂x Harmonic function ∂v ∂u ∂u ∂v ∂u ∂u ∂u Differentiating C-R equations ∂y = ∂x and ∂y = ∂x , δf = (δx + iδy) (x , y ) + iδy (x , y ) (x , y ) − ∂x 1 1 ∂x 2 2 − ∂x 1 1   ∂2u ∂2v ∂2u ∂2v ∂2u ∂2v ∂2u ∂2v ∂v ∂v ∂v = , = , = , = + i(δx + iδy) (x1, y1) + iδx (x2, y2) (x1, y1) ∂x2 ∂x∂y ∂y 2 −∂y∂x ∂y∂x ∂y 2 ∂x∂y −∂x2 ∂x ∂x − ∂x   δf ∂u ∂v ∂2u ∂2u ∂2v ∂2v = (x1, y1) + i (x1, y1) + + = 0 = + . ⇒ δz ∂x ∂x ⇒ ∂x2 ∂y 2 ∂x2 ∂y 2 δx ∂v ∂v δy ∂u ∂u i (x , y ) (x , y ) + i (x , y ) (x , y ) . δz ∂x 2 2 − ∂x 1 1 δz ∂x 2 2 − ∂x 1 1 Real and imaginary components of an analytic functions     are harmonic functions. Since δx , δy 1, as δz 0, the limit exists and δz δz ≤ → Conjugate harmonic function of u(x, y): v(x, y)

∂u ∂v ∂u ∂v f 0(z) = + i = i + . Families of curves u(x, y) = c and v(x, y) = k are mutually ∂x ∂x − ∂y ∂y orthogonal, except possibly at points where f 0(z) = 0. Cauchy-Riemann conditions are necessary and sufficient Question: If u(x, y) is given, then how to develop the complete for function w = f (z) = u(x, y) + iv(x, y) to be analytic. analytic function w = f (z) = u(x, y) + iv(x, y)? Applied Mathematical Methods Analytic Functions 529, Applied Mathematical Methods Analytic Functions 530, Analyticity of Complex Functions Analyticity of Complex Functions Conformal Mapping Conformal Mapping Conformal Mapping Conformal Mapping Potential Theory Potential Theory Function: mapping of elements in domain to their images in range Conformal mapping: a mapping that preserves the angle between Depiction of a complex variable requires a plane with two axes. any two directions in magnitude and sense. Mapping of a complex function w = f (z) is shown in two planes. Verify: w = ez defines a conformal mapping. z Example: mapping of a rectangle under transformation w = e Through relative orientations of curves at the points of

3.5 intersection, ‘local’ shape of a figure is preserved.

2 3

C’ Take curve z(t), z(0) = z and image w(t) = f [z(t)], w = f (z ). D C 0 0 0 1.5 2.5 For analytic f (z), w˙ (0) = f 0(z0)z˙ (0), implying 2 1 1.5 w˙ (0) = f 0(z0) z˙ (0) and arg w˙ (0) = arg f 0(z0) + arg z˙(0). v

y 0.5 | | | | | | 1 D’ For several curves through z , 0.5 0 0 O A B 0 image curves pass through w0 and all of them turn by the O’ B’ −0.5 A’ −0.5 same angle arg f 0(z0).

−1 −1 −1 −0.5 0 0.5 1 1.5 2 −1 −0.5 0 0.5 1 1.5 2 2.5 3 3.5 x u Cautions (a) The z-plane (b) The w-plane I f 0(z) varies from point to point. Different scaling and turning effects take place at different points. ‘Global’ shape changes. z I Figure: Mapping corresponding to function w = e For f 0(z) = 0, argument is undefined and conformality is lost.

Applied Mathematical Methods Analytic Functions 531, Applied Mathematical Methods Analytic Functions 532, Analyticity of Complex Functions Analyticity of Complex Functions Conformal Mapping Conformal Mapping Potential Theory Conformal Mapping Potential Theory Potential Theory An analytic function defines a conformal mapping except Riemann mapping theorem: Let D be a simply connected at its critical points where its derivative vanishes. domain in the z-plane bounded by a closed curve C. Then there exists a conformal mapping that gives a one-to-one correspondence Except at critical points, an analytic function is invertible. between D and the unit disc w < 1 as well as between C and the | | We can establish an inverse of any conformal mapping. unit circle w = 1, bounding the unit disc. | | Examples Application to boundary value problems I Linear function w = az + b (for a = 0) I 6 First, establish a conformal mapping between the given I Linear fractional transformation domain and a domain of simple geometry. az + b I w = , ad bc = 0 Next, solve the BVP in this simple domain. cz + d − 6 I Finally, using the inverse of the conformal mapping, construct n z I Other elementary functions like z , e etc the solution for the given domain. Special significance of conformal mappings: Example: Dirichlet problem with Poisson’s integral formula A harmonic function φ(u, v) in the w-plane is also a harmonic function, in the form φ(x, y) in the z-plane, as 1 2π (R2 r 2)f (Reiφ) f (reiθ) = dφ long as the two planes are related through a conformal 2 − 2 2π 0 R 2Rr cos(θ φ) + r mapping. Z − − Applied Mathematical Methods Analytic Functions 533, Applied Mathematical Methods Analytic Functions 534, Analyticity of Complex Functions Analyticity of Complex Functions Potential Theory Conformal Mapping Points to note Conformal Mapping Potential Theory Potential Theory

Two-dimensional potential flow I ∂φ Velocity potential φ(x, y) gives velocity components Vx = ∂x ∂φ I Analytic functions and Cauchy-Riemann conditions and Vy = ∂y . I I A streamline is a curve in the flow field, the tangent to which Conformality of analytic functions at any point is along the local velocity vector. I Applications in solving BVP’s and flow description I Stream function ψ(x, y) remains constant along a streamline. I ψ(x, y) is the conjugate harmonic function of φ(x, y). I Complex potential function Φ(z) = φ(x, y) + iψ(x, y) defines Necessary Exercises: 1,2,3,4,7,9 the flow.

If a flow field encounters a solid boundary of a complicated shape, transform the boundary conformally to a simple boundary to facilitate the study of the flow pattern.

Applied Mathematical Methods Integrals in the Complex Plane 535, Applied Mathematical Methods Integrals in the Complex Plane 536, Line Integral Line Integral Outline Cauchy’s Integral Theorem Line Integral Cauchy’s Integral Theorem Cauchy’s Integral Formula Cauchy’s Integral Formula For w = f (z) = u(x, y) + iv(x, y), over a smooth curve C,

f (z)dz = (u+iv)(dx+idy) = (udx vdy)+i (vdx+udy). − ZC ZC ZC ZC Extension to piecewise smooth curves is obvious. Integrals in the Complex Plane With parametrization, for z = z(t), a t b, with z˙ (t) = 0, Line Integral ≤ ≤ 6 Cauchy’s Integral Theorem b f (z)dz = f [z(t)]z˙ (t)dt. Cauchy’s Integral Formula ZC Za Over a simple closed curve, contour integral: C f (z)dz Example: zndz for integer n, around circle z = ρe iθ C H H 2π 0 for n = 1, zndz = iρn+1 ei(n+1)θdθ = 6 − 2πi for n = 1. IC Z0  − The M-L inequality: If C is a curve of finite length L and f (z) < M on C, then | | f (z)dz f (z) dz < M dz = ML. ≤ | | | | | | ZC ZC ZC

Applied Mathematical Methods Integrals in the Complex Plane 537, Applied Mathematical Methods Integrals in the Complex Plane 538, Line Integral Line Integral Cauchy’s Integral Theorem Cauchy’s Integral Theorem Cauchy’s Integral Theorem Cauchy’s Integral Theorem Cauchy’s Integral Formula Cauchy’s Integral Formula I C is a simple closed curve in a simply connected domain D. Principle of path independence I Function f (z) = u + iv is analytic in D. Two points z1 and z2 on the close curve C I two open paths C1 and C2 from z1 to z2 Contour integral C f (z)dz =? Cauchy’s theorem on C, comprising of C1 in the forward direction If f 0(z) is continuous, then by Green’s theorem in the plane, H and C2 in the reverse direction: ∂v ∂u ∂u ∂v z2 f (z)dz = dxdy+i dxdy, f (z)dz f (z)dz = 0 f (z)dz = f (z)dz = f (z)dz C R −∂x − ∂y R ∂x − ∂y − ⇒ I Z Z   Z Z   ZC1 ZC2 Zz1 ZC1 ZC2 where R is the region enclosed by C. From C-R conditions, f (z)dz = 0. For an analytic function f (z) in a simply connected C domain D, z2 f (z)dz is independent of the path and Proof by Goursat: without the hypothesis of continuity of f (z) z1 H 0 depends only on the end-points, as long as the path is R Cauchy-Goursat theorem completely contained in D. If f (z) is analytic in a simply connected domain D, then Consequence: Definition of the function C f (z)dz = 0 for every simple closed curve C in D. z F (z) = f (ξ)dξ ImpoHrtance of Goursat’s contribution: z Z 0 I continuity of f 0(z) appears as consequence! What does the formulation suggest?

Applied Mathematical Methods Integrals in the Complex Plane 539, Applied Mathematical Methods Integrals in the Complex Plane 540, Line Integral Line Integral Cauchy’s Integral Theorem Cauchy’s Integral Theorem Cauchy’s Integral Theorem Cauchy’s Integral Theorem Cauchy’s Integral Formula Cauchy’s Integral Formula Indefinite integral Principle of deformation of paths Question: Is F (z) analytic? Is F 0(z) = f (z)?

C * z+δz z F (z + δz) F (z) 1 f (z) analytic everywhere other z2 − f (z) = f (ξ)dξ f (ξ)dξ f (z) s1 D δz − δz z − z − Z 0 Z 0  than isolated points s1, s2, s3 C1 z z C 1 +δ 2 = [f (ξ) f (z)]dξ δz − z z1 s3 Z C3 s f (z)dz = f (z)dz = f (z)dz 2 f is continuous , δ such that ξ z < δ f (ξ) f (z) <  ⇒ ∀ ∃ | − | ⇒ | − | C1 C2 C3 Choosing δz < δ, Z Z Z Not so for path C . F (z + δz) F (z)  z+δz ∗ Figure: Path deformation − f (z) < dξ = . δz − δz Zz

The line integral remains unaltered through a continuous If f (z ) is analytic in a simply connected domain D, then there exists an analytic function F (z) in D such that deformation of the path of integration with fixed end-points, as long as the sweep of the deformation z2 includes no point where the integrand is non-analytic. F 0(z) = f (z) and f (z)dz = F (z ) F (z ). 2 − 1 Zz1 Applied Mathematical Methods Integrals in the Complex Plane 541, Applied Mathematical Methods Integrals in the Complex Plane 542, Line Integral Line Integral Cauchy’s Integral Theorem Cauchy’s Integral Theorem Cauchy’s Integral Formula Cauchy’s Integral Theorem Cauchy’s Integral Formula Cauchy’s Integral Formula Cauchy’s theorem in multiply connected domain f (z): analytic function in a simply connected domain D

L1 C C For z D and simple closed curve C in D, 1 0 ∈ C2 L2 f (z) dz = 2πif (z ). z z 0 IC − 0 C3 Consider C as a circle with centre at z and radius ρ, L3 0 with no loss of generality (why?). Figure: Contour for multiply connected domain

f (z)dz f (z)dz f (z)dz f (z)dz = 0. f (z) dz f (z) f (z0) C − C − C − C dz = f (z0) + − dz I I 1 I 2 I 3 C z z0 C z z0 C z z0 I − I − I − If f (z) is analytic in a region bounded by the contour C From continuity of f (z), δ such that for any , as the outer boundary and non-overlapping contours C1, ∃ C2, C3, , Cn as inner boundaries, then f (z) f (z0)  · · · z z0 < δ f (z) f (z0) <  and − < , n | − | ⇒ | − | z z ρ − 0 f (z)dz = f (z)dz. with ρ < δ. From M-L inequality, the second integral vanishes. C i Ci I X=1 I

Applied Mathematical Methods Integrals in the Complex Plane 543, Applied Mathematical Methods Integrals in the Complex Plane 544, Line Integral Line Integral Cauchy’s Integral Formula Cauchy’s Integral Theorem Cauchy’s Integral Formula Cauchy’s Integral Theorem Cauchy’s Integral Formula Cauchy’s Integral Formula Direct applications Poisson’s integral formula iθ iφ I Evaluation of contour integral: Taking z0 = re and z = Re (with r < R) in Cauchy’s formula, I If g(z) is analytic on the contour and in the enclosed region, 2π f (Reiφ) 2πif (reiθ) = (iReiφ)dφ. the Cauchy’s theorem implies C g(z)dz = 0. iφ iθ I 0 Re re If the contour encloses a singularity at z0, then Cauchy’s Z − formula supplies a non-zero contributionH to the integral, if How to get rid of imaginary quantities from the expression? 2 f (z) = g(z)(z z ) is analytic. Develop a complement. With R in place of r, − 0 r I Evaluation of function at a point: If finding the integral on 2π iφ 2π iφ f (Re ) iφ f (Re ) iθ the left-hand-side is relatively simple, then we use it to 0 = 2 (iRe )dφ = (ire− )dφ. iφ R iθ re iθ Re iφ 0 Re r e 0 − − evaluate f (z0). Z − Z − Subtracting, Significant in the solution of boundary value 2π iφ iθ problems! Re re− 2πif (reiθ) = i f (Reiφ) + dφ Reiφ reiθ Re iφ re iθ Example: Poisson’s integral formula Z0  − − − −  2π (R2 r 2)f (Reiφ) 2π 2 2 1 (R r )u(R, φ) = i iφ −iθ iφ iθ dφ u(r, θ) = − dφ 0 (Re re )(Re− re− ) 2π R2 2Rr cos(θ φ) + r 2 Z − − Z0 − − 1 2π (R2 r 2)f (Reiφ) f (reiθ) = − dφ. for the Dirichlet problem over a circular disc. ⇒ 2π R2 2Rr cos(θ φ) + r 2 Z0 − − Applied Mathematical Methods Integrals in the Complex Plane 545, Applied Mathematical Methods Integrals in the Complex Plane 546, Line Integral Line Integral Cauchy’s Integral Formula Cauchy’s Integral Theorem Cauchy’s Integral Formula Cauchy’s Integral Theorem Cauchy’s Integral Formula Cauchy’s Integral Formula Cauchy’s integral formula evaluates contour integral of g(z), f (z + δz) f (z ) 1 1 1 if the contour encloses a point z0 where g(z) is 0 − 0 = f (z) dz non-analytic but g(z)(z z ) is analytic. δz 2πiδz C z z0 δz − z z0 0 I  − − −  − 1 f (z)dz 2 If g(z)(z z0) is also non-analytic, but g(z)(z z0) is analytic? = − − 2πi C (z z0 δz)(z z0) I − − − 1 f (z)dz 1 1 1 1 f (z) = + f (z) dz f (z ) = dz, 2πi (z z )2 2πi (z z δz)(z z ) − (z z )2 0 2πi z z IC − 0 IC  − 0 − − 0 − 0  IC − 0 1 f (z)dz 1 f (z)dz 1 f (z) = 2 + δz 2 f 0(z0) = 2 dz, 2πi C (z z0) 2πi C (z z0 δz)(z z0) 2πi C (z z0) I − I − − − I − If f (z) < M on C, L is path length and d = min z z , 2! f (z) | | 0 | − 0| f 00(z ) = dz, 0 2πi (z z )3 f (z)dz ML δz IC 0 δz < | | 0 as δz 0. = − , (z z δz)(z z )2 d 2(d δz ) → → · · · · · · · · · · · · IC 0 0 0 0 n! f (z) − − − − | | (n) f (z0) = n+1 dz. An analytic function possesses derivatives of all orders at 2πi C (z z0) I − every point in its domain. The formal expressions can be established through differentiation under the integral sign. Analyticity implies much more than mere differentiability!

Applied Mathematical Methods Integrals in the Complex Plane 547, Applied Mathematical Methods Singularities of Complex Functions 548, Line Integral Series Representations of Complex Functions Points to note Cauchy’s Integral Theorem Outline Zeros and Singularities Cauchy’s Integral Formula Residues Evaluation of Real Integrals

I Concept of line integral in complex plane I Cauchy’s integral theorem Singularities of Complex Functions I Consequences of analyticity Series Representations of Complex Functions I Cauchy’s integral formula Zeros and Singularities I Derivatives of arbitrary order for analytic functions Residues Evaluation of Real Integrals

Necessary Exercises: 1,2,5,7 Applied Mathematical Methods Singularities of Complex Functions 549, Applied Mathematical Methods Singularities of Complex Functions 550, Series Representations of Complex Functions Series Representations of Complex Functions Series Representations of Complex FunctionsZeros and Singularities Series Representations of Complex FunctionsZeros and Singularities Residues Residues Evaluation of Real Integrals Evaluation of Real Integrals Taylor’s series of function f (z), analytic in a neighbourhood of z0: Laurent’s series: If f (z) is analytic on circles C1 (outer) and C2 (inner) with centre at z0, and in the annulus in between, then ∞ n 2 3 f (z) = an(z z ) = a +a (z z )+a (z z ) +a (z z ) + , − 0 0 1 − 0 2 − 0 3 − 0 · · · n=0 ∞ n ∞ m ∞ cm X f (z) = an(z z ) = bm(z z ) + ; − 0 − 0 (z z )m with coefficients n= m=0 m=1 0 X−∞ X X − 1 1 f (w)dw (n) with coefficients an = f (z0) = n+1 , n! 2πi C (w z0) I − 1 f (w)dw where C is a circle with centre at z . a = ; 0 n 2πi (w z )n+1 Form of the series and coefficients: similar to real functions IC − 0 1 f (w)dw 1 The series representation is convergent within a disc m 1 or, bm = m+1 , cm = f (w)(w z0) − dw; z z < R, where radius of convergence R is the 2πi C (w z0) 2πi C − | − 0| I − I distance of the nearest singularity from z0. the contour C lying in the annulus and enclosing C2.

Note: No valid power series representation around z0, i.e. in Validity of this series representation: in annular region obtained by powers of (z z ), if f(z) is not analytic at z growing C1 and shrinking C2 till f (z) ceases to be analytic. − 0 0 Question: In that case, what about a series representation that Observation: If f (z) is analytic inside C2 as well, then cm = 0 and includes negative powers of (z z ) as well? Laurent’s series reduces to Taylor’s series. − 0

Applied Mathematical Methods Singularities of Complex Functions 551, Applied Mathematical Methods Singularities of Complex Functions 552, Series Representations of Complex Functions Series Representations of Complex Functions Series Representations of Complex FunctionsZeros and Singularities Series Representations of Complex FunctionsZeros and Singularities Residues Residues Proof of Laurent’s series Evaluation of Real Integrals Proof of Laurent’s series (contd) Evaluation of Real Integrals z z0 Cauchy’s integral formula for any point z in the annulus, Using q = w− z , − 0 1 f (w)dw 1 f (w)dw n 1 n f (z) = . 1 1 z z0 (z z0) − z z0 1 2πi w z − 2πi w z = + − 2 + + − n + − C1 C2 w z w z (w z ) · · · (w z ) w z w z I − I − − − 0 − 0 − 0  − 0  − 1 f (w)dw n 1 Organization of the series: = a0 + a1(z z0) + + an 1(z z0) − + Tn, z − ⇒ 2πi C1 w z − · · · − w I − with coefficients as required and 1 1 z0 = C C n w z (w z0)[1 (z z0)/(w z0)] 2 1 1 z z0 f (w) − − − − − Tn = − dw. 1 1 2πi w z w z = IC1  − 0  − w z −(z z0)[1 (w z0)/(z z0)] w z0 − − − − − Figure: The annulus Similarly, with q = z −z , − 0 1 f (w)dw 1 n = a 1(z z0)− + + a n(z z0)− + T n, Using the expression for the sum of a geometric series, − − − −2πi C2 w z − · · · − n n I − 2 n 1 1 q 1 2 n 1 q 1+q+q + +q − = − = 1+q+q + +q − + . with appropriate coefficients and the remainder term · · · 1 q ⇒ 1 q · · · 1 q n − − − 1 w z0 f (w) z z0 w z0 T n = − dw. We use q = − for integral over C1 and q = − over C2. w z0 z z0 − 2πi C z z0 z w − − I 2  −  − Applied Mathematical Methods Singularities of Complex Functions 553, Applied Mathematical Methods Singularities of Complex Functions 554, Series Representations of Complex Functions Series Representations of Complex Functions Series Representations of Complex FunctionsZeros and Singularities Zeros and Singularities Zeros and Singularities Residues Residues Evaluation of Real Integrals Evaluation of Real Integrals Convergence of Laurent’s series Zeros of an analytic function: points where the function vanishes

n 1 If, at a point z0, − k f (z) = ak (z z0) + Tn + T n, a function f (z) vanishes along with first m 1 of its − − (m) − k= n derivatives, but f (z0) = 0; X− 6 n 1 z z0 f (w) then z0 is a zero of f (z) of order m, giving the Taylor’s series as where Tn = − dw 2πi C w z0 w z I 1  −  − m n f (z) = (z z0) g(z). 1 w z0 f (w) − and T n = − dw. − 2πi z z z w IC2  0  An isolated zero has a neighbourhood containing no other zero. I f (w) is bounded − − For an analytic function, not identically zero, every point I z z0 w z0 w−z < 1 over C1 and z −z < 1 over C2 − 0 − 0 has a neighbourhood free of zeros of the function, except

Use M -L inequalit y to show that possibly for that point itself. In particular, zeros of such

remainder terms Tn and T n approach zero as n . an analytic function are always isolated. − → ∞ Remark: For actually developing Taylor’s or Laurent’s series of a Implication: If f (z) has a zero in every neighbourhood around function, algebraic manipulation of known facts are employed quite z0 then it cannot be analytic at z0, unless it is the zero function often, rather than evaluating so many contour integrals! [i.e. f (z) = 0 everywhere].

Applied Mathematical Methods Singularities of Complex Functions 555, Applied Mathematical Methods Singularities of Complex Functions 556, Series Representations of Complex Functions Series Representations of Complex Functions Zeros and Singularities Zeros and Singularities Zeros and Singularities Zeros and Singularities Residues Residues Entire function: A function which is analyticEvaluationeverywhereof Real Integrals Zeros and poles: complementary to each otherEvaluation of Real Integrals n z Examples: z (for positive integer n), e , sin z etc. I Poles are necessarily isolated singularities. I 1 The Taylor’s series of an entire function has an infinite A zero of f (z) of order m is a pole of f (z) of the same order radius of convergence. and vice versa. I If f (z) has a zero of order m at z where g(z) has a pole of Singularities: points where a function ceases to be analytic 0 the same order, then f (z)g(z) is either analytic at z0 or has a Removable singularity: If f (z) is not defined at z0, but has a limit. removable singularity there. ez 1 Example: f (z) = z− at z = 0. I Argument theorem: Pole: If f (z) has a Laurent’s series around z0, with a finite If f (z) is analytic inside and on a simple closed number of terms with negative powers. If an = 0 for curve C except for a finite number of poles inside n < m, but a m = 0, then z0 is a pole of order m, − −m 6 and f (z) = 0 on C, then limz z (z z0) f (z) being a non-zero finite number. 6 → 0 − A simple pole: a pole of order one. 1 f (z) 0 dz = N P, Essential singularity: A singularity which is neither a removable 2πi f (z) − IC singularity nor a pole. If the function has a Laurent’s where N and P are total numbers of zeros and poles series, then it has infinite terms with negative inside C respectively, counting multiplicities (orders). powers. Example: f (z) = e1/z at z = 0. Applied Mathematical Methods Singularities of Complex Functions 557, Applied Mathematical Methods Singularities of Complex Functions 558, Series Representations of Complex Functions Series Representations of Complex Functions Residues Zeros and Singularities Evaluation of Real Integrals Zeros and Singularities Residues Residues Evaluation of Real Integrals Evaluation of Real Integrals Term by term integration of Laurent’s series: C f (z)dz = 2πia 1 General strategy 1 − Residue: Resf (z) = a 1 = f (z)dz I z0 − 2πi C H Identify the required integral as a contour integral of a If f (z) has a pole (of order m) atH z0, then complex function, or a part thereof. I If the domain of integration is infinite, then extend the m ∞ m+n (z z ) f (z) = an(z z ) − 0 − 0 contour infinitely, without enclosing new singularities. n= m X− Example: is analytic at z0, and 2π m 1 I = φ(cos θ, sin θ)dθ d − m ∞ (m + n)! n+1 [(z z ) f (z)] = an(z z ) 0 dzm 1 − 0 (n + 1)! − 0 Z − n= 1 With z = eiθ and dz = izdθ, X− 1 d m 1 1 1 1 1 dz Res − m f (z) = a 1 = lim m 1 [(z z0) f (z)]. I = φ z + , z = f (z)dz, ⇒ z0 − (m 1)! z z0 dz − 2 z 2i − z iz − → − IC      IC Residue theorem: If f (z) is analytic inside and on simple closed where C is the unit circle centred at the origin. curve C, with singularities at z , z , z , , zk inside C; then 1 2 3 · · · Denoting poles falling inside the unit circle C as pj , k Res f (z)dz = 2πi Resf (z). I = 2πi p f (z). zi j C i j I X=1 X

Applied Mathematical Methods Singularities of Complex Functions 559, Applied Mathematical Methods Singularities of Complex Functions 560, Series Representations of Complex Functions Series Representations of Complex Functions Evaluation of Real Integrals Zeros and Singularities Evaluation of Real Integrals Zeros and Singularities Residues Residues Example: For real rational function f (x), Evaluation of Real Integrals Example: Fourier integral coefficients Evaluation of Real Integrals

∞ ∞ I = ∞ f (x)dx, A(s) = f (x) cos sx dx and B(s) = f (x) sin sx dx Z−∞ Z−∞ Z−∞ denominator of f (x) being of degree two higher than numerator. Consider ∞ isx Consider contour C enclosing semi-circular region z R, y 0, I = A(s) + iB(s) = f (x)e dx. | | ≤ ≥ Z−∞ large enough to enclose all singularities above the x-axis. Similar to the previous case,

R y R f (z)eisz dz = f (x)eisx dx + f (z)eisz dz. f (z)dz = f (x)dx + f (z)dz iR C R S C R S I Z− Z I Z− Z p R S isz isx sy sy M As e = e e− = e− 1 for y 0, we have p | | | | | | | | ≤ ≥ For finite M, f (z) < R2 on C p | | −R O R x M πM f (z)eisz dz < πR = , M πM R2 R f (z)dz < πR = . ZS 2 S R R Figure: The contour which yields, as R , Z → ∞ I = 2πi Res[f (z)eisz ]. ∞ pj I = f (x)dx = 2πi Resf (z) as R . j pj → ∞ X j Z−∞ X Applied Mathematical Methods Singularities of Complex Functions 561, Applied Mathematical Methods Variational Calculus* 562, Series Representations of Complex Functions Introduction Points to note Zeros and Singularities Outline Euler’s Equation Residues Direct Methods Evaluation of Real Integrals

I Taylor’s series and Laurent’s series I Zeros and poles of analytic functions Variational Calculus* I Residue theorem Introduction I Evaluation of real integrals through contour integration of Euler’s Equation suitable complex functions Direct Methods

Necessary Exercises: 1,2,3,5,8,9,10

Applied Mathematical Methods Variational Calculus* 563, Applied Mathematical Methods Variational Calculus* 564, Introduction Introduction Introduction Euler’s Equation Introduction Euler’s Equation Direct Methods Direct Methods Functionals and their extremization Consider a particle moving on a smooth surface z = ψ(q1, q2). Suppose that a candidate curve is represented as a sequence of T With position r = [q1(t) q2(t) ψ(q1(t), q2(t))] on the surface points qj = q(tj ) at time instants and δr = [δq δq ( ψ)T δq]T in the tangent plane, length of the 1 2 ∇ path from q = q(t ) to q = q(t ) is ti = t0 < t1 < t2 < t3 < < tN 1 < tN = tf . i i f f · · · −

tf tf 1/2 2 2 T 2 Geodesic problem: a multivariate optimization problem with the l = δr = r˙ dt = q˙ 1 + q˙ 2 + ( ψ q˙ ) dt. k k t k k t ∇ 2(N 1) variables in qj , 1 j N 1 . Z Z i Z i h i − { ≤ ≤ − } With N , we obtain the actual function. For shortest path or geodesic, minimize the path length l. → ∞ Question: What are the variables of the problem? First order necessary condition: Functional is stationary with respect to arbitrary small variations in qj . { } Answer: The entire curve or function q(t). [Equivalent to vanishing of the gradient] Variational problem: This gives equations for the stationary points. Optimization of a function of functions, i.e. a functional. Here, these equations are differential equations! Applied Mathematical Methods Variational Calculus* 565, Applied Mathematical Methods Variational Calculus* 566, Introduction Introduction Introduction Euler’s Equation Euler’s Equation Euler’s Equation Direct Methods Direct Methods Examples of variational problems Find out a function y(x), that will make the functional b Geodesic path: Minimize l = r (t) dt x a k 0 k 2 Minimal surface of revolution: Minimize I [y(x)] = f [x, y(x), y 0(x)]dx R b 2 x1 S = 2πyds = 2π a y 1 + y 0 dx Z The brachistochrone problem: To find the curve along which the stationary, with boundary conditions y(x1) = y1 and y(x2) = y2. R R p descent is fastest. Consider variation δy(x) with δy(x1) = δy(x2) = 0 and consistent b 2 ds 1+y 0 variation δy 0(x). Minimize T = v = a 2gy dx Fermat’s principle: Light takes the fastestq path. x2 ∂f ∂f R R2 2 2 u2 √x +y +z δI = δy + δy dx Minimize T = 0 0 0 du 0 u1 c(x,y,z) x1 ∂y ∂y 0 Isoperimetric problem: Largest area in the plane enclosed by a Z   R Integration of the second term by parts: closed curve of given perimeter. By extension, x x2 ∂f x2 ∂f d ∂f 2 x2 d ∂f extremize a functional under one or more equality δy 0dx = (δy)dx = δy δy dx x ∂y x ∂y dx ∂y − x dx ∂y constraints. Z 1 0 Z 1 0  0 x1 Z 1 0 Hamilton’s principle of least action: Evolution of a dynamic With δy(x1) = δy(x2) = 0, the first term vanishes identically, and system through the minimization of the action x2 ∂f d ∂f t2 t2 δI = δy dx. s = Ldt = (K P)dt x ∂y − dx ∂y − Z 1  0  Zt1 Zt1

Applied Mathematical Methods Variational Calculus* 567, Applied Mathematical Methods Variational Calculus* 568, Introduction Introduction Euler’s Equation Euler’s Equation Euler’s Equation Euler’s Equation Direct Methods Direct Methods For δI to vanish for arbitrary δy(x), Functionals of a vector function d ∂f ∂f = 0. t dx ∂y 0 − ∂y 2 I [r(t)] = f (t, r, r˙)dt Functions involving higher order derivatives Zt1 ∂f ∂f x2 In terms of partial gradients ∂r and ∂r˙ , (n) I [y(x)] = f x, y, y 0, y 00, , y dx T T · · · t2 Zx1 ∂f ∂f  (n 1) δI = δr + δr˙ dt with prescribed boundary values for y, y 0, y 00, , y − t1 " ∂r ∂r˙ # · · · Z     t2 t2 T T t2 T x2 ∂f ∂f d ∂f ∂f ∂f ∂f ∂f (n) = δrdt + δr δrdt δI = δy + δy 0 + δy 00 + + δy dx (n) t1 ∂r " ∂r˙ # − t1 dt ∂r˙ x1 ∂y ∂y 0 ∂y 00 · · · ∂y Z     t1 Z   Z   T Working rule: Starting from the last term, integrate one term at t2 ∂f d ∂f = δrdt. a time by parts, using consistency of variations and BC’s. ∂r − dt ∂r˙ t1   Euler’s equation: Z Euler’s equation: a system of second order ODE’s ∂f d ∂f d 2 ∂f d n ∂f + + ( 1)n = 0, d ∂f ∂f d ∂f ∂f 2 n (n) ∂y − dx ∂y 0 dx ∂y 00 − · · · − dx ∂y = 0 or = 0 for each i. dt ∂r˙ − ∂r dt ∂r˙i − ∂ri an ODE of order 2n, in general. Applied Mathematical Methods Variational Calculus* 569, Applied Mathematical Methods Variational Calculus* 570, Introduction Introduction Euler’s Equation Euler’s Equation Direct Methods Euler’s Equation Direct Methods Direct Methods Functionals of functions of several variables Finite difference method With given boundary values y(a) and y(b),

I [u(x, y)] = f (x, y, u, ux , uy )dx dy b D I [y(x)] = f [x, y(x), y 0(x)]dx Z Z a Euler’s equation: ∂ ∂f + ∂ ∂f ∂f = 0 Z ∂x ∂ux ∂y ∂uy − ∂u I Represent y(x) by its values over x = a + ih with Moving boundaries i i = 0, 1, 2, , N, where b a = Nh. Revision of the basic case: allowing non-zero δy(x1), δy(x2) · · · − I Approximate the functional by At an end-point, ∂f δy has to vanish for arbitrary δy(x). ∂y 0 N ∂f vanishes at the boundary. ∂y 0 I [y(x)] φ(y1, y2, y3, , yN 1) = f (x¯i , y¯i , y¯i0)h, ≈ · · · − i=1 Euler boundary condition or natural boundary condition X xi +xi 1 yi +yi 1 yi yi 1 where x¯ = − , y¯ = − and y¯ = − − . Equality constraints and isoperimetric problems i 2 i 2 i0 h I x2 x2 Minimize φ(y1, y2, y3, , yN 1) with respect to yi ; Minimize I = f (x, y, y 0)dx subject to J = g(x, y, y 0)dx = J0. · · · − x1 x1 for example, by solving ∂φ = 0 for all i. In another level of generalization, constraint φ(x, y, y ) = 0. ∂yi R 0 R ∂φ Operate with f ∗(x, y, y 0, λ) = f (x, y, y 0) + λ(x)g(x, y, y 0). Exercise: Show that = 0 is equivalent to Euler’s equation. ∂yi

Applied Mathematical Methods Variational Calculus* 571, Applied Mathematical Methods Variational Calculus* 572, Introduction Introduction Direct Methods Euler’s Equation Direct Methods Euler’s Equation Direct Methods Direct Methods The inverse problem: From Rayleigh-Ritz method N N In terms of a set of basis functions, express the solution as b I [y(x)] φ(α) = f x, αi wi (x), αi w 0(x) dx, ≈ i a i i ! N Z X=1 X=1 y(x) = αi wi (x). ∂φ b ∂f N N ∂f N N = 2 0x, αi wi , αi wi01 wi (x) + 0x, αi wi , αi wi01 wi0(x)3 dx. ∂α Z ∂y ∂y i=1 i a Xi=1 Xi=1 0 Xi=1 Xi=1 X 4 @ A @ A 5 Integrating the second term by parts and using w (a) = w (b) = 0, Represent functional I [y(x)] as a multivariate function φ(α). i i N ∂φ b Optimize φ(α) to determine αi ’s. = αi wi wi (x)dx, ∂αi R Za " i=1 # Note: As N , the numerical solution approaches exactitude. X → ∞ where [y] ∂f d ∂f = 0 is the Euler’s equation of the For a particular tolerance, one can truncate appropriately. R ≡ ∂y − dx ∂y 0 variational problem. Observation: With these direct methods, no need to reduce the Def.: [z(x)]: residual of the differential equation [y] = 0 R R variational (optimization) problem to Euler’s equation! operated over the function z(x) Question: Is it possible to reformulate a BVP as a variational Residual of the Euler’s equation of a variational problem problem and then use a direct method? operated upon the solution obtained by Rayleigh-Ritz method is orthogonal to basis functions wi (x). Applied Mathematical Methods Variational Calculus* 573, Applied Mathematical Methods Variational Calculus* 574, Introduction Introduction Direct Methods Euler’s Equation Direct Methods Euler’s Equation Direct Methods Direct Methods Galerkin method Question: What if we cannot find a ‘corresponding’ variational Finite element methods problem for the differential equation? I discretization of the domain into elements of simple geometry Answer: Work with the residual directly and demand I basis functions of low order polynomials with local scope b I design of basis functions so as to achieve enough order of [z(x)]wi (x)dx = 0. R Za continuity or smoothness across element boundaries Freedom to choose two different families of functions as basis I piecewise continuous/smooth basis functions for entire functions ψj (x) and trial functions wi (x): domain, with a built-in sparse structure

b I some weighted residual method to frame the algebraic αj ψj (x) wi (x)dx = 0 equations R   a j Z X I solution gives coefficients which are actually the nodal values A singular case of the Galerkin method: delta functions, at discrete points, as trial functions Suitability of finite element analysis in software environments I effectiveness and efficiency Satisfaction of the differential equation exactly at the chosen points, known as collocation points: I neatness and modularity Collocation method

Applied Mathematical Methods Variational Calculus* 575, Applied Mathematical Methods Epilogue 576, Introduction Points to note Euler’s Equation Outline Direct Methods

I Optimization with respect to a function I Concept of a functional I Euler’s equation Epilogue I Rayleigh-Ritz and Galerkin methods I Optimization and equation-solving in the infinite-dimensional function space: practical methods and connections

Necessary Exercises: 1,2,4,5 Applied Mathematical Methods Epilogue 577, Applied Mathematical Methods Epilogue 578, Epilogue Epilogue

Source for further information: http://home.iitk.ac.in/˜ dasgupta/MathBook Some specialized courses in immediate continuation Destination for feedback: I Linear Algebra and Matrix Theory [email protected] I Approximation Theory I Variational Calculus and Optimal Control Some general courses in immediate continuation I Advanced Mathematical Physics I Advanced Mathematical Methods I Geometric Modelling I Scientific Computing I Computational Geometry I Advanced Numerical Analysis I Computer Graphics I Optimization I Signal Processing I Advanced Differential Equations I Image Processing I Partial Differential Equations I Finite Element Methods

Applied Mathematical Methods Selected References 579, Applied Mathematical Methods Selected References 580, Outline Selected References I

F. S. Acton. Numerical Methods that usually Work. The Mathematical Association of America (1990). C. M. Bender and S. A. Orszag. Selected References Advanced Mathematical Methods for Scientists and Engineers.

Springer-Verlag (1999). G. Birkhoff and G.-C. Rota. Ordinary Differential Equations. John Wiley and Sons (1989). G. H. Golub and C. F. Van Loan. Matrix Computations. The John Hopkins University Press (1983). Applied Mathematical Methods Selected References 581, Applied Mathematical Methods Selected References 582, Selected References II Selected References III

M. T. Heath. Scientific Computing. W. H. Press, S. A. Teukolsky, W. T. Vellerling and B. P. Tata McGraw-Hill Co. Ltd (2000). Flannery. Numerical Recipes. E. Kreyszig. Cambridge University Press (1998). Advanced Engineering Mathematics. John Wiley and Sons (2002). G. F. Simmons. Differential Equations with Applications and Historical Notes. E. V. Krishnamurthy and S. K. Sen. Tata McGraw-Hill Co. Ltd (1991). Numerical Algorithms. Affiliated East-West Press Pvt Ltd (1986). J. Stoer and R. Bulirsch. Introduction to Numerical Analysis. D. G. Luenberger. Springer-Verlag (1993). Linear and Nonlinear Programming. Addison-Wesley (1984). C. R. Wylie and L. C. Barrett. Advanced Engineering Mathematics. P. V. O’Neil. Tata McGraw-Hill Co. Ltd (2003). Advanced Engineering Mathematics. Thomson Books (2004).