A Appendix: Linear Algebra and Functional Analysis In this appendix, we have collected some results of functional analysis and matrix algebra. Since the computational aspects are in the foreground in this book, we give separate proofs for the linear algebraic and functional analyses, although finite-dimensional spaces are special cases of Hilbert spaces. A.1 Linear Algebra A general reference concerning the results in this section is [47]. The following theorem gives the singular value decomposition of an arbitrary real matrix. Theorem A.1. Every matrix A ∈ Rm×n allows a decomposition A = UΛV T, where U ∈ Rm×m and V ∈ Rn×n are orthogonal matrices and Λ ∈ Rm×n is diagonal with nonnegative diagonal elements λj such that λ1 ≥ λ2 ≥ ··· ≥ λmin(m,n) ≥ 0. Proof: Before going to the proof, we recall that a diagonal matrix is of the form ⎡ ⎤ λ1 0 ... 00... 0 ⎢ ⎥ ⎢ . ⎥ ⎢ 0 λ2 . ⎥ Λ = = [diag(λ ,...,λm), 0] , ⎢ . ⎥ 1 ⎣ . .. ⎦ 0 ... λm 0 ... 0 if m ≤ n, and 0 denotes a zero matrix of size m×(n−m). Similarly, if m>n, Λ is of the form 312 A Appendix: Linear Algebra and Functional Analysis ⎡ ⎤ λ1 0 ... 0 ⎢ ⎥ ⎢ . ⎥ ⎢ 0 λ2 . ⎥ ⎢ ⎥ ⎢ . .. ⎥ ⎢ . ⎥ diag(λ ,...,λn) Λ = ⎢ ⎥ = 1 , ⎢ λn ⎥ ⎢ ⎥ 0 ⎢ 0 ... 0 ⎥ ⎢ . ⎥ ⎣ . ⎦ 0 ... 0 where 0 is a zero matrix of the size (m − n) × n. Briefly, we write Λ = diag(λ1,...,λmin(m,n)). n Let A = λ1, and we assume that λ1 =0.Let x ∈ R be a unit vector m with Ax = A ,andy =(1/λ1)Ax ∈ R , i.e., y is also a unit vector. We n m pick vectors v2,...,vn ∈ R and u2,...,um ∈ R such that {x, v2,...,vn} n is an orthonormal basis in R and {y,u2,...,um} is an orthonormal basis in Rm, respectively. Writing n×n m×m V1 =[x, v2,...,vn] ∈ R ,U2 =[y,u2,...,um] ∈ R , we see that these matrices are orthogonal and it holds that ⎡ ⎤ yT ⎢ uT ⎥ T T ⎢ 2 ⎥ λ1 w A1 = U1 AV1 = ⎢ . ⎥ [λ1y,Av2,...,Avn]= , ⎣ . ⎦ 0 B T um where w ∈ R(n−1), B ∈ R(m−1)×(n−1).Since λ λ2 + w 2 A 1 = 1 , 1 w Bw it follows that λ A 1 ≥ λ2 + w 2, 1 w 1 or ≥ 2 2 A1 λ1 + w . On the other hand, an orthogonal transformation preserves the matrix norm, so ≥ 2 2 A = A1 λ1 + w . Therefore w =0. Now we continue inductively. Let λ2 = B .Wehave λ2 ≤ A1 = A = λ1. A.1 Linear Algebra 313 If λ2 = 0, i.e., B = 0, we are finished. Assume that λ2 > 0. Proceeding as (m−1)×(m−1) (n−1)×(n−1) above, we find orthogonal matrices U˜2 ∈ R and V˜2 ∈ R such that λ 0 U˜ TBV˜ = 2 2 2 0 C for some C ∈ R(m−2)×(n−2). By defining 10 10 U2 = ,V2 = , 0 U˜2 0 V˜2 we get orthogonal matrices that have the property ⎡ ⎤ λ1 00 T T ⎣ ⎦ U2 U1 AV1V2 = 0 λ2 0 . 00C Inductively, performing the same partial diagonalization by orthogonal trans- formations, we finally get the claim. 2 Theorem A.2. Let A = UΛV T ∈ Rm×n be the singular value decomposition as above, and assume that λ1 ≥ λ2 ≥ ···λp >λp+1 = ··· = λmin(m,n) =0. By denoting U =[u1,...,um],V=[v1,...,vn], we have T ⊥ Ker(A)=sp{vp+1,...,vn} =Ran(A ) , T ⊥ Ker(A )=sp{up+1,...,um} =Ran(A) . Proof: By the orthogonality of U,wehaveAx =0ifandonlyif ⎡ ⎤ T λ1v1 x ⎢ ⎥ ⎢ . ⎥ ⎢ . ⎥ ⎢ T ⎥ T ⎢ λpvp x ⎥ ΛV x = ⎢ ⎥ =0, ⎢ 0 ⎥ ⎢ . ⎥ ⎣ . ⎦ 0 from where the claim can be seen immediately. Furthermore, we have AT = VDTU T, so a similar result by interchanging U and V holds for the transpose. On the other hand, we see that x ∈ Ker(A)isequivalentto (Ax)Ty = xTATy =0 for all y ∈ Rm, i.e., x is perpendicular to the range of AT. 2 In particular, if we divide U and V as 314 A Appendix: Linear Algebra and Functional Analysis U =[[u1,...,up], [up+1,...,um]] = [U1,U2], V =[[v1,...,vp], [vp+1,...,vn]] = [V1,V2], the orthogonal projectors P : Rn → Ker(A)andP˜ : Rn → Ker(AT) assume thesimpleforms ˜ T P = V2V2, P = U2U2 . A.2 Functional Analysis We confine the discussion to the Hilbert spaces. Definition A.3. Let A : H1 → H2 be a continuous linear operator between Hilbert spaces H1 and H2. The operator is said to be compact if the sets A(U) ⊂ H2 are compact for every bounded set U ⊂ H1. In the following simple lemmas, we denote by X⊥ the orthocomplement of the subspace X ⊂ H. Lemma A.4. Let X ⊂ H be a closed subspace of the Hilbert space H.Then (X⊥)⊥ = X. Proof: Let x ∈ X. Then for every y ∈ X⊥, x, y = 0, i.e., x ∈ (X⊥)⊥. To prove the converse inclusion, letx ∈ (X⊥)⊥.DenotebyP : H → X the orthogonal projection. We have x − Px ∈ X⊥. On the other hand, Px ∈ X ⊂ (X⊥)⊥,sox − Px ∈ (X⊥)⊥. Therefore, x − Px is orthogonal with itself and thus x = Px ∈ X. The proof is complete. 2 ⊥ Lemma A.5. Let X ⊂ H be a subspace. Then X = X⊥. The proof is rather obvious and therefore omitted. Lemma A.6. Let A : H1 → H2 be a bounded linear operator. Then Ker(A∗)=Ran(A)⊥, Ker(A)=Ran(A∗)⊥. Proof: We have x ∈ Ker(A∗) if and only if 0= A∗x, y = x, Ay for all y ∈ H1, proving the claim. 2 Based on these three lemmas, the proof of part (i) of Proposition 2.1 follows. Indeed, Ker(A) ⊂ H1 is closed and we have ⊥ ∗ ⊥ ⊥ H1 =Ker(A) ⊕ Ker(A) =Ker(A) ⊕ (Ran(A ) ) by Lemma A.6, and by Lemmas A.4 and A.5, A.2 Functional Analysis 315 ⊥ (Ran(A∗)⊥)⊥ =(Ran(A∗) )⊥ = Ran(A∗). Part (ii) of Proposition 2.1 is based on the fact that the spectrum of a compact operator is discrete and accumulates possibly only at the origin. If ∗ A : H1 → H2 is compact, then A A : H1 → H1 is a compact and positive 2 operator. Denote the positive eigenvalues by λn, and let the corresponding ∗ normalized eigenvectors be vn ∈ H1.SinceA A is self-adjoint, the eigenvectors can be chosen mutually orthogonal. Defining 1 un = Avn, λn we observe that 1 ∗ un,uk = vn,A Avk = δnk, λnλk hence the vectors un are also orthonormal. The triplet (vn,un,λn)isthe singular system of the operator A. Next, we prove the fixed point theorem 2.8 used in Section 2.4.1. Proof of Proposition 2.8: Let T : H → H be a mapping, S ⊂ H an invariant closed set such that T (S) ⊂ S and let T be a contraction in S, T (x) − T (y) <κ x − y for all x, y ∈ S, where κ<1. For any j>1, we have the estimate xj+1 − xj = T (xj) − T (xj−1) <κ xj − xj−1 , and inductively j−1 xj+1 − xj <κ x2 − x1 . It follows that for any n, k ∈ N,wehave k k n+j−2 xn+k − xn ≤ xn+j − xn+j−1 < κ x2 − x1 j=1 j=1 κn−1 ≤ x − x 1 − κ 2 1 by the geometric series sum formula. Therefore, (xj ) is a Cauchy sequence, and thus is convergent and the limit is in S since S is closed. 2. We finish this section by giving without proof a version of the Riesz rep- resentation theorem. Theorem A.7. Let B : H × H → R be a bilinear quadratic form satisfying |B(x, y)|≤C x y for all x, y ∈ H, B(x, x) ≥ c x 2 for all x ∈ H for some constants 0 <c≤ C<∞. Then there is a unique linear bounded operator T : H → H such that B(x, y)= Tx,y for all y ∈ H. 316 A Appendix: Linear Algebra and Functional Analysis The variant of this for Hilbert spaces with complex coefficients is known as the Lax–Milgram lemma. See [142] for a proof. A.3 Sobolev Spaces In this section, we review the central definitions and results concerning Sobolev spaces. For square integrable functions f : Rn → R, define the L2-norm over Rn as 1/2 f = |f(x)|2dx < ∞. (A.1) Rn The linear space of square integrable functions is denoted by L2 = L2(Rn). The norm (A.1) defines an inner product, f,q = f(x)g(x)dx, Rn that defines a Hilbert space structure in L2. The Fourier transform for integrable functions of Rn is defined as Ff (ξ)=f(ξ)= e−iξ,xf(x)dx. Rn The Fourier transform can be extended to L2, and it defines a bijective map- ping F : L2 → L2, see, e.g., [108]. Its inverse transformation is then n 1 F −1f (x)=f(x)= eiξ,xf(ξ)dξ. 2π Rn In L2, Fourier transform is an isometry up to a constant, and it satisfies the identities f =(2π)n/2 f (Parseval) , f, g =(2π)n f,g (Plancherel) . Defining the convolution of functions as f ∗ g(x)= f(x − y)g(y)dy, provided that the integral converges, we have F(f ∗ g)(ξ)=f(ξ)g(ξ). n Let α be a multiindex, α =(α1,α2,...,αn) ∈ N . Define the length of the multiindex, |α|,as|α| = α1 + ···+ αn.Wedefine A.3 Sobolev Spaces 317 ∂α1 ∂α2 ∂αn ∂|α| Dαf(x)=(−i)|α| ... f(x)=(−i)|α| f(x). α1 α2 αn α1 ··· αn ∂x1 ∂x2 ∂xn ∂x1 ∂xn Similarly, for x ∈ Rn,wedefine α α1 α2 ··· αn x = x1 x2 xn .