Numerical Linear Algebra Chap. 1: Basic Concepts from Linear Algebra Heinrich Voss [email protected] Hamburg University of Technology Institute for Numerical Simulation TUHH Heinrich Voss Chapter 1 2005 1 / 60 Basic Ideas from Linear Algebra Vectors The vector space Rn is defined by n T R := {(x1,..., xn) : xj ∈ R, j = 1,..., n}, x1 y1 x1 + y1 x1 αx1 x2 y2 x2 + y2 x2 αx2 + = , α = . . . . . . . . . . xn yn xn + yn xn αxn and Cn correspondingly. A subset X ⊂ Rn is a subspace of Rn if it is closed with respect to addition and multiplication by scalars, i.e. x, y ∈ X ⇒ x + y ∈ X, n α ∈ R, x ∈ R ⇒ αx ∈ X. TUHH Heinrich Voss Chapter 1 2005 2 / 60 Basic Ideas from Linear Algebra Subspaces A set of vectors {a1,..., am} ⊂ Cn is linearly independent if m X j αj a = 0 ⇒ αj = 0, j = 1,..., m. j=1 Otherwise, a nontrivial combination of the aj is zero, and {a1,..., am} is said to be linearly dependent. Given vectors a1,..., am the set of all linear combinations of these vectors is a subspace referred to as the span of a1,..., am: m 1 m X j span{a ,..., a } = αj a : αj ∈ C . j=1 TUHH Heinrich Voss Chapter 1 2005 3 / 60 Basic Ideas from Linear Algebra Subspaces ct. If {a1,..., am} is linearly independent and b ∈ span{a1,..., am}, then b has a unique representation as linear combination of the aj . n If S1,..., Sk are subspace of C then their sum k X j j S := S1 + ··· + Sk := a : a ∈ Sj , j = 1,..., k j=1 is also a subspace of Cn. S is said to be the direct sum if each x ∈ S has a 1 k j unique representattion x = a + ··· + a with a ∈ Sj . In this case we write S = S1 ⊕ · · · ⊕ Sk . The intersection of the subspaces S1,..., Sk is also a subspace, S = S1 ∩ · · · ∩ Sk . TUHH Heinrich Voss Chapter 1 2005 4 / 60 Basic Ideas from Linear Algebra Dimension The subset {ai1 ,..., aik } is a maximal linearly independent subset of {a1,..., am} if it is linearly independent and is not contained properly in any linearly independent subset of {a1,..., am}. If {ai1 ,..., aik } is a maximal linearly independent subset then span{ai1 ,..., aik } = span{a1,..., am}. If S ⊂ Cn is a subspace it is always possible to find a maximal linearly independent subset {a1,..., ak }. Then S = span{a1,..., am}, and {a1,..., am} is called a basis of S. All bases for a subspace S have the same number of elements. This number is the dimension of S, and it is denoted by dim(S). TUHH Heinrich Voss Chapter 1 2005 5 / 60 Basic Ideas from Linear Algebra Linear map A map A : Cn → Cm is called linear, if n n A(x+y) = Ax+Ay for every x, y ∈ C and A(λx) = λAx for every x ∈ C , λ ∈ C. n For j = 1,..., n let ej ∈ C be the j-th canonical unit vector having a 1 in its n j-th component and zeros elsewhere. Then for x = (xj )j=1,...,n ∈ C n n n n X X X X Ax = A xj ej = A(xj ej ) = xj Aej =: xj aj j=1 j=1 j=1 j=1 Hence, the images aj := Aej of the canonical basis vectors characterize the linear map A. We therefore identify A with the m × n matrix a11 a12 ... a1n a21 a22 ... a2n A = . . .. . am1 a2m ... amn TUHH Heinrich Voss Chapter 1 2005 6 / 60 Basic Ideas from Linear Algebra Matrix-vector product m×n n For A = (ajk ) ∈ C and x = (xk ) ∈ C we have n X m Ax = ajk xk =: b ∈ C . j=1,...,m k=1 The vector b is called matrix-vector product of A and x. For every x ∈ Cn the matrix-vector product b = Ax is a linear combination of the columns aj of the matrix A. TUHH Heinrich Voss Chapter 1 2005 7 / 60 Basic Ideas from Linear Algebra Matrix-matrix product Let A : Cn → Cm and B : Cm → Cp. Then the composition n p B ◦ A : C → C , (B ◦ A)x = B(Ax) is linear as well, and n m n n m X X X X X BAx = B(Ax) = B ajk xk = bij ajk xk = bij ajk xk . k=1 j=1 k=1 k=1 j=1 Hence, the composit map of B and A is represented by the matrix-matrix product C := BA ∈ Cp×n with elements m X cik = bij ajk , i = 1,..., p, k = 1,..., n. j=1 Notice that the matrix-matrix product of B and A is only defined if the number of columns of B equals the number of rows of A. TUHH Heinrich Voss Chapter 1 2005 8 / 60 Basic Ideas from Linear Algebra Range of a matrix The range of a matrix A, written range(A), is the set of vectors that can be expressed as Ax for some x. The formula b = Ax leads naturally to the following characterization of range(A). Theorem 1 range(A) is the space spanned by the columns of A. Proof Pn Ax = j=1 aj xj is a linear combination of the columns aj of A. Conversely, any vector y in the space spanned by the columns of A can be Pn written as a linear combination of the columns, y = j=1 aj xj . Forming a vector x out of the coefficients xj , we have y = Ax, and thus y is in the range of A. In view of Theorem 1, the range of a matrix A is also called the column space of A. TUHH Heinrich Voss Chapter 1 2005 9 / 60 Basic Ideas from Linear Algebra Nullspace of A The nullspace of A ∈ Cm×n, written null(A), is the set of vectors x that satisfy Ax = 0, where 0 is the 0-vector in Cn. The entries of each vector x ∈ null(A) give the coefficients of an expansion of zero as a linear combination of columns of A: 0 = xl al + x2a2 + ··· + xnan. The column rank of a matrix is the dimension of its column space. Similarly, the row rank of a matrix is the dimension of the space spanned by its rows. Row rank always equals column rank (among other proofs, this is a corollary of the singular value decomposition, discussed later), so we refer to this number simply as the rank of a matrix. TUHH Heinrich Voss Chapter 1 2005 10 / 60 Basic Ideas from Linear Algebra Full rank An m × n matrix of full rank is one that has the maximal possible rank (the lesser of m and n). This means that a matrix of full rank with m ≥ n must have n linearly independent columns. Such a matrix can also be characterized by the property that the map it defines is one-to-one. Theorem 2 A matrix A ∈ Cm×n with m ≥ n has full rank if and only if it maps no two distinct vectors to the same vector. Proof If A is of full rank, its columns are linearly independent, so they form a basis for range(A). This means that every b ∈ range(A) has a unique linear expansion in terms of the columns of A, and therefore, every b ∈ range(A) has a unique linear expansion in terms of the columns of A, and thus, every b ∈ range(A) has a unique x such that b = Ax. Conversely, if A is not of full rank, its columns aj are dependent, and there is a Pn nontrivial linear combination such that j=1 cj aj = 0. The nonzero vector c formed from the coefficients cj satisfies Ac = 0. But then A maps distinct vectors to the same vector since, for any x it holds that Ax = A(x + c). TUHH Heinrich Voss Chapter 1 2005 11 / 60 Basic Ideas from Linear Algebra Inverse matrix A nonsingular or invertible matrix is a square matrix of full rank. Note that the m columns of a nonsingular m × m matrix A form a basis for the whole space Cm. Therefore, we can uniquely express any vector as a linear combination of them. In particular, the canonical unit vector ej , can be expanded: m X ej = zij aj . i=1 m×m Let Z ∈ C be the matrix with entries zij , and let zj denote the jth column of Z. Then it holds ej = Azj , and putting these vectos together AZ = (e1,..., em) =: I where I ist the m × m identity. Z is the inverse of A, and is denoted by Z =: A−1. TUHH Heinrich Voss Chapter 1 2005 12 / 60 Basic Ideas from Linear Algebra Gaussian elimination The simplest way to solve a linear system (by hand or on a computer) is Gaussian elimination. It transforms a linear system to an equivalent one with upper-triangular system matrix by applying simple linear transformations. Let A ∈ Cm×n be given. The idea is to transform A into an upper-triangular matrix by introducing zeros below the diagonal, first in column 1, then in column 2 , etc. This is done by subtracting suitable multiples of each row from the subsequent ones. This eleimination process is equivalent to multiplying A by a sequence of lower triangular matrices Lj on the left: Ln−1Ln−2 ··· L1A = U. TUHH Heinrich Voss Chapter 1 2005 13 / 60 Basic Ideas from Linear Algebra LU factorization −1 −1 −1 Setting L := L1 L2 ··· Ln−1 gives A = LU. Thus we obtain an LU factorization of A A = LU, where U is upper-triangular, and L is (as a product of lower-triangular matrices) lower-triangular.