Part I Linear Algebra 4 Chapter 1 Vectors 1.1 Vectors Basics 1.1.1 Definition Vectors • A vector is a collection of n real numbers, x1; x2; :::; xn, as a single point in a n-dimensional space arranged in a column or a row, and can be thought of as point in space, or as providing a direction. • Each number xi is called a component or element of the vector. • The inner product define the length of a vector, as well as generalize the notion of angle between two vectors. • Via the inner product, we can view a vector as a linear function. We can also compute the projection of a vector onto a line defined by another. 5 • We usually write vectors in column format: 2 3 x1 6 x2 7 x = 6 7 6 . 7 4 . 5 xm • Geometry. A vector represents both a direction from the origin and a point in the multi-dimensional space Rn, where each component corresponds to coor- dinate of the point. • Transpose. If x is a column vector, xT (transpose) denotes the corresponding T row vector, and vice-versa. x = x1; :::xn . 1.1.2 Independence n • A set of m vectors x1; :::; xm in R is said to be independent if no vector in the set can be expressed as a linear combination of the others. This means that the condition m m X λ 2 R : λixi = 0 i=1 implies λ = 0. • If two vectors are linear independent, then they cannot be a scaled version of each other. • Example. The vectors x1 = [1; 2; 3] and x2 = [3; 6; 9] are not independent, since 3x1 − x2 = 0, x2 is a scaled version of x1. 1.1.3 Subspace, Span, Affine Sets Subspace and Span • An nonempty subspace, V, of Rn is a subset that is closed under addition and scalar multiplication. That is, for any scalars α; β x; y 2 V ! αx + βy 2 V 6 • A subspace always contains the zero element. • Geometrically, subspaces are "flat" (like a line or plane in 3D) and pass through the origin. n • A subspace S can always be represented as the span of a set of vectors xi in R , that is, a set of the form m X m S = span x1; :::; xm := λixi : λ 2 R i=0 • The set of all possible linear combinations of the vectors in S = fx(1); ··· ; x(m)g forms a subspace, which is called the subspace generated by S, or the span of S, denoted by span(S). Direct Sum • Given two subspaces X ; Y 2 Rn, the direct sum of X ; Y, denoted X ⊕ Y, is the set of vectors of the form x + y, with x 2 X ; y 2 Y. • X ⊕ Y is itself a subspace. Affine Sets • An affine set is a translation of a subspace. It is "flat" but does not necessarily pass through 0, as a subspace would. (Like a line or a plane that does not go through the origin.) • Affine set A can always be represented as the translation (a constant term) of the subspace spanned by some vectors: m X m A = x0 + λixi : λ 2 R = x0 + S i=1 where x0 is a given point and S is a given subspace. Affine is linear plus a constant term. 7 • Subspaces (or sometimes called linear subspaces) are just affine spaces contain- ing the origin. • Line. When S is the span of a single non-zero vector (1 dimension), the set A is called a line passing through the point x0. u is the direction of the line, t is the magnitude and x0 is a point through which it passes. A = x0 + tu : t 2 R 1.1.4 Basis and Dimension Basis • A basis of Rn is a set of n independent (irreducible) vectors. • If the vectors u1; ··· ; un form a basis, we can express any vector as a linear Pn combination of ui, x = i=1 λiui for appropriate numbers λ1; ··· ; λn. n • Standard basis. Standard basis (natural basis) in R consists of the vector ei, where the i-th element is 1 and the rest are 0. 213 203 203 3 e1 = 405 ; e2 = 415 ; e3 = 405 2 R 0 0 1 Basis of a Subspace • The basis of a given subspace S ⊆ Rn is any independent set of vectors whose span is S. • If vectors (u1; ··· ; ur) form a basis of S, we can express any vector in the Pr subspace S as a linear combination of (u1; ··· ; ur), x = i=1 λiui. • Dimension. The number of vectors in the basis is independent of the choice of the basis. We will always find a fixed minimum number of independent (ir- reducible) vectors for the subspace S. This minimum number is called the di- mension of S. 8 • Example. In R3, you need 2 independent vectors to describe a plane contain- ing the origin. (dimension of 2). The dimension of a line is 1, since a line is x0 + span(x1) for non-zero x1. Dimension of an Affine Subspace • The set L in R3, x1 − 13x2 + 4x3 = 2 3x2 − x3 = 9 is an affine subspace of dimension 1. The linear subspace can be obtained by setting the constant term to 0, x1 − 13x2 + 4x3 = 0 3x2 − x3 = 0 • Solve for x3 and we get x1 = x2; x3 = 3x2. The representation of linear subspace x 2 R3: 213 x = 415 t; for scalar t = 2 3 • The linear subspace is the span of u = (1; 1; 3) of dimension 1. We can find a particular solution x0 = (38; 0; −9) and the affine subspace L is thus the line x0 + span(u). 1.2 Orthogonality and Orthogonal Complements 1.2.1 Orthogonal Vectors • Orthogonal. Two vectors x; y in an inner product space X are orthogonal, denoted x ? y, if hx; yi = 0. 9 • Mutually orthogonal. Nonzeros vectors x(1); x(2); ··· ; x(d) are said to be mutually orthogonal if hx(i); x(j)i = 0 whenever i 6= j. In other words, each vector is orthogonal to all other vectors in the collection. • Mutually orthogonal vectors are linearly independent but linearly independent vectors are not necessary mutually orthogonal. 1.2.2 Orthogonal Complement • Orthogonal complement. A vector x 2 X is orthogonal to a subset S of an inner product space X if x ? s; 8s 2 S. The set of vectors in X that are orthogonal to S is called the orthogonal complement of S, denoted as S?. • Direct sum and orthogonal decomposition. If X is a subspace of an inner product space X , then any vector x 2 X can be written in a unique way of the sum of one element in S and one in the orthogonal complement S?. X = S ⊕ S?; for any subspace S ⊆ X x = y + z; x 2 X ; y 2 S; z 2 S? • Fundamental properties of inner product spaces. Let x; z be any two ele- ments of a inner product space X , let kxk = phx; xi, and let α be a scalar. Then: – jhx; zij ≤ kxkkzk, and equality holds iff x = αz, or z = 0 (Cauchy- Schwartz). – kx + zk2 + kx − zk2 = 2kxk2 + 2kzk2 (parallelogram law) – if x ? z, then kx + zk2 = kxk2 + kzk2 (Pythagoras theorem) – for any subspace S ⊆ X it holds that X = S ⊕ S? – for any subspace S ⊆ X it holds that dim X = dim S + dim S? 10 Figure 1.1: Left: Two dimension subspace X in R3 and its orthogonal complement S?. Right: Any vector can be written as the sum of an element x in a subspace S and one y in its orthogonal complement S? 1.3 Inner Product, Norms and Angles 1.3.1 Inner Product • The Inner product. The inner product (scalar product, dot product) on a (real) vector space X is a real-valued function which maps any pair of elements x; y 2 X into a scalar denoted by hx; yi. • Axioms The inner product satisfies the following axioms: for any x; y; z 2 X and scalar α – hx; yi ≥ 0 – hx; x = 0i if and only inf x = 0 – hx + y; zi = hx; zi + hy; zi – hαx; yi = αhx; yi – hx; yi = hy; xi • The standard inner product defined in Rn is the "row-column" product of two 11 vectors n T X hx; yi = x y = xiyi i=1 • Orthogonality. Two vectors x; y 2 Rn are orthogonal if xT y = 0. 1.3.2 Norms • When we try to define the notion of size, or length, of a vector in high dimen- sions (not just a scalar), we are faced with many choices. These choices are called norm. • The norm of a vector x, denoted by kxk, is a real-valued function that maps any element x 2 X into a real number kvk that satisfy a set of rules that the notion of size should involved. • Definition of norm. A function from X to Rn is a norm if 1: kxk ≥ 0 8x 2 X ; and kxk = 0 if and only if x = 0 2: kx + yk ≤ kxk + kyk; for any x; y 2 X (triangle inequality) 3: kαxk = jαjkxk; (for any scalar α and any x 2 X ) • The Euclidean Norm (l2-norm). The euclidean norm corresponds to the usual notion of distance in two or three dimensions. The set of points with equal l2-norm is a circle (in 2D) and a sphere (in 3D), or a hyper-sphere in higher dimensions. v u m p uX 2 T p jjxjj2 = t xi = x x = hx; xi i=1 12 • The l1-norm.