Chapters 5 Vector Spaces David Hilbert (1862–1943, Germany) Giuseppe Peano (1858–1932, Italy) 1 5.1 Vector multiplication: Geometric interpretation Think of a vector (an Euclidian vector) as a directed line segment in N- x2 dimensions! (has “length” and “direction”) 6 64 2U 5 Scalar multiplication 4 (“scales” the vector –i.e., 32 U changes length) 3 Source of linear 2 dependence 1 132U x1 -4 -3 -2 -1 1 2 3 4 5 6 -2 2 1 5.1 Vector Addition: Geometric interpretation x2 v' = [2 3] 5 u' = [3 2] 4 u w’ = v‘ + u' = [5 5] 3 v w Note: Two vectors plus the 2 concepts of addition and 1 u multiplication can create a two- x1 dimensional space. (Space = A set 1 2 3 4 5 of points, say R3, a “universe”.) • A vector space is a mathematical structure formed by a collection of vectors, which may be added together and multiplied by scalars. (It is closed under multiplication and addition.) Giuseppe Peano in 1888 gave a precise definition to this concept. 3 5.2 Vector (Linear) Space We introduce an algebraic structure called vector space over a field. We use it to provide an abstract notion of a vector: an element of such algebraic structure. Given a field R (of scalars) and a set V of objects (vectors), on which “vector addition” (VxV→V), denoted by “+”, and “scalar multiplication” (RxV →V), denoted by “. ”, are defined. If the following axioms are true for all objects u, v, and w ∈V and all scalars c and k in R, then V is called a vector space and the objects in V are called vectors. 1. u + v ∈V (closed under addition). 2. u + v = v + u (vector addition is commutative). 3. Ø ∈V, such that u+ Ø = u (Ø = null element). 4. u + (v + w) = (v + u) +w (distributive law of vector addition) 4 2 5.2 Vector Space 5. For each v, there is a –v, such that v + (–v) = Ø 6. c .u ∈ V (closed under scalar multiplication). 7. c. (k . u) = (c .k) u (scalar multiplication is associative). 8. c. (v + u) = (c. v) + (c. u) 9. (c + k) . u = (c. u) + (k. u) 10. 1.u=u (unit element). 11. 0.u= Ø (zero element). We can write S = {V, R, +, .}to denote an abstract vector space. This is a general definition. If the field R represents the real numbers, then we define a real vector space. Giuseppe Peano (1858 – 1932, Italy) 5 5.2 Vector Space: Examples 1. n-dimensional vectors: x ⋮ ∈ Rn, Cn Note: The vector space consisting of n-column vectors, with vector addition and multiplication corresponding to matrix operations is an n-dimensional vector space (Euclidean n-space), which we will denote Rn. 2. An infinite sequence of real numbers. We will be interested in bounded sequences such that {xk} < M, for M < ∞. x ⋮ ∈ R∞, C∞ ⋮ 6 3 5.2 Vector Space: Examples 3. The collection of all continuous real valued functions f(t) on the interval C[a,b] on the real line is a linear vector space. The zero vector is the function identically zero on [a,b]. 4. The collection of polynomial functions on the interval [a,b] is a linear vector space. 7 5.2 Vector Space: Subspace Definition: Subspace Given the vector space V and W a set of vectors, such that W∈V. Then, W is a subspace if it also a vector space. That is, -u, v ∈ W u + v ∈V,and -u ∈ W & for every c ∈R c.u ∈V. -Wcontains the 0-vector of V. Thus, a nonempty subset W of a vector space V that is closed under addition and scalar multiplication and contains the 0-vector of V is a subspace of V. That is, a subspace is a subset of V that can be considered a vector space! 8 4 5.2 Vector Space: Subspace v Example: Subspace 2 W2 W2 is not a subspace of V, it does not include the 0-vector of V. v3 v W2 is just a “hyperplane.” 1 Definition: Linear Combination Given vectors u1, ..., uk,, the vector w = c1 u1+ ....+ ck uk is called a linear combination of the vectors u1, ..., uk,. Notation: <u1, ..., uk> is the set of all linear combinations of uj’s. Recall that a set of vectors is linearly dependent if any one of them can be expressed as a linear combination of the remaining vectors; otherwise, the set is linearly independent (LI). 9 5.2 Vector Space: Rank Definition: Maximally linear independent (max-LI) subset Given a set U={u1, ..., uk} of vectors in a vector space V. If W ={ui,1, ..., ui,q} ∈ U containing q vectors is LI and every subset with more than q vectors is LD, then W is called maximally linear independent subset of U. Moreover, we will call q the rank of the set U, written q = rank(U). (The max-LI subset is not unique.) Definition: Full rank Given A (mxn). We say A has full rank if rank(A) = min(m,n). 38 5 5.2 Vector Space: Spanning Set Definition: Spanning set Given the set Z in V and U = {u1, ..., uk} in Z, we say U spans Z, or U is a spanning set for Z, if Z ∈<u1, ..., uk> or Z ∈<U>. That is, a set of vectors spans Z if all the vectors in Z can be expressed in terms of this set of vectors. v2 Example: Vectors v1 & v3 span the v3 – v1 plane. Also, v3 & u1 also span the same v3 – v1 plane. v3 v 1 u1 35 5.2 Vector Space: Basis Definition: Basis set (“basis”) Given U = {u1, ..., uk} and a subspace W∈V. Then, U is a basis set for W if 1) span the subspace W, 2) U is linearly independent (LI). Example: The N-dimensional subspace WN of the V space (N=2). v2 W2 u1 u2 v3 v1 36 6 5.2 Vector Space: Basis and Space Dimension Theorem: If a vector space has a basis with a finite number N of elements, then, every other basis also has N elements. Definition: Space dimension If a vector space V has a basis with N < ∞ elements, we say that V is a finite dimensional vector space and that V has dimension N, or N = dim(V). Theorem: If {u1, ..., uk} are linearly independent vectors for k < N, where N is the dimension of the vector space, we can always construct a basis by adding additional independent vectors: {u , ..., u , u , ..., u }. 1 k k+1 N 13 5.2 Vector Space: Basis and Space Dimension A vector space V can be decomposed into independent subspaces instead of vectors, say V1, V2, …, Vm. Then, V = V1 + V2 + V3 + … + Vm We call this direct sum decomposition. It is similar to a basis decomposition when the Vi all have dimension 1. Example: The 3-dimensional space V can be decompose as: V = v1+ v2 + v3, where the vi’s are LI. Alternatively, V can be decomposed as V = V1+ v3,where V1 = v1+ v2. If V=U+W, then dim(V)= dim(U)+dim(W), where U∩W = {0}. 14 7 5.2 Vector Space: Measuring Length As we defined them, vector spaces do not provide enough structure to study issues in real analysis, for example convergence of sequences. More structure is needed. For example, we can introduce as an additional structure the concept of order (≤), to compare vectors. This additional structure creates ordered vector spaces. We can introduce a norm, which we will use to measure the length or magnitude of vectors. This creates a normed vector space, denoted as a pair (V, ║.║) where V is a vector space and ║.║ is a norm on V. A normed vector space has a defined mapping from V → R1. 15 5.3 Notes on Vector Operations • An (mx1) column vector u and a (1xn) row vector v, yield a product matrix uv of dimension (mxn). 3 u 2 v 1 4 5 3 x 1 1x3 1 3 3 12 15 u v 2 1 4 5 2 8 10 A matrix 3 x 3 1 1 4 5 1 u ' v 3 2 1 4 16 A scalar 1 x 1 5 16 8 5.3 Vector Multiplication: Dot (inner) Product • The dot or Inner product (IP), “•”, is a function that takes pairs of vectors and produces a number. For vectors c & z, it is defined as: • ∗ + ∗ + ... + ∗ ∑ … • ⋮ • The dot product produces a scalar! y = c’z =(1x1=1xnnx1)= z’c. Note that from the definition, the dot product is commutative. • When c is a vector of 1’s, usually noted as ι, then: • 1 ∗ + 1 ∗ + ... + 1 ∗ ∑ 17 5.3 Vectors: Dot Product • Inner products (IP) in econometrics are common. For example, the Residual Sum of Squares (RSS), where e is a vector of residuals: • ′ ∗+ ∗+...+ ∗ ∑ • It is possible to define an inner product for functions. Instead of a sum over the corresponding elements of a vector, the inner product on functions is defined as an integral over some interval. For example, for functions f(x) & g(x): f • g = f(x) g(x) dx 9 5.3 Vectors: Dot Product - Intuition • Some intuition. - IP is used as a tool to define length or size for vectors: ∥ α ∥ = sqrt[α′ α] = α ⋯α The square root makes the inner product to be expressed in the same units as the original vector. - Now, it is possible to compare vectors and measure “distances” between vectors and, eventually, convergence! - IP also can be used to define a notion like angle between vectors, since any two vectors, say α and β, determine a plane.