CIRA DA Training Mathematical introduction to data assimilation. Linear algebra and functional analysis Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University Fort Collins, Colorado Outline § Inner product, orthogonality § Norm and distance § Basis vectors § Matrix § Singular and eigenvalue decomposition 2 Inner product of vectors Why is inner product relevant to data assimilation? - cost function calculation - minimization - verification a = [ a1 a2 aN ] b = [ b1 b2 bN ] § Inner product is a scalar T T a,b = b,a = a b = b a = ∑aibi = a1b1 + a2b2 ++ aNbN i Fortran program: ab_inner=0. do i=1,N OR ab_inner=DOT_PRODUCT (a,b) ab_inner=ad_inner+a(i)*b(i) enddo 3 Inner product examples (N=2) a,b = a b + a b ++ a b 1 1 2 2 N N Example 2: Example 1: a = [ 3 1 ] b = [ 2 4 ] a = [ 3 1 ] b = [ −1 3 ] b b a a a,b = a1b1 + a2b2 = 3⋅(−1) +1⋅ 3 = 0 a,b = a1b1 + a2b2 = 3⋅2 +1⋅ 4 = 10 Orthogonality: If inner product is zero, vectors are orthogonal, and vice versa 4 Vector norm Why is vector norm relevant to data assimilation? - defines “magnitude” of a vector - all aspects (cost function, minimization, verification In data assimilation (e.g., Hilbert and Euclidian spaces), norm is connected with inner product 1/2 a = a,a Hilbert/Euclidian spaces are vector spaces with defined inner product and metric d(x,y): d(x, y) = x − y (Euclidian space is a finite-dimensional Hilbert space) The above characteristics of vector spaces used in data assimilation allow us to talk about “close”, “distant”, “small”, “large”, “orthogonal”, etc. 5 Properties and types of vector norms a = [ a1 a2 aN ] b = [ b1 b2 bN ] Elementary properties of norms Lp norm (p ≥ 1): § Non-negative function a ≥ 0 N 1/p ⎛ p ⎞ § Scalar multiplication γ a = γ a a = ⎜ ∑ ai ⎟ ⎝ i=1 ⎠ § Triangle inequality a + b ≤ a + b Important Lp-norms (1) Absolute value norm: a = a + a ++ a p=1 1 2 N (2) Quadratic norm: a = a2 + a2 ++ a2 p=2 1 2 N (3) Maximum (infinity) norm: a max a , a , , a p=∞ = ( 1 2 … N ) 6 What is the distance between a and b? a = [ 0 5 ] a-b d(a,b) = a − b = ? a b = [ 3 1 ] b A1: Absolute norm a − b p=1 = a1 − b1 + a2 − b2 = 0 − 3 + 5 −1 = 3+ 4 = 7 A2: Quadratic norm 2 2 2 2 a − b p=2 = (a1 − b1) + (a2 − b2 ) = (0 − 3) + (5 −1) = 5 A3: Maximum norm a b max a b , a b max 0 3 , 5 1 4 − p=∞ = ( 1 − 1 2 − 2 ) = ( − − ) = a − b p=1 ≥ a − b p=2 ≥ a − b p=∞ Distance may not be intuitive, it depends on the used norm 7 Function norm and distance df (x) Define a function and its derivative: y = f (x) y' = f '(x) = dx § One can define L2 norm in terms of function values, or its derivatives: (1) f [ f (x)]2 [ f (x)]2 [ f (x)]2 [ f (x)]2 VALUE = ∑ i = 1 + 2 ++ N i (2) f [ f '(x)]2 [ f '(x)]2 [ f '(x)]2 [ f '(x)]2 DERIV = ∑ i = 1 + 2 ++ N i 8 Example y1 y2 Q: Are the functions y1 and y2 “close” or “distant”? is “small” y1 − y2 VALUE is “large” y1 − y2 DERIV The two functions are “close” in norm (1), but “distant” in norm (2) 9 Basis of a vector space Why is basis relevant to data assimilation? - representation of forecast and analysis errors - defines analysis adjustment - uncertainty dimension Minimal set of linearly independent vectors that spans a space γ e Linearly independent: Given scalars i and vectors i, i = 1,…, N γ e +γ e + +γ e ≠ 0 1 1 2 2 N N Span: Any vector from that space can be expressed as a linear combination of basis vectors N F = (e1,e2 ,,eN ) ∃i ∈(1, N) :γ i ≠ 0 N b = γ 1e1 +γ 2e2 ++γ N eN ⇒ b ∈F 10 Orthogonal basis Most relevant basis is the orthogonal basis: ⎧ T ⎪ 1 for i = j ei ,ej = ei ej = δ ij = ⎨ 0 for i ≠ j ⎩⎪ Orthogonal basis with unit norm is called the orthonormal basis: e = 1 Example: Standard basis y e1 = (1,0) e2 e2 = (0,1) x e1 T 1 1 2 2 e1 e2 = e1,e2 = e1e2 + e1 e2 = 1⋅0 + 0 ⋅1 = 0 11 Basis representation of a vector Vector is generally represented by its components a (a ,a , ,a ) = 1 2 … N Given an orthonormal basis (e ,e , ,e ) 1 2 N This means a a e a e a e = 1 1 + 2 2 ++ N N Q: What is the meaning of vector components? A: Orthogonal projection of a vector to the basis component. T T a1 = e1 a a2 = e2 a y a1 Example: a eT a 1 2 0 3 2 a = (2,3) 1 = 1 = ⋅ + ⋅ = T e1 = (1,0) a2 = e2 a = 0 ⋅2 +1⋅ 3 = 3 a a2 e2 = (0,1) a = 2e1 + 3e2 e2 e1 x 12 Basis examples (calculate, plot) α = 900 e1 = (sinα,cosα ) Q1: e1 = (?,?) e2 = (− cosα,sinα ) e2 = (?,?) Q2: α = 450 e1 = (?,?) e2 = (?,?) 0 Q3 (with 45 ): a = ( 2,3 2) a1 = ? a2 = ? 13 Linear transformation and matrices § A linear transformation from vector spaces V to W is a function L such that L :V → W L(µa + b) = µ(La) + Lb § Matrix is a linear transformation defined relative to a particular basis N M V = (v1,v2 ,,vN ) Lvj = Aijwi M ∑ i 1 W = (w1,w2 ,,wM ) = The matrix A is given by its elements Aij and represents the linear transformation L relative to the pair of bases {v} and {w} Why are matrices and linear transformations relevant to data assimilation? - Error covariance (analysis and forecast uncertainty) - Forecast model, observation operator - Minimization, Hessian matrix, Kalman gain 14 Matrix § Matrix is a rectangular array of numbers (real numbers in our applications) § Matrix represents linear transformation (scaling and rotation) § Matrix has rows (horizontal) and columns (vertical) ⎛ ⎞ d11 d12 d1N ⎜ ⎟ d d D = ⎜ 21 22 ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ dN1 dNN ⎠ Transpose matrix Symmetric matrix Diagonal matrix Identity matrix ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ d11 d21 dN1 d11 d12 d1N d11 0 0 ⎛ 1 0 0 ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ d d d d 0 d ⎜ 0 1 ⎟ DT = ⎜ 12 22 ⎟ D = ⎜ 12 22 ⎟ D = ⎜ 22 ⎟ I = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ d1N dNN ⎠ ⎝ d1N dNN ⎠ ⎝ 0 dNN ⎠ ⎝ 0 1 ⎠ § All these types of matrices are relevant in data assimilation 15 Matrix properties § Inverse of a product (ABC)−1 = C −1B−1A−1 § Transpose of a product (ABC)T = CT BT AT § Symmetric positive-definite matrix ⎛ ⎞ d11 d12 d1N ⎜ ⎟ d d T D = ⎜ 12 22 ⎟ x, Dx = x Dx ≥ 0 for any x ⎜ ⎟ ⎜ ⎟ ⎝ d1N dNN ⎠ T § Inner-product matrix A A Inner and outer product matrices are § Outer-product matrix AAT symmetric and positive-definite 2 xT (AT A)x = (xT AT )(Ax) = (Ax)T (Ax) = Ax ≥ 0 Error covariance matrices are symmetric and positive-definite 16 Examples Q1: What is the value of J(x)? Is it a scalar, vector, or matrix? 1 J(x) = xTWx 2 T ⎛ 9 1 ⎞ x = (1,2) W = ⎝⎜ 1 4 ⎠⎟ T T T ⎛ 1 ⎞ ⎛ 9 1 ⎞ ⎛ 1 ⎞ ⎛ 1 ⎞ ⎛ 9 ⋅1+1⋅2 ⎞ ⎛ 1 ⎞ ⎛ 11 ⎞ J(x) = = = = 1⋅11+ 2 ⋅ 7 = 25 ⎝⎜ 2 ⎠⎟ ⎝⎜ 1 4 ⎠⎟ ⎝⎜ 2 ⎠⎟ ⎝⎜ 2 ⎠⎟ ⎝⎜ 1⋅1+ 4 ⋅2 ⎠⎟ ⎝⎜ 2 ⎠⎟ ⎝⎜ 7 ⎠⎟ Q2: What is D-inverse (D-1)? ⎛ 1 ⎞ 0 0 0 ⎜ 7 ⎟ ⎜ ⎟ ⎛ 7 0 0 0 ⎞ ⎜ 1 ⎟ 0 − 0 0 ⎜ 0 −2 0 0 ⎟ D−1 = ⎜ ⎟ D = ⎜ ⎟ 2 0 0 3 0 ⎜ ⎟ ⎜ ⎟ ⎜ 1 ⎟ ⎜ ⎟ 0 0 0 ⎝ 0 0 0 1 ⎠ ⎜ 3 ⎟ ⎜ ⎟ ⎝ 0 0 0 1 ⎠ 17 Linear algebra, functional analysis § Vector space (e.g., linear space) is an object consisting of (1) field, (2) set of vectors and (3) addition rule § Linear algebra is a vector space with multiplication rule § Functional analysis is a branch of mathematics that deals with normed linear spaces associated with geometry and linear operators In data assimilation, we utilize all these components: 1- vector space: Any physical field (e.g., temperature, pressure, aerosol, chemical constituents, satellite radiance) is mathematically represented as a vector. A set of these vectors forms a vector space. 2- linear algebra: We use matrices to represent uncertainties, linear functions. 3- functional analysis: All operations with matrices and vectors are based on the notion of norms, distances, and inner products 18 Literature: Conway, J. B., 1985: A course in functional analysis. Springer-Verlag New York, pp. 404. Golub, G. H., and C.F. van Loan, 1989: Matrix computations. THe JoHns Hopkins Univ. Press, pp. 642. Hoffman, K., and R. Kunze, 1971: Linear algebra. Prentice Hall, Englewood Cliffs, New Jersey, pp.407. Horn, A.R., and C. R. JoHnson, 2005: Matrix analysis. Cambridge Univ. Press, pp. 561. 19.