Linear Algebra and Functional Analysis
CIRA DA Training
Mathematical introduction to data assimilation. Linear algebra and functional analysis
Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University Fort Collins, Colorado Outline
§ Inner product, orthogonality § Norm and distance § Basis vectors § Matrix § Singular and eigenvalue decomposition
2 Inner product of vectors
Why is inner product relevant to data assimilation? - cost function calculation - minimization - verification
a = [ a a a ] b = [ b b b ] 1 2 N 1 2 N § Inner product is a scalar
T T a,b = b,a = a b = b a = ∑aibi = a1b1 + a2b2 ++ aNbN i
Fortran program:
ab_inner=0. do i=1,N OR ab_inner=DOT_PRODUCT (a,b) ab_inner=ad_inner+a(i)*b(i) enddo
3 Inner product examples (N=2)
a,b = a b + a b ++ a b 1 1 2 2 N N Example 2: Example 1: a = [ 3 1 ] b = [ 2 4 ] a = [ 3 1 ] b = [ −1 3 ]
b b
a a
a,b = a1b1 + a2b2 = 3⋅(−1) +1⋅ 3 = 0 a,b = a1b1 + a2b2 = 3⋅2 +1⋅ 4 = 10
Orthogonality: If inner product is zero, vectors are orthogonal, and vice versa 4 Vector norm
Why is vector norm relevant to data assimilation? - defines “magnitude” of a vector - all aspects (cost function, minimization, verification
In data assimilation (e.g., Hilbert and Euclidian spaces), norm is connected with inner product 1/2 a = a,a
Hilbert/Euclidian spaces are vector spaces with defined inner product and metric d(x,y): d(x, y) = x − y
(Euclidian space is a finite-dimensional Hilbert space)
The above characteristics of vector spaces used in data assimilation allow us to talk about “close”, “distant”, “small”, “large”, “orthogonal”, etc.
5 Properties and types of vector norms
a = [ a a a ] b = [ b b b ] 1 2 N 1 2 N Elementary properties of norms Lp norm (p ≥ 1): § Non-negative function a ≥ 0 N 1/p ⎛ p ⎞ § Scalar multiplication γ a = γ a a = ⎜ ∑ ai ⎟ ⎝ i=1 ⎠ § Triangle inequality a + b ≤ a + b
Important Lp-norms
(1) Absolute value norm: a = a + a ++ a p=1 1 2 N
(2) Quadratic norm: a = a2 + a2 ++ a2 p=2 1 2 N
(3) Maximum (infinity) norm: a = max a , a ,…, a p=∞ ( 1 2 N ) 6 What is the distance between a and b?
a = [ 0 5 ] a-b d(a,b) = a − b = ? a b = [ 3 1 ] b
A1: Absolute norm a − b p=1 = a1 − b1 + a2 − b2 = 0 − 3 + 5 −1 = 3+ 4 = 7
A2: Quadratic norm 2 2 2 2 a − b p=2 = (a1 − b1) + (a2 − b2 ) = (0 − 3) + (5 −1) = 5
A3: Maximum norm a b max a b , a b max 0 3 , 5 1 4 − p=∞ = ( 1 − 1 2 − 2 ) = ( − − ) =
a − b p=1 ≥ a − b p=2 ≥ a − b p=∞
Distance may not be intuitive, it depends on the used norm 7 Function norm and distance
df (x) Define a function and its derivative: y = f (x) y' = f '(x) = dx
§ One can define L2 norm in terms of function values, or its derivatives:
(1) f [ f (x)]2 [ f (x)]2 [ f (x)]2 [ f (x)]2 VALUE = ∑ i = 1 + 2 ++ N i
(2) f [ f '(x)]2 [ f '(x)]2 [ f '(x)]2 [ f '(x)]2 DERIV = ∑ i = 1 + 2 ++ N i
8 Example
y1
y2
Q: Are the functions y1 and y2 “close” or “distant”?
is “small” y1 − y2 VALUE
is “large” y1 − y2 DERIV
The two functions are “close” in norm (1), but “distant” in norm (2)
9 Basis of a vector space
Why is basis relevant to data assimilation? - representation of forecast and analysis errors - defines analysis adjustment - uncertainty dimension
Minimal set of linearly independent vectors that spans a space
γ e Linearly independent: Given scalars i and vectors i, i = 1,…, N e e e 0 γ 1 1 +γ 2 2 ++γ N N ≠ Span: Any vector from that space can be expressed as a linear combination of basis vectors
F N = (e ,e ,,e ) ∃i ∈(1, N) :γ ≠ 0 1 2 N i
b = γ e +γ e ++γ e ⇒ b ∈F N 1 1 2 2 N N
Most relevant basis is the orthogonal basis:
⎧ T ⎪ 1 for i = j ei ,ej = ei ej = δ ij = ⎨ 0 for i ≠ j ⎩⎪
Orthogonal basis with unit norm is called the orthonormal basis: e = 1
Example: Standard basis y
e1 = (1,0) e2 e2 = (0,1) x e1 T 1 1 2 2 e1 e2 = e1,e2 = e1e2 + e1 e2 = 1⋅0 + 0 ⋅1 = 0
11 Basis representation of a vector
Vector is generally represented by its components a (a ,a , ,a ) = 1 2 … N Given an orthonormal basis (e ,e , ,e ) 1 2 N This means a a e a e a e = 1 1 + 2 2 ++ N N
Q: What is the meaning of vector components? A: Orthogonal projection of a vector to the basis component.
T T a1 = e1 a a2 = e2 a
y a1 Example: a eT a 1 2 0 3 2 a = (2,3) 1 = 1 = ⋅ + ⋅ = T e1 = (1,0) a2 = e2 a = 0 ⋅2 +1⋅ 3 = 3 a a2 e2 = (0,1) a = 2e1 + 3e2 e2
e1 x
12 Basis examples (calculate, plot)
α = 900 e1 = (sinα,cosα ) Q1: e1 = (?,?) e2 = (− cosα,sinα ) e2 = (?,?)
Q2: α = 450
e1 = (?,?)
e2 = (?,?)
0 Q3 (with 45 ): a = ( 2,3 2)
a1 = ?
a2 = ?
13 Linear transformation and matrices
§ A linear transformation from vector spaces V to W is a function L such that L :V → W L(µa + b) = µ(La) + Lb
§ Matrix is a linear transformation defined relative to a particular basis
V N (v ,v , ,v ) M = 1 2 N Lvj = Aijwi M ∑ W (w ,w , ,w ) i=1 = 1 2 M
The matrix A is given by its elements Aij and represents the linear transformation L relative to the pair of bases {v} and {w}
Why are matrices and linear transformations relevant to data assimilation? - Error covariance (analysis and forecast uncertainty) - Forecast model, observation operator - Minimization, Hessian matrix, Kalman gain 14 Matrix
§ Matrix is a rectangular array of numbers (real numbers in our applications) § Matrix represents linear transformation (scaling and rotation) § Matrix has rows (horizontal) and columns (vertical)
⎛ ⎞ d11 d12 d1N ⎜ ⎟ d d D = ⎜ 21 22 ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ dN1 dNN ⎠
Transpose matrix Symmetric matrix Diagonal matrix Identity matrix
⎛ ⎞ ⎛ ⎞ ⎛ ⎞ d11 d21 dN1 d11 d12 d1N d11 0 0 ⎛ 1 0 0 ⎞ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ d d d d 0 d ⎜ 0 1 ⎟ DT = ⎜ 12 22 ⎟ D = ⎜ 12 22 ⎟ D = ⎜ 22 ⎟ I = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎝ d1N dNN ⎠ ⎝ d1N dNN ⎠ ⎝ 0 dNN ⎠ ⎝ 0 1 ⎠
§ All these types of matrices are relevant in data assimilation 15 Matrix properties
§ Inverse of a product (ABC)−1 = C −1B−1A−1
§ Transpose of a product (ABC)T = CT BT AT
§ Symmetric positive-definite matrix
⎛ ⎞ d11 d12 d1N ⎜ ⎟ d d T D = ⎜ 12 22 ⎟ x, Dx = x Dx ≥ 0 for any x ⎜ ⎟ ⎜ ⎟ ⎝ d1N dNN ⎠
T § Inner-product matrix A A Inner and outer product matrices are § Outer-product matrix AAT symmetric and positive-definite
2 xT (AT A)x = (xT AT )(Ax) = (Ax)T (Ax) = Ax ≥ 0
Error covariance matrices are symmetric and positive-definite 16 Examples
Q1: What is the value of J(x)? Is it a scalar, vector, or matrix?
1 J(x) = xTWx 2
T ⎛ 9 1 ⎞ x = (1,2) W = ⎝⎜ 1 4 ⎠⎟
T T T ⎛ 1 ⎞ ⎛ 9 1 ⎞ ⎛ 1 ⎞ ⎛ 1 ⎞ ⎛ 9 ⋅1+1⋅2 ⎞ ⎛ 1 ⎞ ⎛ 11 ⎞ J(x) = = = = 1⋅11+ 2 ⋅ 7 = 25 ⎝⎜ 2 ⎠⎟ ⎝⎜ 1 4 ⎠⎟ ⎝⎜ 2 ⎠⎟ ⎝⎜ 2 ⎠⎟ ⎝⎜ 1⋅1+ 4 ⋅2 ⎠⎟ ⎝⎜ 2 ⎠⎟ ⎝⎜ 7 ⎠⎟
Q2: What is D-inverse (D-1)? ⎛ 1 ⎞ 0 0 0 ⎜ 7 ⎟ ⎜ ⎟ ⎛ 7 0 0 0 ⎞ ⎜ 1 ⎟ 0 − 0 0 ⎜ 0 −2 0 0 ⎟ D−1 = ⎜ ⎟ D = ⎜ ⎟ 2 0 0 3 0 ⎜ ⎟ ⎜ ⎟ ⎜ 1 ⎟ ⎜ ⎟ 0 0 0 ⎝ 0 0 0 1 ⎠ ⎜ 3 ⎟ ⎜ ⎟ ⎝ 0 0 0 1 ⎠
17 Linear algebra, functional analysis
§ Vector space (e.g., linear space) is an object consisting of (1) field, (2) set of vectors and (3) addition rule § Linear algebra is a vector space with multiplication rule § Functional analysis is a branch of mathematics that deals with normed linear spaces associated with geometry and linear operators
In data assimilation, we utilize all these components:
1- vector space: Any physical field (e.g., temperature, pressure, aerosol, chemical constituents, satellite radiance) is mathematically represented as a vector. A set of these vectors forms a vector space. 2- linear algebra: We use matrices to represent uncertainties, linear functions. 3- functional analysis: All operations with matrices and vectors are based on the notion of norms, distances, and inner products
18 Literature:
Conway, J. B., 1985: A course in functional analysis. Springer-Verlag New York, pp. 404.
Golub, G. H., and C.F. van Loan, 1989: Matrix computations. The Johns Hopkins Univ. Press, pp. 642.
Hoffman, K., and R. Kunze, 1971: Linear algebra. Prentice Hall, Englewood Cliffs, New Jersey, pp.407.
Horn, A.R., and C. R. Johnson, 2005: Matrix analysis. Cambridge Univ. Press, pp. 561.
19