θ 3.7 CRLB for Vector θParameter Case θ T Vector Parameter: θ = [1 θ2 θ p ] ˆ ˆ ˆ ˆ T Its Estimate: θ = [1 2 θ p ]

Assume that estimate is unbiased: E{θˆ}= θ For a scalar parameter we looked at its variance… but for a vector parameter we look at its covariance matrix:

ˆ  ˆ ˆ T  var{θ}= E [θ − θ][θ − θ]  = C ˆ   θ For example:  var(xˆ) cov(xˆ, yˆ) cov(xˆ, zˆ)     T for θ = [x y z] C ˆ = cov( yˆ, xˆ) var( yˆ) cov( yˆ, zˆ) θ       cov(zˆ, xˆ) cov(zˆ, yˆ) var(zˆ)  1 Fisher Information Matrix For the vector parameter case…

Fisher Info becomes the Fisher Info Matrix (FIM) I(θ) whose mnth element is given by:

Evaluate at 2 true value of θ ∂ ln p(x;θ) [] []I(θ) mn = −E θ , m,n = 1, 2,… , p  ∂ n ∂θm 

2 The CRLB Matrix Then, under the same kind of regularity conditions, the CRLB matrix is the inverse of the FIM: CRLB = I−1(θ)

2 −1 So what this means is: σ ˆ = [C ˆ ]nn ≥ [I (θ)]nn θn θ (!)

Diagonal elements of Inverse FIM bound the parameter variances, which are the diagonal elements of the parameter covariance matrix

 var(xˆ) cov(xˆ, yˆ) cov(xˆ, zˆ) b11 b12 b13          −1 C ˆ = cov( yˆ, xˆ) var( yˆ) cov( yˆ, zˆ) b21 b22 b23 = I (θ) θ             cov(zˆ, xˆ) cov(zˆ, yˆ) var(zˆ)  b31 b32 b33 3 More General Form of The CRLB Matrix

C I−1(θ) is positive semi - definite θˆ −

Mathematical Notation for this is:

C I−1(θ) 0 θˆ − ≥ (!!)

Note: property #5 about p.d. matrices on p. 573 states that (!!) ⇒ (!)

4 CRLB Off-Diagonal Elements Insight Not In Book T Let θ = [xe ye] represent the 2-D x-y location of a transmitter (emitter) to be estimated. Consider the two cases of “scatter plots” for the estimated location: yˆe yˆe

σ yˆe σ yˆ ye ye e

σ σ xˆe xˆe ˆ ˆ xe xe xe xe Each case has the same variances… but location accuracy characteristics are very different. ⇒ This is the effect of the off-diagonal elements of the covariance

Should consider effect of off-diagonal CRLB elements!!! 5 CRLB Matrix and Error Ellipsoids Not In Book ˆ T Assume θ = []xˆe yˆe is 2-D Gaussian w/ zero mean and a cov matrix C θˆ Only For Convenience Then its PDF is given by: Let : 1  1  p θˆ = exp − θˆ T C −1θˆ −1 ()  θˆ  C ˆ = A 2 N C  2  θ ()π θˆ Quadratic Form!! for ease (recall: it’s scalar valued)

So the “equi-height contours” of this PDF are given by the values of θˆ such that: T θˆ Aθˆ = k Some constant

Note: A is symmetric so a12 = a21 …because any cov. matrix is symmetric and the inverse of symmetric is symmetric 6 2 2 What does this look like? a11 xˆ e + 2a12 xˆe yˆ e + a 22 yˆ e = k An Ellipse!!! (Look it up in your calculus book!!!)

Recall: If a12 = 0, then the ellipse is aligned w/ the axes & the a11 and a22 control the size of the ellipse along the axes  1  a 0  0 11 a  −1    11  Note: a12 = 0 ⇒ Cˆ = ⇒ C = θ   θˆ  1   0 a22   0   a22  ⇒ xˆe & yˆe are uncorrelated

Note: a ≠ 0 ⇒ xˆ & yˆ are correlated 12 e σ e  2 σ  xˆ ˆ ˆ  e xe ye  C = σ θˆ    2  yˆ xˆ σ yˆ  e e e  7 Error Ellipsoids and Correlation Not In Book

if xˆe & yˆe are uncorrelated yˆe Choosing k Value yˆe For the 2-D case…

~ 2σ ~ 2σ k = -2 ln(1-Pe) yˆe yˆe xˆe xˆe where Pe is the prob. ~ 2σ that the estimate will xˆe ~ 2σ lie inside the ellipse xˆe

if xˆe & yˆe are correlated yˆe yˆe See posted paper by ~ 2σ ˆ ~ 2σ ˆ ye ye Torrieri xˆe xˆe

~ 2σ xˆe ~ 2σ 8 xˆe Ellipsoids and Eigen-Structure Not In Book Consider a symmetric matrix A & its quadratic form xTAx ⇒ Ellipsoid:x T Ax = k or Ax , x = k

Principle Axes of Ellipse are orthogonal to each other… and are orthogonal to the tangent line on the ellipse:

x2

x1

Theorem: The principle axes of the ellipsoid xTAx = k are eigenvectors of matrix A.

9 Proof: From multi-dimensional calculus: gradient of a scalar-valued function φ(x1,…, xn) is orthogonal to the surface:

x2

φ x1 φ Different φ Notations ∂ (x) grad (x1,…, xn ) = ∇x (x) = = φ ∂x T  ∂ ∂φ  =   ∂x1 ∂xn 

See handout posted on Blackboard on Gradients and Derivatives

10 φ

For our quadratic form function we have:

T ∂φ ∂(xi x j ) (x) = x A x = ∑∑aij xi x j ⇒ = ∑∑aij (♣) ij ∂xk i j ∂xk

∂(x x ) ∂x ∂x Product rule: i j = i x + x j ∂x ∂x j i ∂x (♣♣) k δk k 1 i=k δ jk = ik = 0 i≠k

Using (♣♣) in (♣) gives: ∂φ = ∑ a jk x j + ∑ aik x j ∂xk j i By Symmetry: = 2 akj x j ∑ aik = aki j And from this we get: T ∇x (x Ax) = 2Ax

11 Since grad ⊥ ellipse, this says Ax is ⊥ ellipse:

x2 Ax x

x1

Ax , x = k

When x is a principle axis, then x and Ax are aligned:

x2 x Ax Ax = λx x1 Eigenvectors are Ax , x = k Principle Axes!!!

< End of Proof > 12 Theorem: The length of the principle axis associated with

eigenvalue λi is k / λi Proof: If x is a principle axis, then Ax = λx. Take inner product of both sides of this with x: λ k k Ax , x = λ x, x x, x = ⇒ x = λ =k 2 = x < End of Proof >

Note: This says that if A has a zero eigenvalue, then the error ellipse will have an infinite length principle axis ⇒ NOT GOOD!!

So… we’ll require that all λi > 0 ⇒ C must be positive definite θˆ

13 Application of Eigen-Results to Error Ellipsoids The Error Ellipsoid corresponding to the estimator covariance matrix C must satisfy: θˆ θˆ T C−1θˆ = k θˆ Note that the error ellipse is formed Thus finding the eigenvectors/values of C−1 using the inverse cov θˆ shows structure of the error ellipse

Recall: Positive definite matrix A and its inverse A-1 have the • same eigenvectors • reciprocal eigenvalues

Thus, we could instead find the eigenvalues of C = I−1(θ) θˆ and then the principle axes would have lengths set by its eigenvalues not inverted Inverse FIM!!

14 ˆ θ2 Illustrate with 2-D case: θˆ T C−1θˆ k θˆ = v v & v 1 1 2 v2 λ1 & λ2 ˆ θ1 Eigenvectors/values for C kλ1 θˆ kλ (not the inverse!) 2

15 The CRLB/FIM Ellipse Can make an ellipse from the CRLB Matrix… instead of the Cov. Matrix

This ellipse will be the smallest error ellipse that an unbiased estimator can achieve!

We can re-state this in terms of the FIM…

Once we find the FIM we can: • Find the inverse FIM • Find its eigenvectors… gives the Principle Axes • Find its eigenvalues… Prin. Axis lengths are then kλi

16