§9.2 Orthogonal Matrices and Similarity Transformations

Home , Diagonal matrix, Orthogonal matrix

n×n Thm: Suppose matrix Q ∈ R is orthogonal. Then −1 T I Q is invertible with Q = Q . n T T I For any x, y ∈ R ,(Q x) (Q y) = x y. n I For any x ∈ R , kQ xk2 = kxk2. Ex  1 1 1 1  2 2 2 2    1 1 1 1   − 2 2 − 2 2    T H =   , H H = I .    − 1 − 1 1 1   2 2 2 2    1 1 1 1 2 − 2 − 2 2

n×n Def: A matrix Q ∈ R is said to be orthogonal if its columns n (1) (2) (n)o n q , q , ··· , q form an orthonormal set in R . Ex  1 1 1 1  2 2 2 2    1 1 1 1   − 2 2 − 2 2    T H =   , H H = I .    − 1 − 1 1 1   2 2 2 2    1 1 1 1 2 − 2 − 2 2

§9.2 Orthogonal Matrices and Similarity Transformations

n×n Def: A matrix Q ∈ R is said to be orthogonal if its columns n (1) (2) (n)o n q , q , ··· , q form an orthonormal set in R . n×n Thm: Suppose matrix Q ∈ R is orthogonal. Then −1 T I Q is invertible with Q = Q . n T T I For any x, y ∈ R ,(Q x) (Q y) = x y. n I For any x ∈ R , kQ xk2 = kxk2. Ex  1 1 1 1  2 2 2 2    1 1 1 1   − 2 2 − 2 2    T H =   , H H = I .    − 1 − 1 1 1   2 2 2 2    1 1 1 1 2 − 2 − 2 2 Thm: Suppose A and B are similar matrices with A = S−1 BS and λ is an eigenvalue of A with associated eigenvector x. Then λ is an eigenvalue of B with associated eigenvector S x. Proof: Let x 6= 0 be such that

A x = S−1 BS x = λ x.

It follows that B (S x) = λ (S x)

Def: Two matrices A and B are similar if a nonsingular matrix S exists with A = S−1 BS. Proof: Let x 6= 0 be such that

A x = S−1 BS x = λ x.

It follows that B (S x) = λ (S x)

Def: Two matrices A and B are similar if a nonsingular matrix S exists with A = S−1 BS. Thm: Suppose A and B are similar matrices with A = S−1 BS and λ is an eigenvalue of A with associated eigenvector x. Then λ is an eigenvalue of B with associated eigenvector S x. Def: Two matrices A and B are similar if a nonsingular matrix S exists with A = S−1 BS. Thm: Suppose A and B are similar matrices with A = S−1 BS and λ is an eigenvalue of A with associated eigenvector x. Then λ is an eigenvalue of B with associated eigenvector S x. Proof: Let x 6= 0 be such that

A x = S−1 BS x = λ x.

It follows that B (S x) = λ (S x) Proof:

−1 (1) (2) (n) A = SDS , S = v , v , ··· , v , D = diag (λ1, λ2, ··· , λn)

(1) (2) (n) (1) (2) (n) ⇐⇒ A v , v , ··· , v = v , v , ··· , v diag (λ1, λ2, ··· , λn) , (j) (j) (1) (2) (n) ⇐⇒ A v = λj v , j = 1, 2, ··· , n. v , v , ··· , v L.I.D.

n Cor: A ∈ R with n distinct eigenvalues is similar to diagonal matrix.

n Thm: A ∈ R is similar to a diagonal matrix D if and only if A has n linearly independent eigenvectors. n Cor: A ∈ R with n distinct eigenvalues is similar to diagonal matrix.

n Thm: A ∈ R is similar to a diagonal matrix D if and only if A has n linearly independent eigenvectors. Proof:

−1 (1) (2) (n) A = SDS , S = v , v , ··· , v , D = diag (λ1, λ2, ··· , λn)

(1) (2) (n) (1) (2) (n) ⇐⇒ A v , v , ··· , v = v , v , ··· , v diag (λ1, λ2, ··· , λn) , (j) (j) (1) (2) (n) ⇐⇒ A v = λj v , j = 1, 2, ··· , n. v , v , ··· , v L.I.D. n Thm: A ∈ R is similar to a diagonal matrix D if and only if A has n linearly independent eigenvectors. Proof:

−1 (1) (2) (n) A = SDS , S = v , v , ··· , v , D = diag (λ1, λ2, ··· , λn)

(1) (2) (n) (1) (2) (n) ⇐⇒ A v , v , ··· , v = v , v , ··· , v diag (λ1, λ2, ··· , λn) , (j) (j) (1) (2) (n) ⇐⇒ A v = λj v , j = 1, 2, ··· , n. v , v , ··· , v L.I.D.

n Cor: A ∈ R with n distinct eigenvalues is similar to diagonal matrix. n Schur Thm: Let A ∈ R .A unitary matrix U exists such that     −1   T = U AU =   is upper-triangular.  

The diagonal entries of T are the eigenvalues of A.

n×n Def: A matrix U ∈ C is unitary if kU xk2 = kxk2 for any vector x. n×n Def: A matrix U ∈ C is unitary if kU xk2 = kxk2 for any vector x. n Schur Thm: Let A ∈ R .A unitary matrix U exists such that     −1   T = U AU =   is upper-triangular.  

The diagonal entries of T are the eigenvalues of A. √ √ T T T Def: kuk2 = a a + b b = u u. T n Thm: Let A = A ∈ R be symmetric. Then all eigenvalues of A are real. Proof: Let λ be an eigenvalue of A with eigenvector u. Then λ is eigenvalue, A u = λ u, −→ A u = λ u.

λ uT u = uT (A u) = (A u)T u = λ uT u .

Therefore λ = λ ∈ R.

Def: The complex√ conjugate of a√ complex vector u = a + −1 b ∈ Cn is u = a − −1 b. T n Thm: Let A = A ∈ R be symmetric. Then all eigenvalues of A are real. Proof: Let λ be an eigenvalue of A with eigenvector u. Then λ is eigenvalue, A u = λ u, −→ A u = λ u.

λ uT u = uT (A u) = (A u)T u = λ uT u .

Therefore λ = λ ∈ R.

Def: The complex√ conjugate of a√ complex vector u = a + −1 b ∈ Cn is u = a − −1 b. √ √ T T T Def: kuk2 = a a + b b = u u. Proof: Let λ be an eigenvalue of A with eigenvector u. Then λ is eigenvalue, A u = λ u, −→ A u = λ u.

λ uT u = uT (A u) = (A u)T u = λ uT u .

Therefore λ = λ ∈ R.

Def: The complex√ conjugate of a√ complex vector u = a + −1 b ∈ Cn is u = a − −1 b. √ √ T T T Def: kuk2 = a a + b b = u u. T n Thm: Let A = A ∈ R be symmetric. Then all eigenvalues of A are real. Def: The complex√ conjugate of a√ complex vector u = a + −1 b ∈ Cn is u = a − −1 b. √ √ T T T Def: kuk2 = a a + b b = u u. T n Thm: Let A = A ∈ R be symmetric. Then all eigenvalues of A are real. Proof: Let λ be an eigenvalue of A with eigenvector u. Then λ is eigenvalue, A u = λ u, −→ A u = λ u.

λ uT u = uT (A u) = (A u)T u = λ uT u .

Therefore λ = λ ∈ R. Proof: I By induction on n. Assume theorem true for n − 1. I Let λ be eigenvalue of A with unit eigenvector u: A u = λ u. n I We extend u into an orthonormal basis for R : u, u2, ··· , un are unit, mutually orthogonal vectors.

def def n×n I U = (u, u2, ··· , un) = u, Ub ∈ R is orthogonal.   T uT (A u) uT A U T u b U AU = T A u, A Ub =   Ub UbT (A u) UbT A Ub λ 0T ! = . 0 UbT A Ub

T (n−1)×(n−1) I Matrix Ub A Ub ∈ R is symmetric.

n Thm: A matrix A ∈ R is symmetric if and only if there exists a n diagonal matrix D ∈ R and an orthogonal matrix Q so that     T   T A = QDQ = Q   Q .   n Thm: A matrix A ∈ R is symmetric if and only if there exists a n diagonal matrix D ∈ R and an orthogonal matrix Q so that     T   T A = QDQ = Q   Q .  

Proof: I By induction on n. Assume theorem true for n − 1. I Let λ be eigenvalue of A with unit eigenvector u: A u = λ u. n I We extend u into an orthonormal basis for R : u, u2, ··· , un are unit, mutually orthogonal vectors.

def def n×n I U = (u, u2, ··· , un) = u, Ub ∈ R is orthogonal.   T uT (A u) uT A U T u b U AU = T A u, A Ub =   Ub UbT (A u) UbT A Ub λ 0T ! = . 0 UbT A Ub

T (n−1)×(n−1) I Matrix Ub A Ub ∈ R is symmetric. Proof: I λ 0T ! UT AU = . 0 UbT A Ub

I By induction, there exist diagonal matrix Db and orthogonal matrix Qb ∈ R(n−1)×(n−1), UbT A Ub = Qb Db QbT .

I therefore λ 0T UT AU = . 0 Qb Db QbT T 1 λ 1 A = U U def= QDQT . Qb Db Qb

n×n Thm: A matrix A ∈ R is symmetric if and only if there exists a n×n diagonal matrix D ∈ R and an orthogonal matrix Q so     T   T that A = QDQ = Q   Q .   n×n Thm: A matrix A ∈ R is symmetric if and only if there exists a n×n diagonal matrix D ∈ R and an orthogonal matrix Q so     T   T that A = QDQ = Q   Q .  

Proof: I λ 0T ! UT AU = . 0 UbT A Ub

I By induction, there exist diagonal matrix Db and orthogonal matrix Qb ∈ R(n−1)×(n−1), UbT A Ub = Qb Db QbT .

I therefore λ 0T UT AU = . 0 Qb Db QbT T 1 λ 1 A = U U def= QDQT . Qb Db Qb n×n Proof: I Let the diagonal matrix D ∈ R and an orthogonal matrix Q be so that A = QDQT . I D = diag (λ1, λ2, ··· , λn). λ1, λ2, ··· , λn eigenvalues of A.

A is positive deﬁnite ⇐⇒ xT A x > 0 for any non-zero x T ⇐⇒ QT x D QT x > 0 for any non-zero x ⇐⇒ yT D y > 0 for any non-zero y ⇐⇒ diagonal entries of D are positive.

n×n Thm: Let matrix A ∈ R be symmetric. Then A is positive deﬁnite if and only if all eigenvalues of A are positive. A is positive deﬁnite ⇐⇒ xT A x > 0 for any non-zero x T ⇐⇒ QT x D QT x > 0 for any non-zero x ⇐⇒ yT D y > 0 for any non-zero y ⇐⇒ diagonal entries of D are positive.

n×n Thm: Let matrix A ∈ R be symmetric. Then A is positive deﬁnite if and only if all eigenvalues of A are positive. n×n Proof: I Let the diagonal matrix D ∈ R and an orthogonal matrix Q be so that A = QDQT . I D = diag (λ1, λ2, ··· , λn). λ1, λ2, ··· , λn eigenvalues of A. T ⇐⇒ QT x D QT x > 0 for any non-zero x ⇐⇒ yT D y > 0 for any non-zero y ⇐⇒ diagonal entries of D are positive.

A is positive deﬁnite ⇐⇒ xT A x > 0 for any non-zero x ⇐⇒ yT D y > 0 for any non-zero y ⇐⇒ diagonal entries of D are positive.

A is positive deﬁnite ⇐⇒ xT A x > 0 for any non-zero x T ⇐⇒ QT x D QT x > 0 for any non-zero x ⇐⇒ diagonal entries of D are positive.

A is positive deﬁnite ⇐⇒ xT A x > 0 for any non-zero x T ⇐⇒ QT x D QT x > 0 for any non-zero x ⇐⇒ yT D y > 0 for any non-zero y n×n Thm: Let matrix A ∈ R be symmetric. Then A is positive deﬁnite if and only if all eigenvalues of A are positive. n×n Proof: I Let the diagonal matrix D ∈ R and an orthogonal matrix Q be so that A = QDQT . I D = diag (λ1, λ2, ··· , λn). λ1, λ2, ··· , λn eigenvalues of A.

A is positive deﬁnite ⇐⇒ xT A x > 0 for any non-zero x T ⇐⇒ QT x D QT x > 0 for any non-zero x ⇐⇒ yT D y > 0 for any non-zero y ⇐⇒ diagonal entries of D are positive. §9.3 The Power Method for Google PageRank (I)

I The PageRank Principle: The importance of each Webpage is proportional to the total size of the other Webpages which are pointing to it. §9.3 The Power Method for Google PageRank (II)

I random surf with jump: A Websurfer surfs the next Webpage

I either jumping to a page chosen at random from the entire Web at 15% likelihood, I or choosing a random link from the Webpage at 85% likelihood. §9.3 The Power Method for Google PageRank (III)

I Google Matrix G: each row/column represents a webpage, each G entry models Web connectivity and Web user surf patterns,

I PageRank vector x is eigenvector for G:

G x = 1 · x,

where 1 is always a simple eigenvalue of G. (0) I Power Method for iteratively computing x, given x ,

x(k+1) = G x(k), k = 0, 1, ··· , Task: Compute λ1 and corresponding eigenvector v1.

Despite condition on λ1, PM usually ﬁrst method to try.

The Power Method, in general

n×n Given: Matrix A ∈ R , with n eigenvalues

|λ1| > |λ2| ≥ |λ3| ≥ · · · ≥ |λn| .

(A has precisely one eigenvalue, λ1, that is largest in magnitude.) Despite condition on λ1, PM usually ﬁrst method to try.

The Power Method, in general

n×n Given: Matrix A ∈ R , with n eigenvalues

|λ1| > |λ2| ≥ |λ3| ≥ · · · ≥ |λn| .

(A has precisely one eigenvalue, λ1, that is largest in magnitude.)

Task: Compute λ1 and corresponding eigenvector v1. The Power Method, in general

n×n Given: Matrix A ∈ R , with n eigenvalues

|λ1| > |λ2| ≥ |λ3| ≥ · · · ≥ |λn| .

(A has precisely one eigenvalue, λ1, that is largest in magnitude.)

Task: Compute λ1 and corresponding eigenvector v1.

Despite condition on λ1, PM usually ﬁrst method to try. I For any k > 0 n k X k A x = βj λj vj j=1   n k k X λj βj = β1λ1 v1 + vj  λ1 β1 j=2 k !! k λ2 = β1λ1 v1 + O λ1

k A x points to the direction of v1 for large k.

The Power Method

I Assume v1, v2, ··· , vn are eigenvectors pertaining to λ1, λ2, ··· , λn. I Given initial vector x 6= 0. Then n X x = βj vj j=1

for some coeﬃcients β1, ··· , βn. Assume β1 6= 0. k A x points to the direction of v1 for large k.

The Power Method

I Assume v1, v2, ··· , vn are eigenvectors pertaining to λ1, λ2, ··· , λn. I Given initial vector x 6= 0. Then n X x = βj vj j=1

for some coeﬃcients β1, ··· , βn. Assume β1 6= 0. I For any k > 0 n k X k A x = βj λj vj j=1   n k k X λj βj = β1λ1 v1 + vj  λ1 β1 j=2 k !! k λ2 = β1λ1 v1 + O λ1 The Power Method

I Assume v1, v2, ··· , vn are eigenvectors pertaining to λ1, λ2, ··· , λn. I Given initial vector x 6= 0. Then n X x = βj vj j=1

for some coeﬃcients β1, ··· , βn. Assume β1 6= 0. I For any k > 0 n k X k A x = βj λj vj j=1   n k k X λj βj = β1λ1 v1 + vj  λ1 β1 j=2 k !! k λ2 = β1λ1 v1 + O λ1

k A x points to the direction of v1 for large k. Task: Find approximate eigenvalue λ. LS for λ: Choose λ in LS sense

minλ kA x − λ xk2 .

LS Solution: xT (A x) λ = xT x

Rayleigh quotient

Given: Approximate eigenvector x. Rayleigh quotient

Given: Approximate eigenvector x. Task: Find approximate eigenvalue λ. LS for λ: Choose λ in LS sense

minλ kA x − λ xk2 .

LS Solution: xT (A x) λ = xT x Algorithm 1 The Power Method n×n Input: Matrix A ∈ R , (0) n initial guess x ∈ R , and tolerance τ > 0. Output: Approximate eigenvalue λ, eigenvector x. Algorithm: (0) (0) (0) (0) (0) Normalize: x = x x 2 , y = A x , k = 0. T λ = x(0) y(0). (k) (k) while y − λ x 2 ≥ τ do (k+1) (k) (k) (k+1) (k+1) x = y y 2 , y = A x . T λ = x(k+1) y(k+1). k = k + 1. end while Algorithm 2 The Symmetric Power Method n×n Input: Symmetric matrix A ∈ R , (0) n initial guess x ∈ R , and tolerance τ > 0. Output: Approximate eigenvalue λ, eigenvector x. Algorithm: (0) (0) (0) (0) (0) Normalize: x = x x 2 , y = A x , k = 0. T λ = x(0) y(0). (k) (k) while y − λ x 2 ≥ τ do (k+1) (k) (k) (k+1) (k+1) x = y y 2 , y = A x . T λ = x(k+1) y(k+1). k = k + 1. end while

Same PM, but Symmetric PM converges much faster.  −4 14 0   1  0) I Ex 1: A =  −5 13 0  with x =  1  for λ1 = 6. −1 0 2 1  −4 −1 1   1  0) I Ex 2: A =  −1 3 −2  with x =  0  for λ1 = 6. 1 −2 3 0 Proof: Let v1, v2, ··· , vn form an orthonormal set of A eigenvectors associated with eigenvalues λ1, λ2, ··· , λn. Then the matrix def Q = (v1, v2, ··· , vn) is orthogonal, and

x = β1 v1 + β2 v2 + ··· + βn vn   β1  .  def T with  .  = Q x unit vector. βn

kA x − λ xk2 = kβ1 (λ1 − λ) v1 + ··· + βn (λn − λ) vnk2 q 2 2 2 2 = β1 (λ1 − λ) + ··· + βn (λn − λ) q 2 2 ≥ (min1≤j≤n |λ − λj |) β1 + ··· + βn = min1≤j≤n |λ − λj |

n×n Thm: Let A ∈ R is symmetric with eigenvalues λ1, λ2, ··· , λn. If we have kA x − λ xk2 ≤ τ for some real number λ and unit vector x, then min1≤j≤n |λ − λj | ≤ τ. kA x − λ xk2 = kβ1 (λ1 − λ) v1 + ··· + βn (λn − λ) vnk2 q 2 2 2 2 = β1 (λ1 − λ) + ··· + βn (λn − λ) q 2 2 ≥ (min1≤j≤n |λ − λj |) β1 + ··· + βn = min1≤j≤n |λ − λj |

n×n Thm: Let A ∈ R is symmetric with eigenvalues λ1, λ2, ··· , λn. If we have kA x − λ xk2 ≤ τ for some real number λ and unit vector x, then min1≤j≤n |λ − λj | ≤ τ.

Proof: Let v1, v2, ··· , vn form an orthonormal set of A eigenvectors associated with eigenvalues λ1, λ2, ··· , λn. Then the matrix def Q = (v1, v2, ··· , vn) is orthogonal, and

x = β1 v1 + β2 v2 + ··· + βn vn   β1  .  def T with  .  = Q x unit vector. βn = min1≤j≤n |λ − λj |

n×n Thm: Let A ∈ R is symmetric with eigenvalues λ1, λ2, ··· , λn. If we have kA x − λ xk2 ≤ τ for some real number λ and unit vector x, then min1≤j≤n |λ − λj | ≤ τ.

Proof: Let v1, v2, ··· , vn form an orthonormal set of A eigenvectors associated with eigenvalues λ1, λ2, ··· , λn. Then the matrix def Q = (v1, v2, ··· , vn) is orthogonal, and

x = β1 v1 + β2 v2 + ··· + βn vn   β1  .  def T with  .  = Q x unit vector. βn kA x − λ xk2 = kβ1 (λ1 − λ) v1 + ··· + βn (λn − λ) vnk2 q 2 2 2 2 = β1 (λ1 − λ) + ··· + βn (λn − λ) q 2 2 ≥ (min1≤j≤n |λ − λj |) β1 + ··· + βn n×n Thm: Let A ∈ R is symmetric with eigenvalues λ1, λ2, ··· , λn. If we have kA x − λ xk2 ≤ τ for some real number λ and unit vector x, then min1≤j≤n |λ − λj | ≤ τ.

Proof: Let v1, v2, ··· , vn form an orthonormal set of A eigenvectors associated with eigenvalues λ1, λ2, ··· , λn. Then the matrix def Q = (v1, v2, ··· , vn) is orthogonal, and

x = β1 v1 + β2 v2 + ··· + βn vn   β1  .  def T with  .  = Q x unit vector. βn kA x − λ xk2 = kβ1 (λ1 − λ) v1 + ··· + βn (λn − λ) vnk2 q 2 2 2 2 = β1 (λ1 − λ) + ··· + βn (λn − λ) q 2 2 ≥ (min1≤j≤n |λ − λj |) β1 + ··· + βn = min1≤j≤n |λ − λj | Apply Power Method to (A − q I )−1.

The Inverse Power Method (I)

n×n Given: Matrix A ∈ R , with n eigenvalues λ1, λ2, ··· , λn; and given shift q.

Task: Compute λi that is closest to q, and corresponding eigenvector vi . The Inverse Power Method (I)

n×n Given: Matrix A ∈ R , with n eigenvalues λ1, λ2, ··· , λn; and given shift q.

Task: Compute λi that is closest to q, and corresponding eigenvector vi .

Apply Power Method to (A − q I )−1. I Assume q closest to λi and λk , but closer to λi . k λi − q I IPM converges to λi at order . λk − q

−1 I Matrix (A − q I ) has eigenvalues 1 1 1 , , ··· , . λ1 − q λ2 − q λn − q k λi − q I IPM converges to λi at order . λk − q

−1 I Matrix (A − q I ) has eigenvalues 1 1 1 , , ··· , . λ1 − q λ2 − q λn − q

I Assume q closest to λi and λk , but closer to λi . −1 I Matrix (A − q I ) has eigenvalues 1 1 1 , , ··· , . λ1 − q λ2 − q λn − q

I Assume q closest to λi and λk , but closer to λi . k λi − q I IPM converges to λi at order . λk − q Algorithm 3 The Inverse Power Method n×n Input: Matrix A ∈ R , shift q, (0) n initial guess x ∈ R , and tolerance τ > 0. Output: Approximate eigenvalue λ, eigenvector x. Algorithm: (0) (0) (0) (0) −1 (0) Normalize: x = x x 2 , y = (A − q I ) x . T λ = x(0) y(0), k = 0. (k) (k) while y − λ x 2 ≥ τ do (k+1) (k) (k) (k+1) −1 (k+1) x = y y 2 , y = (A − q I ) x . T λ = x(k+1) y(k+1). k = k + 1. end while I Symmetric/Non-symmetric PM Errors

I Symmetric IPM Errors Proof: I Let λ be eigenvalue of A with unit eigenvector u: A u = λ u. n I We extend u into an orthonormal basis for R : u, u2, ··· , un are unit, mutually orthogonal vectors.

def def n×n I U = (u, u2, ··· , un) = u, Ub ∈ R is orthogonal.   T uT (A u) uT A U T u b U AU = T A u, A Ub =   Ub UbT (A u) UbT A Ub λ 0T ! = . 0 UbT A Ub

T (n−1)×(n−1) I Repeat on symmetric matrix Ub A Ub ∈ R .

Review n×n Thm: A matrix A ∈ R is symmetric if and only if there exists a n diagonal matrix D ∈ R and an orthogonal matrix Q so that     T   T A = QDQ = Q   Q .   Review n×n Thm: A matrix A ∈ R is symmetric if and only if there exists a n diagonal matrix D ∈ R and an orthogonal matrix Q so that     T   T A = QDQ = Q   Q .  

Proof: I Let λ be eigenvalue of A with unit eigenvector u: A u = λ u. n I We extend u into an orthonormal basis for R : u, u2, ··· , un are unit, mutually orthogonal vectors.

def def n×n I U = (u, u2, ··· , un) = u, Ub ∈ R is orthogonal.   T uT (A u) uT A U T u b U AU = T A u, A Ub =   Ub UbT (A u) UbT A Ub λ 0T ! = . 0 UbT A Ub

T (n−1)×(n−1) I Repeat on symmetric matrix Ub A Ub ∈ R . I Compute one approximate eigenvalue λ of A with unit eigenvector u: A u = λ u. n I Extend u into an orthonormal basis for R : u, u2, ··· , un are unit, mutually orthogonal vectors.

def def n×n U = (u, u2, ··· , un) = u, Ub ∈ R is orthogonal. I   T uT (A u) uT A U T u b U AU = T A u, A Ub =   Ub UbT (A u) UbT A Ub   λ uT A Ub =   . (Deflation) 0 UbT A Ub

def T (n−1)×(n−1) I Continue on matrix Ab = Ub A Ub ∈ R .

Computing all eigenvalues of matrix A ∈ Rn×n Computing all eigenvalues of matrix A ∈ Rn×n

I Compute one approximate eigenvalue λ of A with unit eigenvector u: A u = λ u. n I Extend u into an orthonormal basis for R : u, u2, ··· , un are unit, mutually orthogonal vectors.

def T (n−1)×(n−1) I Continue on matrix Ab = Ub A Ub ∈ R . Householder Reflection n Let v ∈ R be a unit vector. Define Householder Reflection matrix T n×n H = I − 2v v ∈ R .

I H is symmetric and orthogonal H = HT , H2 = I − 4v vT + 4v vT = I .

n † def T I For any vector x ∈ R , x = H x = x − 2v v x reflects x in the direction v⊥: Deflation with Householder Reflection (I)

I Given eigenvalue λ of A with unit eigenvector u: A u = λ u.

I Extend u into an orthonormal basis with a Householder reﬂection

T def def U = I − 2v v = (u, u2, ··· , un) = u, Ub I   λ uT A U T b U AU =   . 0 UbT A Ub

Find unit vector v so first column of I − 2v vT is u. Deflation with Householder Reflection (II)

I Partition µ ν u = , v = . ub bv T I First column of I − 2v v is µ 1 ν = − 2 ν. ub 0 bv I If µ ≤ 0, then r1 − µ u ν = , v = − b , U = I − 2v vT = u, Ub . (1) 2 b 2 ν

I If µ > 0, then −u is also unit eigenvector. Compute v with equation (1) on −u: r1 + µ u ν = , v = b , U = I − 2v vT = −u, Ub . (2) 2 b 2 ν Equations (1) and (2) ensure numerical stability Householder Reflection (II) n Let v ∈ R be a unit vector. Define Householder Reflection matrix

T n×n H = I − 2 v v ∈ R .

n I For any vector x ∈ R , choose v so that ± kxk H x = 2 , (sign to be chosen for numerical stability.) 0

I ξ ± kxk ξ Partition x = , 2 = H x = − 2 v vT x, bx 0 bx

± kxk − ξ sign (ξ)(kxk + |ξ|) u def= 2 =====choose − 2 −bx bx

and v = u /kuk2