The Poisson matrix and Kronecker Products

Tom Lyche

University of Oslo Norway

The Poisson matrix and Kronecker Products – p. 1/27 Plan for the day

Numerical solution of the Poisson problem The finite difference scheme Formulation as a matrix equation Formulation in standard form Ax = b. The Poisson matrix A is block tridiagonal LU factorization of a block tridiagonal matrix Solving the Poisson problem, complexity Kronecker product Relation to Poisson problem Basic properties Eigenvalues and eigenvectors Eigenpairs for the Poisson matrix

The Poisson matrix and Kronecker Products – p. 2/27 Numerical solution of the Poisson Problem

u=0

u=0 −(u +u ) =f u=0 xx yy

u=0 m N, h = 1/(m + 1), vj,k u(jh, kh) ∈ ≈ (m+1)h j,k+1

(jh,kh) kh j-1,k j,k j+1,k

0 j,k-1

0 jh (m+1)h

The Poisson matrix and Kronecker Products – p. 3/27 Discrete Equation

′′ g(t h) 2g(t)+ g(t + h) g (t) − − ≈ h2

v j,k+1

v v j-1,k vj,k j+1,k

v j,k-1 From differential equation for j,k = 1, 2,...,m

2 ( vj−1,k + 2vj,k vj+1,k)+( vj,k−1 + 2vj,k vj,k+1)= h fj,k (1) − − − − From boundary conditions

vj,k = 0 if j = 0, m + 1 or k = 0, m + 1 (2)

The Poisson matrix and Kronecker Products – p. 4/27 Matrix Equation T V + V T = h2F

2 1 v11 v12 v11 v12 2 1 1 f11 f12  −    +    −  =   1 2 v21 v22 v21 v22 1 2 9 f21 f22 −      −    m,m T := T m = tridiag ( 1, 2, 1) R m − − ∈

v11 v1m f11 f1m  ···   ···  . . Rm,m . . Rm,m V :=  . .  F :=  . .    ∈   ∈     vm1 vmm fm1 fmm  ···   ···  2 (1), (2) TV + VT = h F ⇔ m m 2 TV + VT = T jivik + vjiT ik = l.h.s in (1) = h (F )jk
jk X X i=1 i=1

The Poisson matrix and Kronecker Products – p. 5/27 Convert T V + V T = h2F to Ax = b

1,3 2,3 3,3 7 8 9

1,2 2,2 3,2 > 4 5 6

1,1 2,1 3,1 1 2 3

v in grid x in grid j,k i

-v j,k+1 -x i+m

- 4 -v -x -x v j-1,k vj,k j+1,k i-1 4x i i+1 -->

-v j,k-1 -xi-m

4xi xi−1 xi+1 xi−m xi+m = bi − − − −

The Poisson matrix and Kronecker Products – p. 6/27 Linear system Ax = b

For m = 3 or n = 9 we have the following linear system:

2 4x1 x2 x4 = h f11 − − 2 x1 4x2 x3 x5 = h f21 − − − 2 x2 4x3 x6 = h f31 − − 2 x1 4x4 x5 x7 = h f12 − − − 2 x2 x4 4x5 x6 x8 = h f22 − − − − 2 x3 x5 4x6 x9 = h f32 − − − 2 x4 4x7 x8 = h f13 − − 2 x5 x7 4x8 x9 = h f23 − − − 2 x6 x8 4x9 = h f33 − −

The Poisson matrix and Kronecker Products – p. 7/27 Blockstructure of A

4 1 0 1 0 0 0 0 0  − −  1 4 1 0 1 0 0 0 0    − − −   0 1 4 0 0 1 0 0 0   − −     1 0 0 4 1 0 1 0 0   − − −    A =  0 1 0 1 4 1 0 1 0   − − − −     0 0 1 0 1 4 0 0 1   − − −     0 0 0 1 0 0 4 1 0   − −   0 0 0 0 1 0 1 4 1     − − −   0 0 0 0 0 1 0 1 4  − − T + 2I I 0  −  A = I T + 2I I    − −   0 I T + 2I  − Block tridiagonal The Poisson matrix and Kronecker Products – p. 8/27 LU-factorization of a tridiagonal matrix

d1 c1   1  u1 c1  a2 d2 c2   .. ..  . . .  l2 1   . .   ......  =     .    .. ..       . .   un−1 cn−1  a −1 d −1 c −1      n n n         ln 1  un   an dn  (3) Note that L has ones on the diagonal, and that A and U have the same super-diagonal. By equating entries in (3) we find

d1 = u1, ak = lkuk−1, dk = lkck−1 + uk, k = 2, 3,...,n.

The Poisson matrix and Kronecker Products – p. 9/27 The Algorithm

A = LU (LU-factorization) Ly = b (forward substitution) Ux = y (backward substituion)

ak u1 = d1, lk = , uk = dk lkck−1, k = 2, 3,...,n. uk−1 − y1 = b1, yk = bk lkyk−1, k = 2, 3,...,n, − xn = yn/dn, xk =(yk ckxk+1)/uk, k = n 1,..., 2, 1. − − This process is well defined if A is strictly diagonally dominant. The number of arithmetic operations (flops) is O(n).

The Poisson matrix and Kronecker Products – p. 10/27 LU of Block Tridiagonal Matrix

D1 C1 1 U 1 C1 A2 D2 C2   L2 1 . . .    .. ..  ...... = ...... U m−1 Cm−1  Am−1 Dm−1 Cm−1   Lm 1   U m   Am Dm 

−1 U 1 = D1, Lk = AkU − , U k = Dk LkCk−1, k = 2, 3,...,m. (4) k 1 − To solve the system Ax = b we partition b conformally with A in the form T T T b =[b1 ,..., bm]. The formulas for solving Ly = b and Ux = y are as follows:

y = b1, y = bk Lky − , k = 2, 3,...,m, 1 k − k 1 −1 −1 (5) xm = U y , xk = U (y Ckxk+1), k = m 1,..., 2, 1. m m k k − −

T T T The solution is then x =[x1 ,..., xm].

The Poisson matrix and Kronecker Products – p. 11/27 Poisson Problem

The discrete Poisson problem can be solved writing it as a block tridiagonal matrix. We can also use Cholesky factorization of a band matrix. The matrix has bandwidth d = √n. Cholesky requires O(2nd2)= O(2n2) flops. The work using block tridiagonal solver is also O(n2). Need better methods (next time).

The Poisson matrix and Kronecker Products – p. 12/27 Kronecker Product

Definition 1. For any positive integers p,q,r,s we define the Kronecker product of two matrices A Rp,q and B Rr,s as a matrix C Rpr,qs given in block form as ∈ ∈ ∈

Ab1,1 Ab1,2 Ab1,s  ···  Ab2,1 Ab2,2 Ab2,s  ···  C =  . . .  .  . . .   . . .     Abr,1 Abr,2 Abr,s  ··· We denote the Kronecker product of A and B by C = A B. ⊗ Defined for rectangular matrices of any dimension # of rows(columns) = product of # of rows(columns) in A and B

The Poisson matrix and Kronecker Products – p. 13/27 Kronecker Product Example

2 1 1 0 T =  −  , I =   1 2 0 1 −    2 1 0 0  −  T 0 1 2 0 0     T I = =  −  ⊗ 0 T  0 0 2 1       −   0 0 1 2  − 2 0 1 0  −  2I I 0 2 0 1     I T = − =  −  ⊗ I 2I  1 0 2 0   −     −   0 1 0 2  −

The Poisson matrix and Kronecker Products – p. 14/27 Poisson Matrix = T I + I T ⊗ ⊗ T 2I I 0    −  A = T + I 2I I = T I + I T     ⊗ ⊗    − −   T   0 I 2I  − r,r s,s Definition 2. Let for positive integers r,s,k, A R , B R and Ik be the ∈ ∈ identity matrix of order k. The sum A Is + Ir B is known as the Kronecker sum of ⊗ ⊗ A and B. The Poisson matrix is the Kronecker sum of T with itself.

The Poisson matrix and Kronecker Products – p. 15/27 Matrix equation Kronecker ↔ Given A Rr,r, B Rs,s, F Rr,s. Find V Rr,s such that ∈ ∈ ∈ ∈ AV BT = F

For B Rm,n define ∈

b1   b1j 

b2 b2j   mn   vec(B) :=   R , bj =   jth column  .  ∈  .   .   .          bn bmj 

Lemma 1.

AV BT = F (A B) vec(V ) = vec(F ) (6) ⇔ ⊗ T AV + V B = F (A Is + Ir B) vec(V ) = vec(F ). (7) ⇔ ⊗ ⊗

The Poisson matrix and Kronecker Products – p. 16/27 Proof

T We partition V , F , and B by columns as V =(v1,..., vs), T F =(f 1,..., f s) and B =(b1,..., bs). Then we have

(A B) vec(V ) = vec(F ) ⊗

Ab11 Ab1s v1 f1  ···      . . . .  . .   .  =  .  ⇔             Abs1 Abss vs fs  ···     

A bij vj = fi, i = 1,...,m ⇔ X j T [AV b1,..., AV bs]= F AV B = F . ⇔ ⇔

This proves (6)

The Poisson matrix and Kronecker Products – p. 17/27 Proof Continued

(A Is + Ir B) vec(V ) = vec(F ) ⊗ ⊗ T T T (AV I + IrV B )= F AV + V B = F . ⇔ s ⇔  This gives a slick way to derive the Poisson matrix. Recall

T AV + V B = F (A Is + Ir B) vec(V ) = vec(F ) ⇔ ⊗ ⊗ Therefore,

TV + VT = h2F (T I + I T ) vec(V )= h2 vec(F ) ⇔ ⊗ ⊗

The Poisson matrix and Kronecker Products – p. 18/27 Mixed Product Rule

(A B)(C D)=(AC) (BD) ⊗ ⊗ ⊗ Proof If B Rr,t and D Rt,s for some integers r, t then ∈ ∈

Ab1,1 Ab1,t Cd1,1 Cd1,s E1,1 E1,s  ···   ···   ···  ......  . .   . .  =  . .  ,             Abr,1 Abr,t Cdt,1 Cdt,s Er,1 Er,s  ···   ···   ···  where for all i, j

t Ei,j = bi,kdk,jAC =(AC)(BD)i,j = ((AC) (BD))i,j. X ⊗ k=1 where in the last formula i, j refers to the ij-block in the Kronecker product. 

The Poisson matrix and Kronecker Products – p. 19/27 Properties of Kronecker Products

The usual arithmetic rules hold. Note however that (A B)T = AT BT ⊗ ⊗ − − (A B)−1 = A 1 B 1 if A and B are nonsingular ⊗ ⊗ A B = B A ⊗ 6 ⊗ (A B)(C D)=(AC) (BD) (mixed product rule) ⊗ ⊗ ⊗ The mixed product rule is valid for any matrices as long as the products AC and BD are defined.

The Poisson matrix and Kronecker Products – p. 20/27 Eigenvalues and Eigenvectors

The Kronecker product of two vectors u Rp and v Rr is a vector pr T ∈ T T ∈ r,r u v R given by u v = u v1,..., u vr Suppose now A R ⊗ ∈ ⊗   ∈ and B Rs,s and ∈

Aui = λiui, i = 1,...,r, Bvj = µj vj, j = 1,...,s, then for i = 1,...,r, j = 1,...,s

(A B)(ui vj )= λiµj(ui vj ), (8) ⊗ ⊗ ⊗ (A Is + Ir B)(ui vj )=(λi + µj)(ui vj). (9) ⊗ ⊗ ⊗ ⊗ Thus the eigenvalues of a Kronecker product(sum) are the products (sums) of the eigenvalues of the factors. The eigenvectors of a Kronecker product(sum) are the products of the eigenvectors of the factors.

The Poisson matrix and Kronecker Products – p. 21/27 Proof of Eigen-formulae

This follows directly from the mixed product rule. For (8)

(A B)(ui vj )=(Aui) (Bvj )=(λiui) (µjvj )=(λiµj)(ui vj ). ⊗ ⊗ ⊗ ⊗ ⊗

From (8)

(A Is)(ui vj )= λi(ui vj ), and (Ir B)(ui vj )= µj (ui vj) ⊗ ⊗ ⊗ ⊗ ⊗ ⊗ The result now follows by summing these relations. 

The Poisson matrix and Kronecker Products – p. 22/27 The Poisson Matrix A = T I + I T ⊗ ⊗ Recall eigenvalues and eigenvectors of the T matrix. Let h = 1/(m + 1).

1. We know that T sj = λj sj for j = 1,...,m, where

T sj = (sin(jπh), sin(2jπh),..., sin(mjπh)) , (10)

2 jπh λj = 2 2cos(jπh) =4sin ( ). (11) − 2 2. The eigenvalues are distinct and the eigenvectors are orthogonal

1 sT s = δ , j,k = 1,...,m. (12) j k 2h j,k

The Poisson matrix and Kronecker Products – p. 23/27 Eigenvalues/-vectors A = T I + I T ⊗ ⊗

1. We have Axj,k = λj,kxj,k for j,k = 1,...,m, where

xj,k = sj sk, (13) ⊗ T sj = (sin(jπh), sin(2jπh),..., sin(mjπh)) , (14)

2 jπh 2 kπh λ = 4sin ( )+4sin ( ). (15) j,k 2 2 2. The eigenvectors are orthogonal

1 xT x = δ δ , j,k,p,q = 1,...,m. (16) j,k p,q 4h2 j,p k,q

3. A is symmetric AT =(T I + I T )T = T T IT + IT T T = A ⊗ ⊗ ⊗ ⊗ 4. A is positive definite (positive eigenvalues)

The Poisson matrix and Kronecker Products – p. 24/27 A = T I + I T for m =2 ⊗ ⊗ π 2π  sin( 3 )  √3 1 sin( 3 ) √3  1  s1 = = , s2 = = sin( 2π ) 2 1 sin( 4π ) 2 1  3     3  − 

2 π 2 2π λ1 = 4sin ( ) = 1, λ2 = 4sin ( ) = 3 6 6

Axij = µijxij, µij = λi + λj , xij = si sj i, j = 1, 2 ⊗

µ11 = 2, µ12 = µ21 = 4, µ22 = 6.

3 T 3 T s1 s1 = [1, 1, 1, 1] , s1 s2 = [1, 1, 1, 1] , ⊗ 4 ⊗ 4 − − 3 T 3 T s2 s1 = [1, 1, 1, 1] , s2 s2 = [1, 1, 1, 1] ⊗ 4 − − ⊗ 4 − − 9 xT x = δ δ , j,k,p,q = 1, 2. j,k p,q 4 j,p k,q

The Poisson matrix and Kronecker Products – p. 25/27 Check

2 1 1 0 1 0 2 1 A =  −    +    −  1 2 ⊗ 0 1 0 1 ⊗ 1 2 −      −  4 1 1 0 1 1  − −      1 4 0 1 1 1        − −    = 2   ,  1 0 4 1  1 1  − −             0 1 1 4  1 1 − − 4 1 1 0 1 1  − −      1 4 0 1 1 1        − −    = 4    1 0 4 1   1  1  − −  −  −         0 1 1 4   1  1 − − − − etc.

The Poisson matrix and Kronecker Products – p. 26/27 Summary

Studied the Poisson matrix A Shown that A can be written as a Kronecker sum studied properties of general Kronecker products and Kronecker sums used this to derive properties of A

The Poisson matrix and Kronecker Products – p. 27/27