Digital Image Processing

Filtering in the Frequency Domain (Circulant Matrices and Convolution)

Christophoros Nikou [email protected]

University of Ioannina - Department of Computer Science and Engineering 2 Toeplitz matrices

• Elements with constant value along the main diagonal and sub-diagonals. • For a NxN matrix, its elements are determined by

a (2N-1)-length sequence tn | (N  1)  n  N  1 T(,)m n t mn t0 t 1 t 2  t(N 1) t t t  1 0 1

T  tt22         t1 t t t t N 1 2 1 0 NN

C. Nikou – Digital Image Processing (E12) 3 Toeplitz matrices (cont.)

• Each row (column) is generated by a shift of the previous row (column). − The last element disappears. − A new element appears.

T(,)m n t mn t0 t 1 t 2  t(N 1) t t t  1 0 1

T  tt22         t1 t t t t N 1 2 1 0 NN

C. Nikou – Digital Image Processing (E12) 4 Circulant matrices

• Elements with constant value along the main diagonal and sub-diagonals. • For a NxN matrix, its elements are determined by

a N-length sequence cn | 01nN   

C(,)m n c(m n )mod N c0 cNN 1 c 2  c1 c c c  1 01N 

C  c21 c c02 cN       cN 1 c c c c NN1 21 0 NN

C. Nikou – Digital Image Processing (E12) 5 Circulant matrices (cont.)

• Special case of a Toeplitz matrix. • Each row (column) is generated by a circular shift (modulo N) of the previous row (column).

C(,)m n c(m n )mod N

c0 cNN 1 c 2  c1 c c c  1 01N 

C  c21 c c02 cN       cN 1 c c c c NN1 21 0 NN C. Nikou – Digital Image Processing (E12) Convolution by matrix-vector 6 operations • 1-D linear convolution between two discrete signals may be expressed as the product of a Toeplitz matrix constructed by the elements of one of the signals and a vector constructed by the elements of the other signal. • 1-D circular convolution between two discrete signals may be expressed as the product of a circulant matrix constructed by the elements of one of the signals and a vector constructed by the elements of the other signal. • Extension to 2D signals.

C. Nikou – Digital Image Processing (E12) 1D linear convolution using Toeplitz 7 matrices

f[ n ] {1, 2, 2}, h [ n ]  {1,  1}, N12  3, N  2 • The linear convolution g[n]=f [n]*h [n] will be of

length N=N1+N2-1=3+2-1=4. • We create a Toeplitz matrix H from the elements of h [n] (zero-padded if needed) with − N=4 lines (the length of the result).

− N1=3 columns (the length of f [n]). − The two signals may be interchanged.

C. Nikou – Digital Image Processing (E12) 1D linear convolution using Toeplitz 8 matrices (cont.)

f[ n ] {1, 2, 2}, h [ n ]  {1,  1}, N12  3, N  2

1 0 0 1 1 0 H  Length of the result =4 0 1 1  0 0 1 43 Notice that H is not circulant (e.g. a -1 appears in the second line Length of f [n] = 3 which is not present in the first line. Zero-padded h[n] in the first column C. Nikou – Digital Image Processing (E12) 1D linear convolution using Toeplitz 9 matrices (cont.)

f[ n ] {1, 2, 2}, h [ n ]  {1,  1}, N12  3, N  2

1 0 0   1  1 1 1 0   1  g = Hf  2   0 1 1   0   2   0 0 1   2 

gn[ ] {1, 1, 0, 2}

C. Nikou – Digital Image Processing (E12) 1D circular convolution using 10 circulant matrices

f[ n ] {1, 2, 2}, h [ n ]  {1,  1}, N12  3, N  2

• The circular convolution g[n]=f [n]h [n] will

be of length N=max{N1, N2}=3. • We create a circulant matrix H from the elements of h [n] (zero-padded if needed) of size NxN. − The two signals may be interchanged.

C. Nikou – Digital Image Processing (E12) 1D circular convolution using 11 circulant matrices (cont.)

f[ n ] {1, 2, 2}, h [ n ]  {1,  1}, N12  3, N  2

1 0 1  H 1 1 0 0 1 1 33

Zero-padded h[n] in the first column

C. Nikou – Digital Image Processing (E12) 1D circular convolution using 12 circulant matrices (cont.)

f[ n ] {1, 2, 2}, h [ n ]  {1,  1}, N12  3, N  2

1 0 1   1   1        g = Hf  1 1 0   2    1  0 1 1   2   0 

gn[ ] {  1, 1, 0}

C. Nikou – Digital Image Processing (E12) 13 Block matrices

AAA11 12 1N AAA A  21 22 2N      AAAM12 M MN

• Aij are matrices. • If the structure of A, with respect to its sub-matrices, is Toeplitz (circulant) then matrix A is called block-Toeplitz (block-circulant).

• If each individual Aij is also a Toeplitz (circulant) matrix then A is called doubly block-Toeplitz (doubly block-circulant).

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 14 block Toeplitz matrices

m m 1 4 1 1 1 2 5 3 1 -1 n n f [m,n] h [m,n]

M1=2, N1=3 M2=2, N2=2

The result will be of size (M1+M2-1) x (N1+N2-1) = 3 x 4

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 15 block Toeplitz matrices (cont.) m m 1 4 1 1 1 2 5 3 1 -1 n n f [m,n] h [m,n]

• At first, h[m,n] is zero-padded to 3 x 4 (the size of the m result). 0 0 0 0 • Then, for each row of h[m,n], 1 1 0 0 a Toeplitz matrix with 3 1 -1 0 0 n columns (the number of h[m,n] columns of f [m,n]) is

constructed. C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 16 block Toeplitz matrices (cont.)

m • For each row of h[m,n], a 0 0 0 0 Toeplitz matrix with 3 columns 1 1 0 0 (the number of columns of f 1 -1 0 0 [m,n]) is constructed. n h[m,n]

1 0 0 1 0 0 000  1 1 0 1 1 0 000 H  H  H   1 0 1 1 2 0 1 1 3 000   0 0 1 0 0 1 000

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 17 block Toeplitz matrices (cont.) m m 1 4 1 1 1 2 5 3 1 -1 n n f [m,n] h [m,n]

• Using matrices H1, H2

and H3 as elements, a HH13 doubly block Toeplitz HHH  matrix H is then 21 HH constructed with 2 3212 6 columns (the number of rows of f [m,n]).

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 18 block Toeplitz matrices (cont.) m m 1 4 1 1 1 2 5 3 1 -1 n n f [m,n] h [m,n] 2  5 • We now construct 3 (2 5 3)T a vector from the f  T elements of f [m,n]. 1 (1 4 1) 4  1

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 19 block Toeplitz matrices (cont.) m m 1 4 1 1 1 2 5 3 1 -1 n n f [m,n] h [m,n] 2  5 HH13 3 g = Hf H21 H  1 HH 324  1 C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 20 block Toeplitz matrices (cont.) 1 0 0 0 0 0   2      1 1 0 0 0 0   3  0 1 1 0 0 0   2      0 0 1 0 0 0  2  3  1 0 0 1 0 0  5  3      (2 3 2 3)T 1 1 0 1 1 0  3  10  T g = Hf    (3 10 5 2) 0 1 1 0 1 1  1  5       (1 5 5 1)T 0 0 1 0 0 1  4  2    0 0 0 1 0 0  1  1      0 0 0 1 1 0   5      0 0 0 0 1 1   5  0 0 0 0 0 1   1  C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 21 block Toeplitz matrices (cont.) m m 1 4 1 1 1 2 5 3 * 1 -1 n n f [m,n] h [m,n]

m 1 5 5 1 (2 3 2 3)T T = 3 10 5 2 g=(3 10 5 2) 2 3 -2 3 T n (1 5 5 1) g [m,n]

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 22 block Toeplitz matrices (cont.) Another example

m m 3 4 1 2 1 -1 n n f [m,n] h [m,n]

M1=2, N1=2 M2=1, N2=2

The result will be of size (M1+M2-1) x (N1+N2-1) = 2 x 3

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 23 block Toeplitz matrices (cont.) m m 3 4 1 2 1 -1 n n f [m,n] h [m,n] • At first, h[m,n] is zero-padded to 2 x 3 (the size of the m result). • Then, for each line of h[m,n], 0 0 0 a Toeplitz matrix with 2 1 -1 0 n columns (the number of h[m,n] columns of f [m,n]) is

constructed. C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 24 block Toeplitz matrices (cont.)

m • For each row of h[m,n], a Toeplitz matrix with 2 columns (the 0 0 0 number of columns of f [m,n]) is 1 -1 0 constructed. n h[m,n]

10 00 H 11 H  00 1  2  01 00

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 25 block Toeplitz matrices (cont.) m m 3 4 1 2 1 -1 n n f [m,n] h [m,n]

• Using matrices H1 and H2 as elements, a HH doubly block Toeplitz H  12 matrix H is then  HH21 constructed with 2 64 columns (the number of rows of f [m,n]).

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 26 block Toeplitz matrices (cont.) m m 3 4 1 2 1 -1 n n f [m,n] h [m,n]

1  T • We now construct a 2 (1 2) f   vector from the elements T 3 (3 4) of f [m,n].  4

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 27 block Toeplitz matrices (cont.) m m 3 4 1 2 1 -1 n n f [m,n] h [m,n]

1 HH2 12 g = Hf   HH213  4

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 28 block Toeplitz matrices (cont.)

1 0 0 0   1      1 1 0 0  1  1      (1 1 2)T 0 1 0 0 2 2 g = Hf       T 0 0 1 0  3  3  (3 1 4)   0 0 1 1  4  1      0 0 0 1   4 

C. Nikou – Digital Image Processing (E12) 2D linear convolution using doubly 29 block Toeplitz matrices (cont.) m m 3 4 * 1 2 1 -1 n n f [m,n] h [m,n] m (1 1 2)T 3 1 4 = g=T 1 1 -2 (3 1 4) n  g [m,n]

C. Nikou – Digital Image Processing (E12) 2D circular convolution using doubly 30 block circulant matrices The circular convolution g[m,n]=f [m,n]h [m,n] with 0m  M  1, 0  n  N  1,

may be expressed in matrix-vector form as: g = Hf

where H is a doubly block circulant matrix generated by h [m,n] and f is a vectorized form of f [m,n].

C. Nikou – Digital Image Processing (E12) 2D circular convolution using doubly 31 block circulant matrices (cont.)

HHHH0MM 1 2 1 HHHH 1 0M  1 2

HHHHH 2 1 0 3        HHHHMMM1  2  3 0

Each Hj, for j=1,..M, is a circulant matrix with N columns (the number of columns of f [m,n]) generated from the elements of the j-th row of h [m,n]. C. Nikou – Digital Image Processing (E12) 2D circular convolution using doubly 32 block circulant matrices (cont.)

h[,0] j h [, j N 1] h [,1] j h[,1] j h [,0] j h [,2] j H   j      h[, j N 1] h [, j N 2] h [,0] j NN

Each Hj, for j=1,..M, is a NxN circulant matrix generated from the elements of the j-th row of h [m,n]. C. Nikou – Digital Image Processing (E12) 2D circular convolution using doubly 33 block circulant matrices (cont.)

m m 0 1 0 0 0 0 1 3 -1  1 0 0 1 2 1 1 -1 0 n n f [m,n] h [m,n]

1 0 1 1 0 0 000   H 1 1 0 H  0 1 0 H2  000 0 1   000 0 1 1 0 0 1 

C. Nikou – Digital Image Processing (E12) 2D circular convolution using doubly 34 block circulant matrices (cont.)

m m 0 1 0 0 0 0 1 3 -1  1 0 0 1 2 1 1 -1 0 n n f [m,n] h [m,n]

T HHH 1 2 1 0 2 1  g = Hf H H H 1 3 1 T 102  T HHH2 1 0 0 1 0  

C. Nikou – Digital Image Processing (E12) 2D circular convolution using doubly 35 block circulant matrices (cont.)

1010001001     0        1110000102     2  0100000011     1        1001010001     3  g = Hf 0101110003     4        0010100001       3  0001001010     1        0000101111     4       0000010100     2 

C. Nikou – Digital Image Processing (E12) 2D circular convolution using doubly 36 block circulant matrices (cont.)

m m 0 0 0 0 1 0 1 0 0 1 3 -1  1 -1 0 1 2 1 n n h [m,n] f [m,n]

m 0 2 1T  1 4 -2 T = 3 4 -3 g=3 4 3  0 2 -1 1 4 2 T n   g [m,n]

C. Nikou – Digital Image Processing (E12) 37 Diagonalization of circulant matrices

Theorem: The columns of the inverse DFT matrix are eigenvectors of any circulant matrix. The corresponding eigenvalues are the DFT values of the signal generating the circulant matrix. Proof: Let

22n j j k nk wNN e  w  e

be the DFT basis elements of length N with: 0k  N  1, 0  n  N  1,

C. Nikou – Digital Image Processing (E12) Diagonalization of circulant matrices 38 (cont.) We know that the DFT F [k] of a 1D signal f [n] may be expressed in matrix-vector form: F = Af where f = f[0], f [1],..., f [ N 1]TT , F =  F [0], F [1],..., F [ N 1]

0 1 2N  1 w0 w 0 w 0 w 0  NNNN       0 1 2N  1 w1 w 1 w 1 w 1 A   NNNN             0 1 2N  1 wNNNN1 w  1 w  1 w  1  NNNN      

C. Nikou – Digital Image Processing (E12) Diagonalization of circulant matrices 39 (cont.) The inverse DFT is then expressed by: f = A-1 F

T where 0 1 2N  1 * w0 w 0 w 0 w 0  NNNN        10 1 1 1 2 1 N  1 T w w w w 1*11 NNNN       AA   NN      0 1 2N  1 wNNNN1 w  1 w  1 w  1  NNNN       The theorem implies that any circulant matrix has eigenvectors the columns of A-1.

C. Nikou – Digital Image Processing (E12) Diagonalization of circulant matrices 40 (cont.) Let H be a NxN circulant matrix generated by the 1D N-length signal h[n], that is:

H(,)()[]m n h m  nmod NN  h m  n

Let also αk be the k-th column of the inverse DFT matrix A-1.

We will prove that αk, for any k, is an eigenvector of H. The m-th element of the vector Hα , denoted by Hα k  k m is the result of the circular convolution of the signal h[n] with αk.

C. Nikou – Digital Image Processing (E12) Diagonalization of circulant matrices 41 (cont.) N 1 1 N 1 Hα h[][] m n n kn  km  N k h[] m nNN w n0 N n0 l m n 1 mN( 1) 1 mN( 1)  h[] l wk() m l k m k l  NN  wNNN h[] l w N lm N lm

1 m 1 k m k l k l wNNNNN h[][] l w h l w N l m ( N  1) l  0

We will break it into two parts

C. Nikou – Digital Image Processing (E12) Diagonalization of circulant matrices 42 (cont.)

1NN  1  1 1 k m k l k l k l wNNNNNNN h[][][] l w   h l w   h l w N l m ( N  1) l  0 l  m  1

Periodic with respect to N.

NNN1  1  1 1 k m k l k l k l wNNNNNNN h[][][] l w   h l w   h l w N l N  m ( N  1) l  0 l  m  1

NNN1  1  1 1 k m k l k l k l wNNNNNNN h[][][] l w   h l w   h l w  N l m 1 l  0 l  m  1

C. Nikou – Digital Image Processing (E12) Diagonalization of circulant matrices 43 (cont.) 1 N 1 Hα  wk m h[] l w k l  Hk[]α  km N N N  k m N l0

DFT of h[n] at k. This holds for any value of m. Therefore:

Hααkk Hk[]

which means that αk, for any k, is an eigenvector of H with corresponding eigenvalue the k-th element of H[k], the DFT of the signal generating H .

C. Nikou – Digital Image Processing (E12) Diagonalization of circulant matrices 44 (cont.)

The above expression may be written in terms of the inverse DFT matrix: HA1 A  1Λ  H  A  1 ΛA Λ=diagHHHN [0], [1],..., [ 1] or equivalently: Λ ΑHA1 Based on this diagonalization, we can prove the property between circular convolution and DFT.

C. Nikou – Digital Image Processing (E12) Diagonalization of circulant matrices 45 (cont.) g = Hf  g = HA-1 Af  Ag = AHA-1 Af  G=ΛF

GHF[0]   [0]0 0   [0]  GHF[1]   0 [1] 0   [1]    =                      GNHNFN[1]   0 0 [1][1]     

DFT of g[n] DFT of h [n] DFT of f [n]

C. Nikou – Digital Image Processing (E12) Diagonalization of doubly block 46 circulant matrices • These properties may be generalized in 2D. • We need to define the Kronecker product: ABMNKL,

a11BBB a 12 a 1N aBBB a a AB21 22 2N      aBBB a a M12 M MN MK NL

C. Nikou – Digital Image Processing (E12) Diagonalization of doubly block 47 circulant matrices (cont.)

• The 2D signal f [m,n], 0m  M  1, 0  n  N  1, may be vectorized in lexicographic ordering (stacking one column after the other) to a vector: f MN1 • The DFT of f [m,n], may be obtained in matrix-vector form: F A A f

C. Nikou – Digital Image Processing (E12) Diagonalization of doubly block 48 circulant matrices (cont.) Theorem: The columns of the inverse 2D DFT matrix AA 1 are eigenvectors of any doubly block circulant matrix. The corresponding eigenvalues are the 2D DFT values of the 2D signal generating the doubly block circulant matrix: ΛAAHAA    1

Diagonal, containing the 2D DFT Doubly block circulant of h[m,n] generating H C. Nikou – Digital Image Processing (E12)