Matrix Vector Product is Linear Let x and y be vectors of length, N, and let A be an N × N matrix and where: y = Ax. Matrix-vector product is linear because it satisfies the following two constraints: 1. A(x + y)= Ax + Ay 2. A(cx)= cAx for scalar constant, c. Matrix Vector Product is Linear (contd.) It is is easy to show that matrix-vector product satisfies the first constraint: N−1 N−1 N−1 ∑ Aij(x j + y j)= ∑ Aijx j + ∑ Aijy j j=0 j=0 j=0 ...and the second: N−1 N−1 ∑ Aij cx j = c ∑ Aijx j. j=0 j=0 Kronecker Delta Function The Kronecker delta function is defined as follows: 1 if i = 0 δ(i)= 0 otherwise. Shift Matrix 1000 0 0100 S = δ(i − j − 0 mod 4)= 0010 0001 0001 1 1000 S = δ(i − j − 1 mod 4)= 0100 0010 0010 2 0001 S = δ(i − j − 2 mod 4)= 1000 0100 0100 3 0010 S = δ(i − j − 3 mod 4)= 0001 1000 Shift Matrix (contd.) 1000 1 1 0 0100 2 2 S x = = 0010 3 3 0001 4 4 0001 1 4 1 1000 2 1 S x = = 0100 3 2 0010 4 3 0010 1 3 2 0001 2 4 S x = = 1000 3 1 0100 4 2 0100 1 2 3 0010 2 3 S x = = 0001 3 4 1000 4 1 Shift-invariance If for matrices, A and B, it is the case that: AB = BA then we say that the matrix product com- mutes. This property can be depicted using a commutative diagram: B x −→ Bx ↓ A ↓ A B Ax −→ y Shift-invariance (contd.) What structure must a 3 × 3 matrix, A, possess if its product with each of Sn for 1 ≤ n ≤ 3 is to commute? abc A = d e f g h i We observe that because: Sn = S1...S1 n it suffices to show that| {zA }commutes with S1. Shift-invariance (contd.) Let’s look at the effect of multiplying A by S1: abc 0 0 1 b c a AS1 = d e f 1 0 0 = e f d g h i 0 1 0 h i g ...and multiplying S1 by A: 0 0 1 abc g h i S1A = 1 0 0 d e f = abc 0 1 0 g h i d e f Shift-invariance (contd.) Clearly, AS1 = S1A, iff: b c a g h i e f d = abc h i g d e f Let’s see if we can see any pattern in these equalities: abc abc abc d e f d e f d e f g h i g h i g h i Shift-invariance (contd.) What structure must an N × N matrix, A, possess if its product with each of Sn for 1 ≤ n ≤ N is to commute? A(i, j)= A(i + n mod N, j + n mod N)
a0 a1 ... aN−1 aN−1 a0 ... aN−2 A = ...... a1 a2 ... a0 A matrix with the above structure is termed circulant. Shift-invariance (contd.) abcd 0 0 dabc AS = S A = cdab bcda bcda 1 1 abcd AS = S A = dabc cdab cdab 2 2 bcda AS = S A = abcd dabc dabc 3 3 cdab AS = S A = bcda abcd Discrete Convolution Recall that for a circulant matrix, A, the following is true for all n: A(i, j)= A(i+n mod N, j +n mod N). In particular, it is true when n = − j: A(i, j)= A(i − j mod N,0). Recall that y = Ax can be written: N−1 yi = ∑ A(i, j)x j j=0 N−1 = ∑ A(i − j mod N,0)x j. j=0 Discrete Convolution (contd.) Let h(i − j mod N)= A(i − j mod N,0) then N−1 yi = ∑ h(i − j mod N)x j. j=0 We say that y is the discrete periodic convolution of h and x: y = h ∗ x. Example The weight matrix for a 1D artificial retina applied to a simple step pattern: 0 1 −10 0 0 0 0 0 0 1 −10 0 0 0 −1 0 0 1 −1 0 0 0 = 0 0 0 0 1 −1 0 1 0 00001 −1 1 1 −100001 1 Example (contd.) The convolution kernel is the first col- umn of the weight matrix: 1 0 0 h = 0 0 −1 N−1 yi = ∑ h(i − j mod N)x j j=0 Operators An operator is a function which takes one or more functions as arguments and returns a function as its value. • The gradient operator: ∇ f −→ f 0 where f 0(x)= d f (x)/dx. • The addition operator: f ,g −→{+ f + g} where { f + g}(x)= f (x)+ g(x). • The convolution operator: f ,g −→{∗ f ∗ g} ∞ where { f ∗g}(x)= −∞ f (y)g(x−y)dy. R Digression on High Level Languages One advantage of high level languages (like Scheme) is that you can actually write operators: (define op+ (lambda (f g) (lambda (x) (+ (f x) (g x))))) Linear Operators Let f and g be functions and let A be an operator: A f −→ A f A g −→ Ag. An operator is linear if and only if: 1. A{ f + g} = A f + Ag 2. A{c · f } = c · A f for scalar constant, c. Examples The gradient operator is linear: ∇ f + g −→ f 0 + g0 ∇ c · f −→ c · f 0 Examples (contd.) The convolution with h operator is lin- ear: h f + g −→∗ f ∗ h + g ∗ h h c · f −→∗ {c · { f ∗ h}} ∞ {{ f +g}∗h}(x)= [ f (y)+g(y)]h(x−y)dy Z−∞ ∞ ∞ = f (y)h(x−y)dy+ g(y)h(x−y)dy Z−∞ Z−∞ = { f ∗ h + g ∗ h}(x) and ∞ {{c · f } ∗ h}(x)= c f (y)h(x − y)dy Z−∞ ∞ = c f (y)h(x − y)dy Z−∞ = {c · { f ∗ h}}(x) Linear Operators Linear operator, A, takes function, f , as input and returns function, g, as output: ∞ g(t)= A(t,τ) f (τ)dτ. Z−∞ It is easy to verify that A is linear: ∞ A(t,τ)[ f (τ)+ g(τ)]dτ = Z−∞ ∞ ∞ A(t,τ) f (τ)dτ + A(t,τ)g(τ)dτ Z−∞ Z−∞ and ∞ A(t,τ) c f (τ)dτ = Z−∞ ∞ c A(t,τ) f (τ)dτ Z−∞ Shift Operator
The shift operator, S∆, takes a function, f , as argument, and returns the same function shifted to the right by ∆:
S∆ f −→ f∆ where f∆(t)= f (t − ∆). An operator, A, is shift-invariant, if and only if it commutes with the shift operator. This property can be depicted using a com- mutative diagram:
S∆ f −→ f∆ ↓ A ↓ A S∆ A f −→ g Linear Shift-invariant Operators Linear operator, A, takes function, f , as input and returns function, g, as output: ∞ g(t)= A(t,τ) f (τ)dτ. Z−∞ Let s∆ f = f∆ and s∆g = g∆. If A is shift- invariant, then: ∞ g∆(t)= A(t,τ) f∆(τ)dτ. Z−∞ However, f∆(τ) is just f (τ−∆) and g∆(t) is just g(t − ∆): ∞ g(t − ∆)= A(t,τ) f (τ − ∆)dτ. Z−∞ Adding ∆ to t and τ throughout: ∞ g(t)= A(t + ∆,τ + ∆) f (τ)dτ. Z−∞ Linear Shift-invariant Operators (contd.) We conclude that the following must be true if A is shift-invariant: A(t,τ)= A(t + ∆,τ + ∆). Observe that A(t,τ) is unchanged by adding ∆ to both arguments. This means that the 2D function, A(t,τ), is equal to a 1D function of the difference of t and τ: A(t,τ)= h(t − τ). Consequently, ∞ g(t)= h(t − τ) f (τ)dτ. Z−∞ This is the convolution integral. Con- sequently: g = h ∗ f . Linear Shift-invariant Operators (contd.) Deep Thought: All linear shift invariant operators can be represented as convo- lutions.