Scale Space a 1000 ways of inventing the Gaussian Tony Lindeberg, Bart M Ter Haar Romeny Presented by: Sameer Agarwal Artificial Intelligence Laboratory, UCSD Scale Space – p.1/60 Understanding an Image Scale Space – p.2/60 Understanding an Image Scale Space – p.3/60 Image as a map An image is a representation of a scene. Is it the best possible representation ? How can we make use the image to navigate in the scene ? What if I don’t want the fine structure ? Scale Space – p.4/60 The problem of scale Most algorithms in computer vision assume that the scale of interpretation of an image has been decided a priori. How do you decide the scale of an operator in the absence of any apriori knowledge? Scale Space – p.5/60 The solution Create a multiscale representation of the image by generating a family of images where fine-scale information is successively supressed. Characterize the basic features in terms of optima of functions defined over this family of images. Problem: How do we construct this multiscale represen- tation ? Scale Space – p.6/60 The problem of Differentiation g(x) = f(x) + sin !x gx(x) = fx(x) + ! cos !x Scale Space – p.7/60 Example 1 1 0.8 0.6 0.4 0.2 0 −0.2 −0.4 −0.6 −0.8 −1 0 1 2 3 4 5 6 7 Scale Space – p.8/60 Example 1 1.5 1 0.5 0 −0.5 −1 −1.5 0 1 2 3 4 5 6 7 Scale Space – p.9/60 Example 1 2 1.5 1 0.5 0 −0.5 −1 −1.5 −2 0 1 2 3 4 5 6 7 Scale Space – p.10/60 Example 2 Scale Space – p.11/60 Example 2 Scale Space – p.12/60 Example 2 Scale Space – p.13/60 Schwartz’s Theory of Distributions Differentiation is an ill posed problem. The output of the procedure is not a unique continuous function of its inputs. The problem gets worse with data which is discontinuous. Images ARE discontinuous data. Scale Space – p.14/60 Schwartz’s Theory of Distributions Instead of observing the function directly, use a probe. Probe = Infinitely differentiable function. Probing = Correlation. Differentiation = correlation with the derivative of the probe. Problem : What is a good probe? Scale Space – p.15/60 Previous Work Quad Trees Pyramids Gaussian Laplacian Scale Space – p.16/60 Quad Trees Consider an image D of size 2K 2K. × Define a measure Σ of variation in any region D0 D. ⊂ Let D(K) = D. if Σ(D(K)) > α, split D(K) into p sub images (K 1) Dj − , j = 1; : : : ; p. Represent the image using the degree p tree constructed above. Scale Space – p.17/60 Pyramids Multiscale representation. 1. f k = f. − k 1 k 2. f − = Reduce(f ). k 1 k 3. f − (x) = c(n)f (2x n) n − Conditions on c.P 1. c(n) > 0 2. c( n ) c( n + 1 ) j j ≥ j j 3. c( n) = c(n) − 4. n c(n) = 1 P 5. n c(2n) = n c(2n + 1) Scale Space – p.18/60 P P Multiscale v/s Multiresolution Pyramids reduce the size of the image as you go up. Sampling rate reduces. Useful for compression. Complicated to analyse. Multiscale methods like scale space preserve the sampling rate. Scale Space – p.19/60 Why not just subsample ? Scale Space – p.20/60 Why not just subsample ? Subsampling creates image features which are not present in the scene. Scale Space – p.21/60 The Desiderata L(x; t) = h(L0(x); t) Shift Invariance Linearity Isotropy Causality Semi-Group structure Separability Scale Space – p.22/60 Linear Shift Invariance and Isotropy Each part of the image is processed without regard to its position in the image. The process itself is linear. Isotropy implies that all directions are treated as equal. Implication: The process giving rise to the scale space is a family of convolution operators. i.e. L( ; t) = h( ; t) L · · ∗ 0 Scale Space – p.23/60 Causality Image features at a scale are only dependent on the image features on the scales finer that itself. The process does not create any new featues. As an implication, the number of extrema is non-increasing with increase in scale. Scale Space – p.24/60 The Heat Equation Koenderink (1984) showed that causality implied @L = c2 2L @t r Describes the evolution of the temperature as a function of space and time. Also describes the process of diffusion, L then represents concentration. Scale Space – p.25/60 Heat Equation in 1-d For the case of a rod of length l and zero boundary conditions, L(0; 0) = 0; L(0; l) = 0 and initial condition L(x; 0) = f(x) the solution is given by nπ 2 2 2 L(x; t) = B sin( x)e n π c t n l − Xn 2 l nπ Bn = f(x) sin( x)dx l Z0 l Scale Space – p.26/60 Semi-Group Structure h( ; t + t ) = h( ; t ) h( ; t ) · 1 2 · 1 ∗ · 2 more precisely 1. L( ; t ) = h( ; t ) L · 2 · 2 ∗ 0 2. L( ; t ) = (h( ; t t ) h( ; t ) L · 2 · 2 − 1 ∗ · 1 ∗ 0 3. L( ; t ) = h( ; t t ) (h( ; t ) L ) · 2 · 2 − 1 ∗ · 1 ∗ 0 4. L( ; t ) = h( ; t t ) L( ; t ) · 2 · 2 − 1 ∗ · 1 Scale Space – p.27/60 Separability The kernel should factor into single dimensional kernels. h(x1; x2; t) = h(x1; t)h(x2; t) Decreases the complexity of calculating the convolution. Scale Space – p.28/60 Presenting : The Gaussian xT x 1 n h(x; t) = e− 2t p2πt 1. Rotationally symmetric 2. Separable 3. Infinite Differentiability 4. L (x; t) = @ L(x; t) = h L (x) i1;i2;:::;in i1;i2;:::;in i1;i2;:::;in ∗ 0 Scale Space – p.29/60 Example of Gaussian Scale Space Scale Space – p.30/60 non non-creation of extrema Scale Space – p.31/60 non non-creation of extrema Scale Space – p.32/60 non non-creation of extrema Scale Space – p.33/60 But images are not continuous ! There exists a complete theory of discrete scale space. L(x; t) = T (n; 2t)f(x n) n − αt T (n; 2t) =Pe− In(2αt) In is the modified Bessel function. In the limit t , T approaches the continuous Gaussian. ! 1 @ L(x; t) = L(x + 1; t) 2L(x; t) + L(x 1; t) t − − Scale Space – p.34/60 Now what? What can you do with a Gaussian scale space ? Scale Space – p.35/60 What is a feature? An image feature is the set of extrema of a function defined on an image. 1. How do you define functions on an image? 2. What are good functions? 3. What are are perceptually meaningful features? 4. What scale should the functions operate on ? Scale Space – p.36/60 Differential Geometric Invariants Functions of the various derivatives of an image. Scale space allows us to take derivatives in spatial as well as scale dimensions. Consider invariants based on derivatives of order two and lower. Scale Space – p.37/60 An observation about gradients Lxα (x; t1) > Lxα (x; t2) if, t1 < t2 Scale Space – p.38/60 Example L0 = f(x) = sin !x !2t=2 L(x; t) = e− sin !x ) 2 ! t=2 Lmax(t) = e− m !2t=2 Lxm;max(t) = ! e− Scale Space – p.39/60 Normalized Coordinates x ξ = pt @ξ = pt@x or, m=2 m !2t=2 Lξm;max(t) = t ! e− goes up and then down ! m t = max !2 Scale Space – p.40/60 The continuing saga of the maxima m m=2 Lξm;max(tmax) = e The maximum value of the mth order normalized derivative is independent of scale and frequency. 2 tmax!max = m Scale Space – p.41/60 A tale of two maxima Image structures are characterized by sudden changes, hence they can be formalized as extrema a function of the image (differential forms). In the absence of other evidence, assume that the scale at which some function of normalized derivatives assumes a local maxima (over scales) is a reflection of the size of the corresponding structure in the data. Scale Space – p.42/60 Scaling property of maxima If the input image image is scaled by a constant scaling factor s, then the scale at which the maxima is assumed will be multiplied by the same factor. i.e. f 0(x) = f(sx) t = spt ) max0 max p Scale Space – p.43/60 can we do scale selection now ? 1. Characterize a feature by a differential form ( L). D 2. Find maxima in scale space, to estimate size and approximate location. ( ( L))(x ; t ) = 0 r D 0 0 (@ ( L))(x ; t ) = 0 t D 0 0 3. Use size information to perform a refine the location of the feature. Scale Space – p.44/60 sunflowers Scale Space – p.45/60 Blob Analysis Assume a circular blob (same scale on both axes) and model is using a Gaussian. 2 2 1 (x1+x2)=2t0 f(x1; x2) = g(x1; x2; t0) = e− 2πt0 by the semi-group property L(x1; x2; t) = g(x1; x2; t0 + t) The chosen differential form is L = trace( L) = L + L D j H j j x1x1 x2x2 j normL = t Lx1x1 + Lx2x2 D j j Scale Space – p.46/60 Blob Analysis x 1 2 2 L 1 e (x1+x2)=2(t0+t) x1x1 = 3 2 − 2π(t0 + t) − 2π(t0 + t) which can be used to show that the spatial minima for L occurs at D (x1; x2) = (0; 0) t L ; t norm (0 0; ) = 2 ) D π(t0 + t) which in turn has a maxima at t = t0 Scale Space – p.47/60 Blob Analysis The example verifies that for round blobs the maximum value is attained at a scale proportional to the ”width” of the blob.