Scale Space a 1000 ways of inventing the Gaussian
Tony Lindeberg, Bart M Ter Haar Romeny
Presented by: Sameer Agarwal Artificial Intelligence Laboratory, UCSD
Scale Space – p.1/60 Understanding an Image
Scale Space – p.2/60 Understanding an Image
Scale Space – p.3/60 Image as a map
An image is a representation of a scene. Is it the best possible representation ? How can we make use the image to navigate in the scene ? What if I don’t want the fine structure ?
Scale Space – p.4/60 The problem of scale
Most algorithms in computer vision assume that the scale of interpretation of an image has been decided a priori. How do you decide the scale of an operator in the absence of any apriori knowledge?
Scale Space – p.5/60 The solution
Create a multiscale representation of the image by generating a family of images where fine-scale information is successively supressed. Characterize the basic features in terms of optima of functions defined over this family of images.
Problem: How do we construct this multiscale represen- tation ?
Scale Space – p.6/60 The problem of Differentiation
g(x) = f(x) + sin ωx
gx(x) = fx(x) + ω cos ωx
Scale Space – p.7/60 Example 1
1
0.8
0.6
0.4
0.2
0
−0.2
−0.4
−0.6
−0.8
−1 0 1 2 3 4 5 6 7
Scale Space – p.8/60 Example 1
1.5
1
0.5
0
−0.5
−1
−1.5 0 1 2 3 4 5 6 7
Scale Space – p.9/60 Example 1
2
1.5
1
0.5
0
−0.5
−1
−1.5
−2 0 1 2 3 4 5 6 7
Scale Space – p.10/60 Example 2
Scale Space – p.11/60 Example 2
Scale Space – p.12/60 Example 2
Scale Space – p.13/60 Schwartz’s Theory of Distributions
Differentiation is an ill posed problem. The output of the procedure is not a unique continuous function of its inputs. The problem gets worse with data which is discontinuous. Images ARE discontinuous data.
Scale Space – p.14/60 Schwartz’s Theory of Distributions
Instead of observing the function directly, use a probe. Probe = Infinitely differentiable function. Probing = Correlation. Differentiation = correlation with the derivative of the probe. Problem : What is a good probe?
Scale Space – p.15/60 Previous Work
Quad Trees Pyramids Gaussian Laplacian
Scale Space – p.16/60 Quad Trees
Consider an image D of size 2K 2K. × Define a measure Σ of variation in any region D0 D. ⊂ Let D(K) = D. if Σ(D(K)) > α, split D(K) into p sub images (K 1) Dj − , j = 1, . . . , p. Represent the image using the degree p tree constructed above.
Scale Space – p.17/60 Pyramids
Multiscale representation. 1. f k = f. − k 1 k 2. f − = Reduce(f ). k 1 k 3. f − (x) = c(n)f (2x n) n − Conditions on c.P 1. c(n) > 0 2. c( n ) c( n + 1 ) | | ≥ | | 3. c( n) = c(n) − 4. n c(n) = 1 P 5. n c(2n) = n c(2n + 1) Scale Space – p.18/60 P P Multiscale v/s Multiresolution
Pyramids reduce the size of the image as you go up. Sampling rate reduces. Useful for compression. Complicated to analyse. Multiscale methods like scale space preserve the sampling rate.
Scale Space – p.19/60 Why not just subsample ?
Scale Space – p.20/60 Why not just subsample ?
Subsampling creates image features which are not present in the scene.
Scale Space – p.21/60 The Desiderata
L(x; t) = h(L0(x), t) Shift Invariance Linearity Isotropy Causality Semi-Group structure Separability
Scale Space – p.22/60 Linear Shift Invariance and Isotropy
Each part of the image is processed without regard to its position in the image. The process itself is linear. Isotropy implies that all directions are treated as equal. Implication: The process giving rise to the scale space is a family of convolution operators. i.e. L( ; t) = h( ; t) L · · ∗ 0
Scale Space – p.23/60 Causality
Image features at a scale are only dependent on the image features on the scales finer that itself. The process does not create any new featues. As an implication, the number of extrema is non-increasing with increase in scale.
Scale Space – p.24/60 The Heat Equation
Koenderink (1984) showed that causality implied
∂L = c2 2L ∂t ∇ Describes the evolution of the temperature as a function of space and time. Also describes the process of diffusion, L then represents concentration.
Scale Space – p.25/60 Heat Equation in 1-d
For the case of a rod of length l and zero boundary conditions, L(0, 0) = 0; L(0, l) = 0 and initial condition L(x; 0) = f(x) the solution is given by
nπ 2 2 2 L(x, t) = B sin( x)e n π c t n l − Xn
2 l nπ Bn = f(x) sin( x)dx l Z0 l Scale Space – p.26/60 Semi-Group Structure
h( ; t + t ) = h( ; t ) h( ; t ) · 1 2 · 1 ∗ · 2 more precisely 1. L( ; t ) = h( ; t ) L · 2 · 2 ∗ 0 2. L( ; t ) = (h( ; t t ) h( ; t ) L · 2 · 2 − 1 ∗ · 1 ∗ 0 3. L( ; t ) = h( ; t t ) (h( ; t ) L ) · 2 · 2 − 1 ∗ · 1 ∗ 0 4. L( ; t ) = h( ; t t ) L( ; t ) · 2 · 2 − 1 ∗ · 1
Scale Space – p.27/60 Separability
The kernel should factor into single dimensional kernels.
h(x1, x2; t) = h(x1; t)h(x2; t) Decreases the complexity of calculating the convolution.
Scale Space – p.28/60 Presenting : The Gaussian
xT x 1 n h(x; t) = e− 2t √2πt 1. Rotationally symmetric 2. Separable 3. Infinite Differentiability 4. L (x; t) = ∂ L(x; t) = h L (x) i1,i2,...,in i1,i2,...,in i1,i2,...,in ∗ 0
Scale Space – p.29/60 Example of Gaussian Scale Space
Scale Space – p.30/60 non non-creation of extrema
Scale Space – p.31/60 non non-creation of extrema
Scale Space – p.32/60 non non-creation of extrema
Scale Space – p.33/60 But images are not continuous !
There exists a complete theory of discrete scale space. L(x; t) = T (n; 2t)f(x n) n − αt T (n; 2t) =Pe− In(2αt)
In is the modified Bessel function. In the limit t , T approaches the continuous Gaussian. → ∞ ∂ L(x; t) = L(x + 1; t) 2L(x; t) + L(x 1; t) t − −
Scale Space – p.34/60 Now what?
What can you do with a Gaussian scale space ?
Scale Space – p.35/60 What is a feature?
An image feature is the set of extrema of a function defined on an image. 1. How do you define functions on an image? 2. What are good functions? 3. What are are perceptually meaningful features? 4. What scale should the functions operate on ?
Scale Space – p.36/60 Differential Geometric Invariants
Functions of the various derivatives of an image. Scale space allows us to take derivatives in spatial as well as scale dimensions. Consider invariants based on derivatives of order two and lower.
Scale Space – p.37/60 An observation about gradients
Lxα (x; t1) > Lxα (x; t2) if, t1 < t2
Scale Space – p.38/60 Example
L0 = f(x) = sin ωx
ω2t/2 L(x; t) = e− sin ωx ⇒ 2 ω t/2 Lmax(t) = e− m ω2t/2 Lxm,max(t) = ω e−
Scale Space – p.39/60 Normalized Coordinates
x ξ = √t
∂ξ = √t∂x or,
m/2 m ω2t/2 Lξm,max(t) = t ω e− goes up and then down ! m t = max ω2
Scale Space – p.40/60 The continuing saga of the maxima
m m/2 Lξm,max(tmax) = e The maximum value of the mth order normalized derivative is independent of scale and frequency.
2 tmaxωmax = m
Scale Space – p.41/60 A tale of two maxima
Image structures are characterized by sudden changes, hence they can be formalized as extrema a function of the image (differential forms). In the absence of other evidence, assume that the scale at which some function of normalized derivatives assumes a local maxima (over scales) is a reflection of the size of the corresponding structure in the data.
Scale Space – p.42/60 Scaling property of maxima
If the input image image is scaled by a constant scaling factor s, then the scale at which the maxima is assumed will be multiplied by the same factor. i.e.
f 0(x) = f(sx)
t = s√t ⇒ max0 max p
Scale Space – p.43/60 can we do scale selection now ?
1. Characterize a feature by a differential form ( L). D 2. Find maxima in scale space, to estimate size and approximate location.
( ( L))(x , t ) = 0 ∇ D 0 0 (∂ ( L))(x , t ) = 0 t D 0 0 3. Use size information to perform a refine the location of the feature.
Scale Space – p.44/60 sunflowers
Scale Space – p.45/60 Blob Analysis
Assume a circular blob (same scale on both axes) and model is using a Gaussian.
2 2 1 (x1+x2)/2t0 f(x1, x2) = g(x1, x2, t0) = e− 2πt0 by the semi-group property
L(x1, x2; t) = g(x1, x2, t0 + t) The chosen differential form is L = trace( L) = L + L D | H | | x1x1 x2x2 |
normL = t Lx1x1 + Lx2x2 D | | Scale Space – p.46/60 Blob Analysis
x 1 2 2 L 1 e (x1+x2)/2(t0+t) x1x1 = 3 2 − 2π(t0 + t) − 2π(t0 + t) which can be used to show that the spatial minima for L occurs at D (x1, x2) = (0, 0) t L , t norm (0 0; ) = 2 ⇒ D π(t0 + t)
which in turn has a maxima at t = t0
Scale Space – p.47/60 Blob Analysis
The example verifies that for round blobs the maximum value is attained at a scale proportional to the ”width” of the blob. In the case of elliptical blobs, the scale is given by t = √t1t2.
Scale Space – p.48/60 sunflowers
Blob scale detection using trace( (L)) = 2L. H ∇ξ
Scale Space – p.49/60 Corner Detection
Find a function of the scale space image which indicates the presence of a corner. Use it to identify the scale and the approximate location. Refine the position using Forstner¨ corner operator.
Scale Space – p.50/60 The story of κ
Corner are charcterized by 1. High curvature in the grey level landscape
2 2 L Lxx 2LxLyLxy + L Lyy κ y − x = 2 2 3/2 (Lx + Ly) 2. High gradient
2 2 1/2 γ = (Lx + Ly)
κ = κγ3 2 2 = LyLxx 2LxLyLxy + LxLyy − Scale Space – p.51/60 κ is skew invariant. Corner Detection
The value of κ in normalized coordinates is given by
2 κnorm = t κ Corner points and their corresponding scale is identified using the scale space maxima for κ
Scale Space – p.52/60 Localizing the corner
x1
x0
Normal
T D (x) = (( L)(x0)) (x x0) x0 ∇ −
T dmin = min Dx Dx0 wx0 (x0)dx0 x 0 Zx0
Scale Space – p.53/60 Localizing the corner
minXT Ax 2xT b + c x −
T A = ( L)(x0)( L) (x0)w(x0)dx0 (1) Zx0 ∇ ∇ T b = ( L)(x0)( L) (x0)x0w(x0)dx0 (2) Zx0 ∇ ∇ T T c = x0 ( L)(x0)( L) (x0)x0w(x0)dx0 (3) Zx0 ∇ ∇ and the minima is 1 x = A− b
Scale Space – p.54/60 Where is the scale?
Shape and size of the window function w(x0): Use the scale estimate as the width of the Gaussian for the weight window. Localization Scale: What scale should the gradient 1 L be calculated ? At x = A− b ∇ T 1 d = c b A− b min − The best location for the corner is the one which minimizes dmin w.r.t a scale parameter again. To reduce sensitivity to local image contrast, we instead use T 1 c b A− b dmin = − trace(A) Scale Space – p.55/60 Corner Detection : Example
Scale Space – p.56/60 Corner Detection : Example
Scale Space – p.57/60 Summary
Scale Space provides us with a formalism in which we can construct multiscale representation of signals cleanly and efficiently. Given a certain class of feature descriptors, it allows us to detect the scale at which certain features express themselves.
Scale Space – p.58/60 References
Lindeberg, Tony Scale Space Theory in Computer Vision, 1994, Kluwer Academic Publishers. Lingberg, Tony and M. Ter Haar Romeny, Bart Linear Scale Space, Geometry Driven Diffusion in Computer Vision, Kluwer Academic Publishers, 1994.
Scale Space – p.59/60 Acknowledgments
The music of Chemical Brothers.
Scale Space – p.60/60