Morphological Operations

1 Outline

• Basic concepts: • Erode and dilate • Open and close. • GltGranulometry • Hit and miss transform • Thinning and thickening • Skeletonization and the medial axis transform • Introduction to gray level morphology.

2 Pixel connectivity

• We need to define which pixels areWarning: neighbors. • Are the Pdarkilixe ls are sampl es, pix els in thisnot arraarray squares . connected?

4 Pixel connectivity

• We need to define which pixels are neighbors. • Are the dark pixels in this array connected?

5 Pixel Neighborhoods

4-neighborhood 8-neighborhood

6 Pixel paths

• A 4-connected path between pixels p1 and pn is a set of pixels {p1, p2, …, pn} such that pi is a 4-neighbor of pi+1, i=1,,,…,n-1.

• In an 8-connectedhd path, pi is an 8-neihbighbor of pi+1.

7 Connected regions

• A region is 4-connected if it contains a 4- connected path between any two of its pixels.

•A regiiion is 8-connectedifid if it contai ns an 8- connected path between any two of its pixels.

8 Connected regions

• Now what can we say about the dark pixels in this array?

• What about the light pixels?

9 Connected components labelling • Labels each connected component of a binary image with a separate number.

1 1 1 1 1 1 1 1 1 112213111 111221111 111111111 141155111 111155511 661111551 761188111 761188111

10 Foreground labelling

• Only extract the connected components of the foreground

1 1 1 1 1 1 1 1 1 110010111 111001111 111111111 101100111 111100011 001111001 201100111 201100111

11 What Are Morphological Operators? • Local pixel transformations for processing region shapes • Most often used on binary images

• Logical transformations based on comparison of pixel neighbourhoods with a pattern.

18 Simple Operations - Examples

• Eight-neighbour erode – a.k.a. Minkowski subtraction

• Erase any foreground pixel that has one eight-connected neighbour that is background.

19 8-neighbour erode

Threshold

Erode ×1 Erode ×2 Erode ×5

20 8-neighbour dilate

• Eight-neighbour dilate • a.k.a.

• Paint any background pixel that has one eight-connected neighbour that is foreground.

21 8-neighbour dilate

Dilate ×1 Dilate ×2 Dilate ×5

22 Why?

• Smooth region boundaries for shape analysis. • Remove noise and artefacts from an imperfect segmentation.

• Match particular pixel configurations in an image for simple object recognition.

23 Structuring Elements

• Morphological operations take two arguments: • A binaryyg image •A . • Compare the structuring element to the neighbourhood of each pixel. • This determines the output of the morppghological o peration.

24 Structuring elements

• The structuring element is also a binary array. • A structuring element has an origin .

1 1 1 0 1 0 0 1 0 1 1 1 1 1 1 1 1 0 1 1 1 1 1 0 1 0 0 1 0 0

25 Binary images as sets

• We can think of the binary image and the structuring element as sets containing the pixels with value 1.

1 1 1 0 0 0 1 1 0 0 I = {(1,1), (2,1), (3,1), 0 0 0 0 0 (2,2), (3,2), (4,4)} 0 0 0 1 0 0 0 0 0 0 (Matlab) S = SPARSE(X) converts a sparse or full matrix to sparse form by squeezing out any zero elements26 Some sets notation

• Union and • Difference iiintersection: I1 \ I2 = {x : x ∈ I1 and x ∉ I2}

I1 ∪ I2 = {}x : x ∈ I1 or x ∈ I2

I1 ∩ I2 = {}x : x ∈ I1 and x ∈ I2 • We use φ for t he • Complement empty set . I C = {}x : x ∉ I

27 More sets notation

•The symmetrical set of S with respect to

point o is 0 1 1 1 0 ( 1 1 1 1 1 S = {o − x : x ∈ S} 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0

0 1 1 1 0 1 1 1 0 0 0 1 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1 1 0 1 1 1 0 28 Fitting, Hitting and Missing

• S fits I at x if {y : y = x + s,s ∈ S} ⊂ I • S hits I at x if {y : y = x + s, s ∈ S} ∩ I ≠ φ • S misses I at x if {y : y = x + s, s ∈ S} ∩ I = φ

29 Fitting, Hitting and Missing Image Structuring element 0 1 0 1 1 1 1 1 1 0 1 1 1 1 1 1 0 1 0 0 1 1 1 0 0 0 0 1 1 1 0 1 1 1 0 0 0 0 0 1 0 0 1 1 1 1 0 0 0 0 1 0 1 0 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0 0 0 0

30

• The image E = ISis the erosion of image I by structuring element S.

⎧1 if S fits I at x E(x) = ⎨ ⎩0 otherwise

E = {y : y = x + s ∈ I for every s ∈ S}

31 Implementation (naïve) % I is the input image, S is a structuring element % with origin (ox, oy). E is the output image. E = erode(I, S, ox, oy) [X,Y] = size(I); [SX, SY] = size(S); E = ones(X, Y); for x=1:X; for y=1:Y fi1fj1for i=1:SX; for j=1:SY if(S(i,j)) E(x,y ) = E(x,y ) & ImageEntry(I, x +i-ox, y+j-oy); end end; end end; end 32 Example

1 1 1 1 1 Structuring 1 1 1 1 1 element 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 33 Example

0 1 1 1 0 Structuring 1 1 1 1 1 element 1 1 1 1 1 1 1 1 1 1

0 1 1 1 0 34 Example

1 0 0 0 0 Structuring 0 1 0 0 0 element 0 0 1 0 0 0 0 0 1 0

0 0 0 0 1 35

• The image D = I ⊕ S is the dilation of image I by structuring element S.

⎧1 if S hits I at y D(x) = ⎨ ⎩0 otherwi se D = {y : y = x + s ∈ I for any s ∈ S}

36 Alternative Implementation

• For another implementation of erosion and dilation, note that: E(x) = min(I(x + s)) s∈S D(x) = max(I(x + s)) s∈S

37 Example

1 1 1 1 1 Structuring 1 1 1 1 1 element 1 1 1 1 1 1 1 1 1 1

1 1 1 1 1 38 Example

0 1 1 1 0 Structuring 1 1 1 1 1 element 1 1 1 1 1 1 1 1 1 1

0 1 1 1 0 39 Example

1 0 0 0 0 Structuring 0 1 0 0 0 element 0 0 1 0 0 0 0 0 1 0

0 0 0 0 1 40 Erosion and dilation

• Erosion and dilation are dual operations: (I S)C = I C ⊕ S(

• Commutativity and associativity I ⊕ S = S ⊕ I (I ⊕ S) ⊕T = I ⊕ (S ⊕T ) I S ≠ S I (I S) T = I (S ⊕T )

41 42 Input Output

Structuring Element:

43 and

•The opening of I by S is

I o S = (I S) ⊕ S

•The closing of I by S is I • S = (I ⊕ S) S

44 Opening and Closing

• Opening and closing are dual transformations:

C C ( (I • S) = I o S • Opening and closing are idempotent operations:

I o S = (I o S) o S I • S = (I • S) • S

45 Example

close

1 1 1 1 1 Structuring 1 1 1 1 1 element 1 1 1 1 1 1 1 1 1 1

open 1 1 1 1 1 46 Example

close

0 1 1 1 0 Structuring 1 1 1 1 1 element 1 1 1 1 1 1 1 1 1 1

open 0 1 1 1 0 47 Example

close

1 0 0 0 0 Structuring 0 1 0 0 0 element 0 0 1 0 0 0 0 0 1 0

open 0 0 0 0 1 48 Morphological filtering

• To remove holes in the foreground and islands in the background , do both opening and closing .

• The si ze and sh ape of th e structuri ng el ement determine which features survive.

• In the absence of knowledge about the shape of filifeatures to remove, use a circular structuring element.

49 Example

Close then open

1 1 1 1 1 Structuring 1 1 1 1 1 element 1 1 1 1 1 1 1 1 1 1

Open then close 1 1 1 1 1 50 Example

Close then open

0 1 1 1 0 Structuring 1 1 1 1 1 element 1 1 1 1 1 1 1 1 1 1

Open then close 0 1 1 1 0 51 Example

Close then open

1 0 0 0 0 Structuring 0 1 0 0 0 element 0 0 1 0 0 0 0 0 1 0

Open then close 0 0 0 0 1 52 Count the Red Blood Cells

53 Granulometry

• Provides a size distribution of distinct regions or “granules” in the image. • We open the image with increasing structuring element size and count the number of regions after each operation .

• Creates a “morphological sieve”.

54 Granulometry function gSpec = granulo(I, T, maxRad) % Segment the image I B = (I>T); % Open the image at each structuring element size up to a %i% maximum and count the rema iiiining regions. for x=1:maxRad O = imopen (B,s tre l(‘dis k’,x )); numRegions(x) = max(max(connectedComponents(O))); end gSpec = diff(numRegions);

55 Count the Red Blood Cells

56 Threshold and Label

57 Disc(11)

58 Disc(19)

59 Disc(59)

60 Number of Regions

ss ss radiu region nn ff ber o ller tha mm Nu sma

61 Struct. Elt. radius Granulometric Pattern Spectrum Spectrum ss region ff ber o mm Nu

62 Struct. Elt. radius Hit-and-miss transform

• Searches for an exact match of the structuring element.

• H = I ⊗ S is the hit-and-miss transform of image I bilby structuring element S.

• Simple form of template matching.

63 Hit-and-miss transform

1 0 1 1 1 0 1 0 0 0 1 1 0 1 0 0 0 1 0 0 1 0 0 1 1 ⊗ 1 0 1 = 0 0 0 0 0 1 1 0 1 1 0 0 1 0 0 1 0 1 0 1 0 1 0 1 0

1 0 1 1 1 0 1 0 0 0 1 1 0 1 0 0 0 0 0 1 1 0 1 0 0 1 1 ⊗ = 0 1 0 0 0 * 1 1 1 0 1 1 0 0 1 0 0 1 0 1 0 1 0 0 0 0 0

64 Upper-Right Corner Detector

bwhitmiss(BW1,SE1,SE2) == imerode(BW,SE1) & imerode(~BW,SE2) 65 Thinning and Thickening

• Defined in terms of the hit-and-miss transform: • The thinning of I by S is I S = I \ (I ⊗ S) •The thickening of I by S is I S = I ∪ (I ⊗ S) • Dual operations: (I S)C = I C S

66 (Note) Difference: I1 \ I2 = {x : x ∈ I1 and x ∉ I2} Sequential Thinning/Thickening • These operations are often performed in sequence with a selection of structuring

elements S1, S2,,, …, Sn. • Sequential thinning:

I {Si :i =1,...,n}= (((I S1) S2 )... Sn ) • Sequential thickening:

I {}Si :i =1,...,n = (((I S1) S2 )... Sn )

67 Sequential Thinning/Thickening • Several sequences of structuring elements are useful in practice • These are usually the set of rotations of a single structuring element.

• Sometimes called the Golay alphabet.

68 Golay element L

0 0 0 * 0 0 1 * 0 * 1 *

L1 * 1 * L2 1 1 0 L3 1 1 0 L4 1 1 0 1 1 1 * 1 * 1 * 0 * 0 0

1 1 1 * 1 * 0 * 1 0 0 *

L5 * 1 * L6 0 1 1 L7 0 1 1 L8 0 1 1 0 0 0 0 0 * 0 * 1 * 1 *

69 Sequential Thinning

• See bwmorph in matlab.

0 iterations 1 iteration 2 iterations 5 iterations Inf iterations

70 StilThikiSequential Thickening

1 iteration 5 iterations

2 iterations Inf iterations 71 Skeletonization and the Medial Axis Transform • The skeleton and medial axis transform (MAT) are sti ck -fifigure represent ati ons of a region X ⊂ℜ2.

• Start a grassfire at the boundary of the region.

• The sk el eton i s th e set of poi nts at whi ch two fire fronts meet.

72 Skeletons

73 Matlab Example

I = imread('im1.bmp'); image(bwdist(I)); colormap(gray)

Idist = bwdist(I); IdistNormed = 255* (Idist / max(max(Idist))); IdistNormedThreshed = IdistNormed; IdiiistNormedThreshed(IdistNormed<1) = NaN; surf(IdistNormedThreshed)

74 Medial axis transform

• Alternative skeleton definition: • The skeleton is the union of centres of maximal discs within X. •A maximal disc is a circular subset of X that touches the boundary in at least two places.

• The MAT is the skeleton with the maximal disc radius retained at each point.

75 Medial axis transform

76 Skeletonization using morphology • Use structuring element B = 0 1 0 1 1 1 0 1 0 • The n-th skeleton subset is n denotes n successive Sn (X ) = (X n B) \ [(X n B) o B] erosions. • The skeleton is the union of all the skeleton subsets: ∞

S(X ) = Sn (X ) Un=1 77 MAT Implementation

• How can we extend the skeletonization algorithm to obtain the MAT?

78 Reconstruction

• We can reconstruct region X from its skeleton subsets: ∞

X = Sn (X ) ⊕n B Un=0 • We can reconstruct X from the MAT. • We cannot reconstruct X from S(X).

79 DiFi: Fast 3D Distance Field Computation Using Graphics Hardware Sud,,y,,gp Otaduy, Manocha, Eurographics 2004

80 MAT in 3D

from Transcendata Europe Medial Object Price, Stops, Butlin Transcendata Europe Ltd 81 Applications and Problems

• The skeleton/MAT provides a stick figure representihing the regi ihon shape • Used in object recognition, in particular, character recognition. • Problems: • Definition of a maximal disc is poorly defined on a digital grid. • Sensitive to noise on the boundary. • Sequential thinning output sometimes preferred to skeleton/MAT.

82 Example

Sklkeletons:

Thinned:

83 Gray-level Morphology

• Erosion, dilation • Opening and closing • The image and the structuring element are gray level arrays.

• Used to remove speckle noise. • Also for smoothing, edge detection and segmentation.

84 Umbra

U ( f ) = {}(x, y) : y ≤ f (x)

f U ( f )

85 Top surface

T (R) = {}(x, y) : y ≥ z for all (x, z)∈ R

R T(R) 86 Gray level erode and dilate

• Erosion: f k = T (U ( f ) U (k))

• Dilation: f ⊕ k = T (U ( f ) ⊕U (k))

•Open and close defined as before.

87 f U ( f )

k

U (k)

88 U ( f ) ⊕U (k) T (U ( f ) ⊕U (k)) Max and min operators

• K is the domain of k. • Erosion: f k(x) = min{f (x + z) − k(z)} z∈K

• Dilation: f ⊕ k(x) = max{f (x + z) + k(z)} z∈K

89 Erosion

S=ones(3,3) S=ones(5,5) S=ones(7,7)

90 Dilation

S=ones(3,3) S=ones(5,5) S=ones(7,7)

91 Speckle removal

Erode

Open

Salt and pepper noise

Close

Dilate 92 Favorites

• E = medfilt2() • [D,L] = bwdist()

93 Summary

• Simple morphological operations • Erode and dilate • Open and close • Applications: • Granulometry • Thinning and thickening • Skeletons and the medial axis transform • Gray level morphology

94 95 Find the Letter ‘e’

96 Find the Letter ‘e’

97 Perception: Change Detection?

• Examples from Daniel Simons’s many projects

• UBC examples of change blindness Background Subtraction

• Chromakeying is often impractical

• Camera pose is sometimes locked – opportunity! Distance Measures

• Chromakey distance – remember?

Iα = I − g >T T20T = ~20 g = ( 0 255 0 ) (for example) • Plain Background-subtraction metric:

Iα = I − Ibg >T T = [[] 20 20 10] ((p)for example)

Ibg = Background Image Kylie Minogue, “Come Into My World” Director: Mich el GdGondry, Effects: Olivier “Twist” Gondry 101 Making of “Come Into My World”