Content-Based Image Retrieval: Near Tolerance Rough Sets

Introduction Visual Perception Near Rough Sets Tolerance Near Sets Measure Example Implementation of Near Sets Concluding Remarks Bibliography Acknowledgments Appendix

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3

1Department of Applied Computer Science, University of Winnipeg Winnipeg, Manitoba R3B2E9 Canada 2Computational Intelligence Laboratory Electrical & Computer Engineering, University of Manitoba 3School of Math., Physics & Information Science, Zhejiang Ocean University, Zhejiang 316004 P.R. China

20 October 2010 Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Tolerance Near Sets Measure Example Implementation of Near Sets Concluding Remarks Bibliography Acknowledgments Appendix Motivation

The notion of nearness in mathematics and the more general notion of resemblance is a dominant part of image correspondence. Near set theory provides a formal basis for discovering aﬃnities between perceptual granules such as images. Focus: Visual Information-centric approach to object recognition.

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Tolerance Near Sets Measure Example Implementation of Near Sets Concluding Remarks Bibliography Acknowledgments Appendix Outline 1 Introduction Philosophical Poem 2 Visual Perception History of Perceptual Representative Spaces Near Sets: Basic Idea Probes SubImage Description Perceptual System All Partition Classes Single Image Class 3 Near Rough Sets Rough Sets Sheela RAMANNATolerance1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Tolerance Near Sets Measure Example Philosophical Poem Implementation of Near Sets Concluding Remarks Bibliography Acknowledgments Appendix Reﬂections on Nearness

Near To How near to the bark of a tree are drifting snowﬂakes, swirling gently round, down from winter skies? How near to the ground are icicles, slowly forming on window ledges? –Fragment of a Philosophical Poem. –Z. Pawlak & J.F. Peters, 2002.

[1] PAWLAK Z., PETERS J. F., Jak blisko, Systemy Wspomagania Decyzji I, 2007, 57..

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception History of Perceptual Representative Spaces Near Rough Sets Near Sets: Basic Idea Tolerance Near Sets Probes Measure Example SubImage Description Implementation of Near Sets Perceptual System Concluding Remarks All Partition Classes Bibliography Single Image Class Acknowledgments Appendix Evolution of perceptual representative spaces

J.H. Poincar´e(1894-1902): framework for the study of resemblance & representative spaces, M. Fr´echet (1906): metric spaces, distance between sets, F. Riesz (1908): concept of proximity or spatial nearness of pairs of sets, E.C. Zeeman (1961): tolerance spaces and visual perception, S. Marcus (1992): Tolerance rough sets, Cech topologies, learning processes, J.F. Peters (2008): Tolerance near sets & perceptual representative spaces.

The basic idea in the near set approach to object recognition is to compare object descriptions. Sets of objects X , Y are considered near each other if the sets contain objects with at least partial matching descriptions. –Near sets. General theory about nearness of objects, –J.F. Peters, 2007.

Deﬁnition Probe(Peters,2007) A probe function φ : X → < represents an observable object feature.

Method: choose probe functions to represent object features, Probe functions provide a basis for describing aﬃnities between objects.

let B = {φ1, φ2} denote a set of two probe functions representing average greyscale and average edge orientation of the pixels in the subimage x. Then

ΦB(x) = (φ1(x), φ2(x))

is a sample description of subimage x.

Deﬁnition Perceptual System(Peters, 2008) A perceptual system hO, Fi consists of a sample space O containing a ﬁnite, non-empty set of observed sample objects and countable, non-empty set F containing probe functions representing object features.

Method: (1) identify sample space O and set of probes B ⊆ F to formulate hO, Fi, Example: Set of microscope images and set of image processing probe functions.

All classes in sample image partitions determined by indiscernibility relation ∼B

(a) Camarg. (b) 5 × 5 (c) 10 × 10 (d) 15 × 15 (e)20 × 20

One class per image partitions determined by indiscernibility relation ∼B. Color identiﬁes subimages in class.

(a) 5 × 5 (b) 10 × 10 (c) 15 × 15 (d) 20 × 20

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Rough Sets Near Rough Sets Tolerance Tolerance Near Sets Equivalence Measure Example Near Rough Sets Origin & Deﬁnition Implementation of Near Sets Nearness Description Principle Concluding Remarks Sample Near Rough Sets Bibliography Near Classes I Acknowledgments Near Classes II Appendix Rough Set Notation

Symbol Interpretation

∼B {(x, y) | f (x)= f (y) ∀f ∈ B}, indiscernibility relation,

x/∼B x/∼B = {y ∈ X | y ∼B x}, elementary set (class),

O/∼B O/∼B = {x/∼B | x ∈ O}, quotient set. B∗(X ) = x (lower approximation of X), /∼B x/ ⊆X ∼B B∗(X ) = S x (upper approximation of X). /∼B x/∼ ∩X =6 ∅ BS

Tolerance on a set formalizes the idea of resemblance (A.B. Sossinsky, 1986) ∼ =B,ε= {(x, y) ∈ O × O : k φB(x) − φB(y) kp ≤ ε}, (1)

1 n p p where k · kp = ( i=1(·i ) (Lp norm) Usually, either pP= 1 (taxicab distance) or p = 2 (Euclidean distance)

Z. Pawlak (1991) introduced the indiscernibility relation in an attribute-based approach to classifying objects Real-valued functions representing object attributes are considered The partition of X is determined by the indiscernibility relation in (2)

∼B = {(x, y) ∈ X × X | ∀φ ∈ B, |φ(x) − φ(y)| = 0}. (2)

∗ A non-empty set X is a rough set ⇐⇒ |B (X ) − B∗(X )|= 6 0. Put ε = 0 in (1)to obtain a form of the indiscernibility relation

Deﬁnition Near Rough Sets(see [8]) A rough set X is near rough set Y , X ∩ Y = ∅ ⇐⇒ there is at least one pair x, y ∈ X × Y such that |φ(x) − φ(y)|≤ ε.

Nearness Property : Disjoint rough sets X , Y are near each other if, and only if one can ﬁnd subsets A ⊂ X , B ⊂ Y such that A is descriptively near B.

[8] J. F. Peters and S. Ramanna, “Aﬃnities between perceptual granules: Foundations and Perspectives,” In: Human-Centric Information Processing Through Granular Modelling, Springer Studies in Computational Intelligence vol. 182, 49-66, 2008.

Deﬁnition Nearness Description Principle(see [9]) For non-empty, disjoint near sets X , Y , one can ﬁnd x ∈ X , y ∈ Y such that the description of x ∈ X is similar to the description of y ∈ Y .

Near rough sets satisfy the Nearness Description Principle .

[9] J. F. Peters, “Near sets: General theory,” Applied Mathematical Sciences vol. 1, 2609-2629, 2007.

The ovals illustrate the idea of near rough sets Near.1 X is a rough set, since no colour class is completely contained inside X . Near.2 lower approximation of X is empty. Near.3 Similarly, Y is a rough set. Near.4 lower approximation of Y is empty. Near.5 upper approx. of X = {yellow, pink, green}∼ Y . Near.6 approximation boundary of X, Y is not empty ⇒ X , Y are rough. Near.7 is common to X , Y ⇒ X near Y .

Problem: How to answer the question “Are these sets similar?” , At some level, we compare objects based on object descriptions, Solution: Group objects based on equivalent descriptions. X Y

Sets with similar equivalence classes are near each other.

B = {φGender(x),φOutﬁt(x)} X Y

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Tolerance Relation I Near Rough Sets Tolerance Relation II Tolerance Near Sets Tolerance Relation III Measure Example Tolerance Relation IV Implementation of Near Sets Quantifying Nearness I Concluding Remarks Quantifying Nearness II Bibliography Quantifying Nearness III Acknowledgments Appendix Tolerance Relation

Matching descriptions unrealistic for objects in the real world, Instead, group objects within some range of feature values.

20 16 11

3 2

6 Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Tolerance Relation I Near Rough Sets Tolerance Relation II Tolerance Near Sets Tolerance Relation III Measure Example Tolerance Relation IV Implementation of Near Sets Quantifying Nearness I Concluding Remarks Quantifying Nearness II Bibliography Quantifying Nearness III Acknowledgments Appendix Tolerance Relation

For example, creating classes where the number of pegs is within 5 of each block in the class, Result: three classes appear containing information about the blocks with respect to the feature “number of pegs”.

2020 16 1111

3 2 Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Tolerance Relation I Near Rough Sets Tolerance Relation II Tolerance Near Sets Tolerance Relation III Measure Example Tolerance Relation IV Implementation of Near Sets Quantifying Nearness I Concluding Remarks Quantifying Nearness II Bibliography Quantifying Nearness III Acknowledgments Appendix Tolerance Relation

Also, objects can now belong to more than one class .

2020 16 1111

3 2

Perceptual Tolerance Relation Let hO, Fi be a perceptual system and let ε ∈ R. For every B ⊆ F a reﬂexive and symmetric ∼ tolerance relation =B,ε is deﬁned as follows: ∼ =B,ε= {(x, y) ∈ O × O : k φ(x) − φ(y) k2 ≤ ε}.

Neighbourhood ∼ N(x)= {y ∈ O : x =B,ε y}

∼ Tolerance Class A set X ⊆ O is a tolerance class when x =B,ε y for any pair x, y ∈ X. Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Tolerance Relation I Near Rough Sets Tolerance Relation II Tolerance Near Sets Tolerance Relation III Measure Example Tolerance Relation IV Implementation of Near Sets Quantifying Nearness I Concluding Remarks Quantifying Nearness II Bibliography Quantifying Nearness III Acknowledgments Appendix Nearness I

Practical applications required a nearness measure, For example, both pairs of sets are near each other.

Required: method to determine similar sets . Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Tolerance Relation I Near Rough Sets Tolerance Relation II Tolerance Near Sets Tolerance Relation III Measure Example Tolerance Relation IV Implementation of Near Sets Quantifying Nearness I Concluding Remarks Quantifying Nearness II Bibliography Quantifying Nearness III Acknowledgments Appendix Nearness II

Problem: How to quantify the nearness of sets?

Solution: Compare cardinalities of classes shared by respective sets Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Tolerance Relation I Near Rough Sets Tolerance Relation II Tolerance Near Sets Tolerance Relation III Measure Example Tolerance Relation IV Implementation of Near Sets Quantifying Nearness I Concluding Remarks Quantifying Nearness II Bibliography Quantifying Nearness III Acknowledgments Appendix Nearness III

Let X and Y be disjoint sets and let Z = X ∪ Y . Then a tolerance Nearness Measure (tNM) [6] is given by

1 min(|C ∩ X |, |[C ∩ Y |) tNM∼ (X , Y ) = · |C| , =B,ε |Hε (Z)| max(|C ∩ X |, |C ∩ Y |) B ∈Hε C XB(Z) ε where HB(Z) denotes the family of all tolerance classes of relation =∼B,ε on the set Z. [6] A. E. Hassanien, A. Abraham, J. F. Peters, G. Schaefer, and C. Henry, “Rough sets and near sets in medical imaging: A review,” IEEE Transactions on Information Technology in Biomedicine, vol. 13, no. 6, pp. 955-968, 2009.

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Tolerance Near Sets Measure Example I Measure Example Measure Example II Implementation of Near Sets Measure Example III Concluding Remarks Measure Example IV Bibliography Acknowledgments Appendix Example I

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Tolerance Near Sets Measure Example I Measure Example Measure Example II Implementation of Near Sets Measure Example III Concluding Remarks Measure Example IV Bibliography Acknowledgments Appendix Example II

1 0.75 0.4375

0.25 0 NM method: counting no. of objects with the same description .

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Tolerance Near Sets Measure Example I Measure Example Measure Example II Implementation of Near Sets Measure Example III Concluding Remarks Measure Example IV Bibliography Acknowledgments Appendix Example III

X Y

Family of Classes Tolerance Class (TC) TC Size Object in X Objects in Y TC Ratio

400 200 200 1

100 100 0 0

100 0 100 0

200 100 100 1 1 tNM∼ (X , Y ) = 1 · 400 + 1 · 200 = 0.75, =B,ε 800 Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Tolerance Near Sets Measure Example I Measure Example Measure Example II Implementation of Near Sets Measure Example III Concluding Remarks Measure Example IV Bibliography Acknowledgments Appendix Example IV

X Y

Family of Classes Tolerance Class (TC) TC Size Object in X Objects in Y TC Ratio

300 200 100 0.5

100 100 0 0

200 0 200 0

200 100 100 1 1 tNM∼ (X , Y ) = 0.5 · 300 + 1 · 200 = 0.4375. =B,ε 800 Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Near System Tolerance Near Sets Probe Functions Table Measure Example Image Regions of Interest (ROI) Implementation of Near Sets Sample Caltech Image Retrieval Plots Concluding Remarks Sample Simplicity Image Retrieval Plots Bibliography Acknowledgments Appendix Image Analysis with Near Sets

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Near System Tolerance Near Sets Probe Functions Table Measure Example Image Regions of Interest (ROI) Implementation of Near Sets Sample Caltech Image Retrieval Plots Concluding Remarks Sample Simplicity Image Retrieval Plots Bibliography Acknowledgments Appendix Set of Probe Functions

Table: Set of probe functions: B = {φ1,φ2, ..., φ11} Probe function Feature Description

φ1 Colour Average grey level of pixels φ3, φ4, φ5 Colour Red, Green and Blue colour components φ6 Shape Average edge intensity φ7 Shape Dominant edge orientation φ2 Texture Entropy of the greylevel values φ8 Texture Contrast φ9 Texture Correlation φ10 Texture Uniformity φ11 Texture Homogeneity

[1] Image archives of CalTech Computational Vision Group, http://www.vision.caltech.edu/html-ﬁles/archive.html, 2005 [2, 3] James Z. Wang, SIMPLIcity-Content-based Image Search Engine,Content Based Image Retrieval Project, http://wang.ist.psu.edu/IMAGE, 1995–2001

QairA QairB 100 100 tNM tNM 90 HdNM 90 HdNM tcNM tcNM 80 80

70 70

60 60

50 50

40 40 relevant images relevant images

30 30

20 20

10 10

0 0 0 100 200 300 400 500 600 700 0 100 200 300 400 500 600 Number of images retrieved Number of images retrieved

Trial 1 includes 600 Caltech archive images , ε = 0.2,

Fig. 1 (left plot) shows results with 2 probes B = {φ6,φ7} in Probe Fn Table, Fig. 2 (right plot) shows results with shape, texture, colour features using all probes B = {φ1,...,φ11} in Probe Fn Table.

Q824A Q824B 100 100 tNM tNM 90 HdNM 90 HdNM tcNM tcNM 80 80

70 70

60 60

50 50

40 40 relevant images relevant images

30 30

20 20

10 10

0 0 0 200 400 600 800 1000 0 200 400 600 800 1000 Number of images retrieved Number of images retrieved

Trial 1 includes 1000 Simplicity archive images , ε = 1.1,

Perceptual tolerance spaces, not abstract tolerance spaces, provide basis for study of tolerance near sets . Perceptual tolerance rough sets deﬁned with feature vector descriptions of objects. Discriminatory power of tNM measure in discerning similarities between images with lots of clutter. Near System and Image Retrieval Experiments.

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Tolerance Near Sets Articles I Measure Example Articles II Implementation of Near Sets Articles III Concluding Remarks Bibliography Acknowledgments Appendix Related Works I

Z. Pawlak. Classiﬁcation of objects by means of attributes. Polish Academy of Sciences 429 (1981). J.F. Peters. Near sets: General theory. Ap. Math. Sci. 1, 2007, 2609-2629. J.F. Peters. Near sets: Special theory about nearness of objects. Fundamenta Informaticae (75) 1-4, 2007, 407-433. J.F. Peters, P. Wasilewski. Foundations of near sets. Information Sciences 79, 2009, 3091-3109.

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Tolerance Near Sets Articles I Measure Example Articles II Implementation of Near Sets Articles III Concluding Remarks Bibliography Acknowledgments Appendix Related Works II

J.F. Peters and S. Ramanna. Aﬃnities between perceptual granules: Foundations and perspectives. Human-Centric Information Processing Through Granular Modelling SCI 182, A. Bargiela and W. Pedrycz (Eds), Springer-Verlag, 2009, 49-66. S. Ramanna. Perceptually near Pawlak partitions. Transactions on Rough Sets XII, LNCS 6190, 2010, 170-192. P. Wasilewski and J.F. Peters and S. Ramanna. Perceptual tolerance intersection. Proceedings of RSCTC 2010, Lecture Notes in Artiﬁcial Intelligence 6086, 2010, 277-286.

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Tolerance Near Sets Articles I Measure Example Articles II Implementation of Near Sets Articles III Concluding Remarks Bibliography Acknowledgments Appendix Related Works III

S. Ramanna. Image Analysis in Poincar´e-Peters Perceptual Representative Spaces. Innovations in Intelligent Image Analysis, Springer, 2011, to appear. Henry, C., Peters, J.F.: NEAR system. Tech. report, Computational Intell. Lab., Univ. of Manitoba (2010) UM CI Laboratory, TR-2010-017, http//wren.ee.umanitoba.ca .

C. Henry. Near Sets. Theory and Applications . Ph.D. thesis, supervisor: J.F. Peters, University of Manitoba, Sept. 2010.

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets Introduction Visual Perception Near Rough Sets Tolerance Near Sets Measure Example Computational Intelligence Laboratory Implementation of Near Sets Concluding Remarks Bibliography Acknowledgments Appendix Acknowlegments

Natural Science and Engineering Council of Canada (NSERC) grants, Manitoba Centre of Excellence Fund (MCEF), Canadian Network Centers of Excellence (CAN grant #SRI-BIO-05).

||A ∩ X |−|A ∩ Y || tcDM(X , Y )= , (3) |A ∩ X | + |A ∩ Y | A∈Hε (X ∪Y ) XB tcDM tcNM(X , Y ) = 1 − ε , (4) s|HB(X ∪ Y )|

Sheela RAMANNA1,2,James F. PETERS2,Wei-Zhi WU3 Content-Based Image Retrieval: Near Tolerance Rough Sets