A PrimeNumb erBased Matrix Strategy for



Ecient Iconic Indexing of Symb olic Pictures

YeIn Chang and BiYen Yang

Dept of Applied Mathematics

National YatSen University

Kaohsiung Taiwan

ROC

fEmail changyimathnsysuedutwg

fTel ext g

fFax g

Abstract

In the previous approaches to represent pictorial data as the complexity of the represen

tation strategy is increased the more spatial relationships can b e represented which also

results in a more complex strategy for query pro cessing and a limited typ es of queries

that can b e answered In this pap er we prop ose an ecient iconic indexing scheme called

PrimeNumberbased Matrix PN Matrix for symb olic pictures whichcombines the ad

vantages of the D Cstring and the DLT matrix Basically the prop osed strategy can

represent those complex relationships which are represented in D Cstrings in a matrix

and an ecient mo dulebased op eration can b e used to supp ort pictorial query spatial

reasoning and similarity retrieval In the prop osed scheme we classify spatial relation

ships b etween two ob jects in D space into ve spatial categories and dene a category rule

for each of those ve spatial categories Those category rules are mo duleop erationbased

therefore they are ecient enough as compared to the previous approaches Following

those category rules we prop ose algorithms to eciently supp ort spatial reasoning picture

queries and similarity retrieval based on a data structure of a PrimeNumb erbased Matrix

PN Matrix

Keywords D string D Cstring image databases pictorial query similarity retrieval

spatial reasoning symb olic databases



This researchwas supp orted in part by the National Science Council of Republic of China under Grant No NSCE

Intro duction

The design of image databases has attracted much attention over the past few years Ap

plications which use image databases include oce automation computer aided design

rob otics and medical pictorial archiving A common requirement of these systems is to



mo del and access pictorial data with ease Thus one of the most imp ortant problems in



the design of image database systems is how images are stored in the image database In

traditional database systems the use of indexing to allow database accessing has b een well

established Analogously picture indexing techniques are needed to make ease pictorial



information retrieval from a pictorial database

Over the last deco de many approaches to represent symbol data have b een prop osed



as shown in Figure Chang et al prop ose a pictorial data structure D stringusing

symb olic pro jections to representsymb olic pictures preserving spatial relationships among



ob jects The basic idea is to pro ject the ob jects of a picture along the xaxis and y

axis to form two strings representing the relative p ositions of ob jects in the xaxis and



y axis resp ectively A picture query can also b e sp ecied as a D string Based on D

strings several algorithms in pictorial querying spatial reasoning and similarity retrieval

are prop osed where picture querying allows the users to query images with a sp ecied

spatial relationship spatial reasoning means the inference of a consistent set of spatial

relationships among the ob jects in an image and the target of similar retrieval is to retrieve

the images that are similar to the query image Based on the D string longest common



subsequence strategy Lee et al prop ose an algorithm for similarity retrieval Based on an

 

objects and spatial relationship OSR table strategyLiaw also prop oses an algorithm for

similarity retrieval which distinguishes whether two images are similar by comparing two

sets of real numb ers that are asso ciated to these images resp ectively Besides Costagliola



et al intro duce a variation of D string representation for symb olic pictures the non

redundant D stringwhich is a more compact representation than the D string

However the representation of D strings is not sucient enough to describ e pictures

   

and Chang et al in of arbitrary complexity completelyFor this reason Jungert

tro duce more spatial op erators to handle more typ es of spatial relationships among ob jects

in image databases Using these extended spatial op erators D Gstring representation Symbolic Picture Representation

2D string 9DLT-matrix [Chang 87] [Chang 91]

* * non-redundant 2D string longest 2D G-string fast spatial match 2D string common subsequence [Chang 88] accessing scheme [Costagliola 95] [Lee 89] [Chang 90]

2D C-string * * [Lee 90] object’s and spatial module-oriented relationship (OSR) table signature extraction [Liaw 90] + [Chang 90] 2D C- string [Huang 94]

*: Query Processing Strategies

Figure Approaches to symb olic picture representation

facilitates spatial reasoning ab out shap es and relative p ositions of ob jects But a D G

string representation scheme is not ideally economic for complex images in terms of storage



space eciency and navigation complexity in spatial reasoning Therefore Lee and Hsu

prop ose a D Cstring representation scheme Since the numb er of subparts generated by

this new cutting mechanism is reduced signicantly the lengths of the strings representing

pictures are much shorter while still preserving the spatial relationships among ob jects

However the previous abstractions ignore the relative sizes and lo cations of ob jects Thus

similarity retrieval of images maybeambiguous and manytyp es of queries concerning sizes

lo cations and distances cannot b e answered due to this inadequacy Therefore Huang and



Jean prop ose a D C string representation scheme which extends the work of Lee and



Hsu D Cstring by including relative metric information ab out the picture into the

strings

As describ ed b efore based on D string representation the problem of picture query

turns out to b e the matching of D subsequences which takes nonp olynomial time com

plexity This makes the picture retrieval metho d inappropriate for implementation esp e



cially when the numb er of ob jects in an image is large Therefore Chang et al prop ose

an ecient approach of iconic indexing byanine direction lowertriangular DLT ma

trix Moreover to handle large amounts of image databases a simple algorithm for spatial

match retrieval of symb olic pictures based up on DLT matrices is prop osed by Chang et



al by using the concept of superimposedcoding which reduces the ratio of false match



when a query picture o ccurs Next Chang et al prop ose a moduleoriented signature

extraction strategy in which prime numb ers are used to comp ose the signatures and the

mo dule op eration will b e applied when the query happ ens

In the previous approaches to represent pictorial data as the complexity of the rep

resentation strategy is increased the more spatial relationships can b e represented which

also results in a more complex strategy for query pro cessing and a limited typ es of queries

that can b e answered In this pap er we prop ose an ecient iconic indexing scheme called

PrimeNumberbased Matrix PN Matrix for symb olic pictures whichcombines the ad

vantages of the D Cstring and the DLT matrix Basically the prop osed strategy can

represent those complex spatial relationships that are represented in D Cstrings in a

matrix while it do es not need any cutting strategy and complex pro cedures to do spa

tial reasoning Moreover the prop osed strategy can b e considered as an extended DLT

matrix strategy in which more than spatial relationships can b e represented and an ef

cient mo dulebased op eration can b e used to supp ort pictorial query spatial reasoning

and similarity retrieval In the prop osed scheme we classify those spatial relationships



between two ob jects in D space as observed by Lee and Hsu into ve spatial categories

and dene a category rule for each of those ve spatial categories Those category rules are

mo duleop erationbased therefore they are ecient enough as compared to the previous

approaches Following those category rules we prop ose algorithms to eciently supp ort

spatial reasoning picture queries and similarity retrieval based on a PrimeNumb erbased

Matrix PN Matrix strategy In a PN Matrix the relationships b etween two ob jects along

the xaxis or y axis is recorded in a numb er which is a pro duct of some prime numbers

Therefore spatial reasoning can b e done very straightforwardlyFor answering a pictorial

query some mo dule op erations or memb ership checking of a set of numb ers are applied In

similarity retrieval some new matrix op erations are applied

The rest of the pap er is organized as follows In Section wegive a brief description d

b c e a

f

Figure A picture f

ab out the previous symb olic picture representations In Section we will present the

prop osed ecient iconic indexing scheme for symb olic pictures Finally Section gives a

conclusion

Background

In this Section we briey describ e several data structures for symb olic picture representa

tion including D string D Gstring D Cstring and DLT matrix Moreover we will

briey describ e the strategy for spatial reasoning based on the D Cstring representation

esp ecially

D string



For pictorial information retrieval Chang et al presentanewway of representing a

picture bya D string and a picture query can also b e sp ecied as a D string The problem

of pictorial information retrieval then b ecomes a problem of D subsequence matching

Let V b e a set of symb ols where eachsymb ol could represent a pictorial ob ject or a

pixel Let A b e the set fgwhere and are three sp ecial symb ols not in V

For example consider the picture shown in Figure V fa b c d e f g The D string

representing the ab ove picture f is as follows

a de f b c a e fb cd

where the symbol denotes the leftright or b elowab ove spatial relationship The

symb ol denotes the at the same spatial lo cation as relationship and the symbol

denotes the in the same set as relation A A C B B C

D D

(a) (b)

Figure The cutting mechanism of the D Gstring a cut along the xaxis b cut

along the y axis

D Gstring

The spatial op erators and are not sucientenoughtogive a complete description of

spatial knowledge for pictures of arbitrary complexityTo represent the spatial relationship

between two nonzero sized ob jects esp ecially for the case of overlapping ob jects Jungert

 

intro duces some local op erators as comp ensation for handling more typ es of relation

 

ships b etween pictorial ob jects in query reasoning Later Chang et al intro duce the

generalized D string D Gstring with a cutting mechanism

Jungert extends the op erators of D strings as g l obal op erators R and intro duces a set

g

of l ocal op erators R to handle more typ es of spatial relationships among nonzero sided

l

ob jects ie the partial overlapping relationships which are dened as follows where the

denitions of those spatial op erators are given in Table

R f jg

g

R fn g

l

The cuttings are p erformed at all extreme p oints of all the ob jects to segment the

ob jects in the image One example is shown in Figure and the corresp onding D G

string representation is as follows

D G xstringf AjA B jA B D jA D jD jC D jD

y stringf DBjB C jA B C jA C jA D G Notation Condition Meaning

A < B center(A) < center(B) A B

A = B center(A) = center(B) A B

edge to edge with A | B A B

A % B min(A) > min(B) A max(A) < max(B) B length(A) < length(B)

A [ B min(A) = min(B) A length(A) < length(B) B

A ] B max(A) = max(B) A length(A) < length(B) B

A \ B min(A) < min(B) A length(A) <= length(B) B

A / B max(A) > max(B) A

length(A) <= length(B) B

Table Denitions of Jungerts spatial op erators

D Cstring

In D Gstring representation the numb er of segmented subparts of an ob ject is dep endent

of the numb er of b ounding lines of other ob jects which are completely or partly overlapping

with this targeted ob ject For the cases of ob jects with overlapping the storage space

overhead is high and it is time consuming in spatial reasoning Therefore to overcome



this problem a D Cstring is prop osed by Lee and Hsu A more ecient and economic

cutting mechanism is describ ed by employing a sound and characteristic set of spatial

op erators

Representation

Table shows the formal denition of the set of spatial op erators where the notation

b eginA denotes the value of b eginb ound of ob ject A and endA denotes the value

of endb ound of ob ject A According to the b eginb ound and endb ound of the picture

ob jects spatial relationships b etween two enclosing rectangles along the xaxis or y axis

can b e categorized into typ es ignoring their length Therefore There are typ es of

spatial relationships b etween two rectangles in D space as shown in Figure Basicallya

cutting of the D Cstring is p erformed at the p ointofpartlyoverlapping and it keeps the

former ob ject intact and partitions the latter ob ject The cutting mechanism is also suitable

for pictures with many ob jects Furthermore the endb ound p oint of the dominating ob ject

do es not partition other ob jects which contain the dominating ob ject Less cuttings and

no unnecessary cuttings in DC string will make the representation more ecient in the

case of overlapping as shown in Figure The corresp onding D Cstring is as follows

xstringf AB C jA D jD C D C

D C y stringf DBC AjAC

Spatial Reasoning

To solve the problem of how to infer the spatial relations along the xaxis or y axis

between two pictorial ob jects from a given D Cstring representation the level and rank

 

of a symb ol are used The rank of each symb ol in a D string prop osed by Chang et al Notation Condition Meaning

A < B end(A) < begin (B) A disjoins B A = B begin(A) = begin(B) A is the same as B end(A) = end(B) A | B end(A) = begin(B) A is edge to edge with B A % B begin(A) < begin(B) A contains B and they end(A) > end(B) have not the same bound A [ B begin(A) = begin(B) A contains B and they end(A) > end(B) have the same begin bound A ] B begin(A) < begin(B) A contains B and they end(A) = end(B) have the same end bound A / B begin(A) < begin(B) A is partly overlapping

< end(A) < end(B) with B

Table Denitions of Lees spatial op erators

Figure The spatial relationship typ es of twoobjects A A C B B C

D D

(a) (b)

Figure The cutting mechanism of the D Cstring a cut along the xaxis b cut

along the y axis

is dened to b e one plus the number of preceding this symbol in a xstring or y string

That is the rank values of ob jects stand for the relative sequence in the xstring or y string

representing the relative spatial p osition in the original symb olic picture Because the

D Cstring representation of symb olic pictures is constructed by employing more spatial



op erators R and R the ranks of pictorial ob jects need to b e redened

g l

i i

Supp ose s isasymbol of xy string the rank of s is denoted as Ranks k k

i i i

 

i i

k where l is the rank level of s denoted as Levels l k is the rank value of the

i i i i

l m

i

mth level of s m l The ranking technique is actually an enco ding metho d The

i i

level l of symbol s means the depth of nesting within a complex ob ject or a lo cal b o dy

i i

which is used in the following rank rules Following complex rank rules recursively the



rank of anysymb ol in a D Cstring can b e dened For instance the D Cstring along

the xaxis of Figure is as follows

AjB CDE F GHJ KL M

where is a pair of separators which is used to describ e a set of symb ols as one lo cal

body



Then the ranks and levels of these symb ols along the xaxis are as follows

RankA LevelA

RankB LevelB

RankC LevelC

RankD LevelD

RankE LevelE

RankF LevelF B

A E F K M G IJ L H

C D

Figure An example of D Cstring representation

RankG LevelG

RankH LevelH

RankI LevelI

RankJ LevelJ

RankK LevelK

RankL LevelL

RankM LevelM

The spatial knowledge is emb edded in the ranks of pictorial ob jects In fact the ranks

b ecome representative of the spatial knowledge of pictorial ob jects in an image All the

spatial relationships except can b e inferred by using the ranks Because the cutting

mechanism of a D Cstring is p erformed for the case of it is imp ossible for anytwo



segmented sub ob jects or symb ols to b e partly overlapping Toidentify the spatial rela

tionships along the xaxis or y axis b etween twosymb ols using their ranks six complex



computing rules are used For example assume that there are twosymbols s s in a

i j

j j j

i i i

picture Rank s rank k k k Lev el s l Rank s r ank k k k

i i i i j j

 

  l

l

i

j

Lev el s l and the rank values of the rst h levels of s are equal to those

j j i

j j j j

i i i i

In this case the follow k k k andk k k k of s ie k

j

 

h h  

h

h

ing computing rule determines whether a symbol s is edge to edge with a symbol s or

i j

j

i i

not If h minl l modk k and x h l k and

i j i

h x

h

j

then s js y h l k

i j j

y

Furthermore to answer the spatial relationship b etween two ob jects along the xaxis

or y axis which are segmented into subparts wehave to compare all subparts of the ob

jects In general according to the spatial relationship b etween these two ob jects b oundary

subparts there are four cases p ossible For each case up to two comparisons b etween the

leftmost or rightmost b ounding subpart of those two ob jects are needed to determine the



spatial relationship

Pictorial Query

The primitive direction relationships can b e inferred from the spatial op erators of D C

strings Four basic orthogonal directional aggregates are the main b o dy of a D Cquery

For example to determine whether A is in the of B the following aggregate is used x

x x    

Ar B r f j g i east A B In general the pictorial queries in a D

AB AB

Cquery can b e summarized and classied into seven classes For example in the category

of a relationship ob ject query to determine what ob jects overlap with X the following rule

is used par tov l p Xf Ajx par tov l p A X y disj oin A X y edgeAX

par tov l p A X x disj oin A X x edg e A X x mor e A X y less A X y

x l ess A X y mor e A X gwherex par tov l p y disj oin is one of D relationship

aggregates along the xaxis y axis Note that spatial reasoning is the basic work to the

pictorial query That is the r ank rules and computing rules are applied in a pictorial

querytoo

Similarity Retrieval

The target of similarity retrieve is to retrieval the images that are similar to the query

image Based on the maximumlikeliho o d or minimumdistance criterion a new denition



of typ ei similar pictures in D Cstrings is prop osed by Lee and Hsu Basically similarity

retrieval in D Cstring representation is pro cessed as follow First the representing D C

strings for the two pictures f and f are constructed By applying the reasoning rules the

 

spatial relationships and categories among ob jects are inferred According to the denition

of typ ei similar picture the set of typ ei similar pairs of picture f and f is constructed

 

Finally the maximal complete subgraphs of typ e typ e and typ e are found

DLT Matrix



Chang et al classify spatial relationship into nine classes according the xaxis and

y axis spatial relative information emb edded in the picture and suggest a nine direction

lowertriangular DLT matrix to represent a symb olic picture Let there b e nine direction

co des as shown in Figure which are used to represent relative spatial relationships among

ob jects In Figure R denotes the referenced ob ject represents at the same spatial 1

2 8

3 R 0 7

4 6

5 Figure The direction co des

ABCDE

A ----- D B 6---- T = C 6 7 - -- A E D 8 1 2 - - B C E 7 8 1 6 -

(a) (b)

Figure DLT representation a a symb olic picture b the related DLT matrix

lo cation as R represents north of R represents north of R represents

west of R and so on For the symb olic picture shown in Figure a Figure b is the

corresp onding DLT matrix

An Ecient Iconic Indexing Scheme for Symbolic

Pictures



In general Lee and Hsus algorithm for spatial reasoning based on D Cstrings can b e

summarized into the following three steps Following r ank rules recursively the rank

value of eachsymb ol is calculated Following computing rules the spatial relationships

between two symb ols are inferred To infer the spatial relationship b etween two par

titioned ob jects the b oundary of their subparts is compared Consequentlytoanswer a

pictorial query based on D Cstring representation a lot of steps are needed Therefore

Figure The spatial relationships of category disj oin

Figure The spatial relationships of category join

in this Section we prop ose a new iconic indexing scheme which can solve spatial queries

easier and more eciently By observing the spatial relationships in Figure we can

overlap classify them into ve spatial categories disjoin join contain belong and part

as shown in Figure and resp ectively According to these ve gures of

these spatial categories we discover some interesting and useful results A category rule is

derived for each spatial categoryFollowing those category rules we prop ose algorithms to

eciently supp ort spatial reasoning picture queries and similarity retrieval based on the

PrimeNumb erbased Matrix PN Matrix representation

Characteristics of Spatial Categories

Now we will describ e our observation of characteristics of those ve spatial categories as

follows Supp ose A and B are two ob jects in a picture f and the spatial relationship

y y

x x

between them in terms of xaxis and y axis is Ar B Ar B where r and r are

AB AB AB AB

the spatial op erators in Table

y

 x

spatial op erators are in f g Disjoin One or b oth the r r

AB AB

Figure The spatial relationships of category par t ov l p

Figure The spatial relationships of category contain

Figure The spatial relationships of category bel ong

y

x 

Join a None of the r r spatial op erators is in f g And b one or

AB AB

y

 x

r spatial op erators are in fj j g b oth the r

AB

AB

y

x

spatial op erators are in f g Contain Both the r r

AB AB

x x   

Belong Both the r r spatial op erators are in f g

AB AB

y

x 

overlap a One of the r r spatial op erators is in f g and the other Part

AB AB

y

    x

spatial op erators is in is in f g Or b one of the r r

AB AB

  

f g and the other is in f g

Tomake those category rules more clear we transform them into more formal descrip

y

x

B then B Ar tions Supp ose the spatial relationship b etween ob jects A and B is Ar

AB AB

the spatial category of A and B is describ ed as follows

y

x  

Disjoin r f gorr f g

AB AB

y

x   x 

Join a r f g and r f g and b r fj j gor

AB AB AB

y



r fj j g

AB

y

x

Contain r f gandr f g

AB AB

y

      x

f g f g and r Belong r

AB AB

y y

x  x    

Part Overlap a r r f g and r r f g

AB AB AB AB

y y

   x x

f g r f g and r r or b r

AB AB AB AB

Assignments of SpatialOp eratorValues SOVforSpa

tial Op erators

Based on the ab ove observation we can supp ort ecient spatial reasoning by making

use the PrimeNumb erbased Matrix Strategy The ma jor step is to assign each spatial

op erator a unique numb er according to these ve spatial categories Supp ose r is a spatial

     

op erator in the set f j j g and A B are two ob jects in the

symb olic picture We dene sov r asthespatialoperatorvalue of r with a initial value

y

x

and sv A B sov r sov r asthespatialvalue of the ob jects pair A B Here

AB AB

comes the steps of assignments

To classify the disj oin category the prime numb er is applied That is

sov sov

  

sov sov



Since only sov and sov are the multiples of one sv A B mo d op eration

y

x 

can determine whether one or b oth the r r spatial op erators are in the set f g

AB AB

That is one sv A B mo d op eration can determine the disj oin category

To classify the join category the prime numb er is applied That is

sov j sov j

  

sov j sov j



are the multiples of one sv A B mo d op eration Since only sov jand sov j

y

x 

can determine whether one or b oth the r r spatial op erators are in the set fj j g

AB AB

y

x 

Moreover none of the r r spatial op erator should b e in the set f gsoone

AB AB

sv A B mo d op eration is needed That is one sv A B mo d op eration and one

sv A B mo d op eration can determine the join category

To classify the contain category the prime numb er is applied Wemultiply to the

spatialop eratorvalue of the spatial op erators which are in the set f g That is

sov r sov r r f g



From the observation of the contain category one sv A B mod op eration can

y

x

determine whether b oth the r r spatial op erators are in the set f gThatis

AB AB



one sv A B mod op eration can determine the contain category But to distinguish

these four symb ols three more prime numbers must b e applied Therefore

sov sov

sov sov

sov sov

To classify the bel ong category the prime numb er is applied Wemultiply to the

  

spatialop eratorvalue of the spatial op erators in the set f gThatis

  

sov r sov r r f g



From the observation of contain categoryonesv A B mod op eration can deter

y

x   

mine whether b oth the r r gThatis spatial op erators are in the set f

AB AB



one sv A B mod op eration can determine the contain category But to distinguish

these four symb ols three more prime numbers must b e applied Therefore

 

sov sov

 

sov sov

 

sov sov

ov er l ap category lets consider the following two cases stated b e To classify the par t

y

x 

fore First to determine whether one of the r r spatial op erators is in the set f g

AB AB

   

and the other is in the set f gtwo prime numb ers and are

applied That is

sov sov

  

sov sov

   

sov r sov r r f g



Therefore one sv A B mod op eration can determine whether one of the

y

x 

r r spatial op erators is in the set f g and the other is in the set f

AB AB

     

g That is one sv A B mo d op eration can determine the rst

overlap category case of the par t

y

x

Second to determine whether one of the r r spatial op erators is in the set

AB AB

  

f g and the other is in the set f gtwo prime numb ers and are applied

That is

sov r sov r r f g

  

sov r sov r r f g

Therefore one sv A B mo d op eration can determine whether one of the

y

x   

r r spatial op erators is in the set f g and the other is in the set f g

AB AB

That is one sv A B mod op eration can determine the second case of the



ov er l ap category Consequentlyonesv A B mo d op eration and one par t

sv A B mo d op eration can determine the par t ov er l ap category

According to the ab ove descriptions wehave assigned each spatial op erator a unique

value which can b e used to determine dierent spatial categories ecientlyHowever in

order to determine the spatial relationships b etween anytwo ob jects ecientlywehave

to makeeach of the spatialop eratorvalues of these spatial op erators indivisible by the

spatialop eratorvalue of any other spatial op erator That is no one spatialop eratorvalue

 

 

j j







 

Figure The assignments of those spatial op erators

Disjoin if sv A B mod

Join if sv A B mod and sv A B mod



Contain if sv A B mod



Belong if sv A B mod



overlap if sv A B mo d or sv A B mod Part

Figure Category rules

of a spatial op erator is a multiple of the spatialop eratorvalue of any other spatial op erator



Therefore we let sovsov to distinguish spatial op erators and let sovj



sovj to distinguish spatial op erators j and j andletsov sov to



distinguish spatial op erators and Finally the assignments of spatialop eratorvalues

for these spatial op erators are shown in Figure and the ve category rules based on

the mo dule op eration are shown in Figure

Data Structure for Pictorial Symb ol Representation The

PrimeNumb erbased Matrix PN Matrix

In the DLT matrix representation the spatial relationships b etween each ob ject pairs

are obvious Thus spatial reasoning and pictorial query could work ecientlyHowever

it is conspicuous that a DLT matrix concluding the spatial relationships b etween two

ob jects into nine typ es is insucient Conversely the D Cstring represents a picture

more precisely since it concludes the spatial relationships into typ es However spatial

reasoning based on the D Cstring representation is not so straightforward Therefore

we prop ose a PrimeNumb erbased Matrix PN Matrix strategy to preserve the spatial

information in the D Cstring representation by using an extended DLT matrix which

can supp ort to answer the spatial relationship directly and supp ort pictorial query and

similarity retrieval easily

Supp ose a picture f contains m ob jects and let V fv v v gLetA b e the set

  m

     

of spatial op erators f j j gAm m spatial matrix S of

picture f is dened as follows

v v v v

  m m

y y

v r r



 m

x

v r





S

y

v r

m

mm

x x

v r r

m

m mm

where the lower triangular matrix stores the spatial information along the xaxis and

the upp er triangular matrix stores the spatial information along the y axis That is

y

x

if i j S v v r if i j S v v ifi j v v V S v v r

i j i j i j i j

ij

ji

y

x x

r r A i j m where r is the spatial op erator b etween ob jects v and v

i j

ij

ji ji

y

is the spatial op erator b etween ob jects v and v along the y axis along the xaxis and r

i j

ij

Note that in this representation wealways record the relationships b etween two ob jects v

i

and v from the view p oint of ob ject v no matter along the xaxis or the y axis where i

j i

x

j That is why S v v r when ij

i j

ji

For the pictorial picture shown in Figure the corresp onding spatial matrix S is

shown as follows

A B C D E

 

A j

  

B

S

 

C j

  

D

  

E

According to the assignments of spatialop eratorvalues for those spatial op erators

describ ed b efore we can transform the spatial matrix S of f into a PN Matrix T by

y

x

with its unique spatialop eratorvalue as follows replacing each spatial op erator r r

ij

ji D B A C

E

Figure An image and its symb olic representation

A B C D E



A

 

B

T



C

  

D

 

E

Spatial Reasoning Based on the PN Matrix Representation

Spatial reasoning means the inference of a consistent set of spatial relationships among the

ob jects in an image Based on the PN Matrix it is easy to retrieve the spatial relationships

of each pair of ob jects along the xaxis and y axis straightforwardly since this information

is recorded directly in the matrix Moreover the category of each pair of ob jects can b e

inferred by following the category rules as shown in Figure in which only one or two

mo dule op erations on the sv value the spatial value are needed

Pictorial Query Based on the PN Matrix Representation

A pictorial query allows the users to query images with a sp ecied spatial relationship

For example display all images with a lakeeastofa mountain In this Section we will

describ e the pictorial query pro cessing based on the PN Matrix representation according



to the query typ es classied by Lee and Hsu First the following basic orthogonal

directional aggregates are the main b o dy of a PN Matrix query

x    x  

x east A B i r f j g ie i sov r f

AB AB



g

y y

    

y north A B ir f j g ie i sov r f

AB AB



g

x x

xwest A B ir f j g ie i sov r f

AB AB

g

y y

y A B i r f j g ie i sov r f

AB AB

g

Therefore the primitive direction relationship problem b ecomes a memb ership checking

of a set of numb ers Then the pictorial queries based on a PN Matrix can b e pro cessed as

follows

A Orthogonal direction object queries

For this class of queries we still have to follow the same spatial rules as describ ed



By Lee and Hsu For example to determine which ob ject is in the eastnorth of

ob ject X the following rule is used ne X f Ajnor th A X east A X g

B Category relationship object queries

This class of queries allows the users to retrieve the ob jects with a sp ecied category

and ob ject for example nd those ob jects which are disjoin with ob ject A Based

on PN Matrix representation this class of queries can b e easily answered by applying

the mo dule op eration as shown in Figure where the constant B is replaced with

avariable X to denote the unknown ob ject

C Auxiliary relationship object queries

This class of queries allows the users to retrieve the ob jects with a sp ecied auxil

iary relationship and an ob ject where auxiliary relations contain same surround and

surroundandtwosymmetricinverse relationships surrounded and part surrounded part

We can make use of sov of spatial op erators to pro cess this class of queries

a A is the same as B ifA is at the same lo cation as B along western eastern

x

southern and northern directions That is A is the same as B i b oth r and

AB

y

  

r are ie sv A B mod

AB

b A surrounds B ifA contains B andA completely surrounds B along four

y

x

and r are ie orthogonal directions That is A surrounds B i b oth r

AB

AB

   

sv A B mod

c A part surrounds B ifA contains B and A surrounds B along two or three

orthogonal directions That is A part surrounds B i A contains B A is not



the same as B and A do es not surround B ie sv A B mod sv A B

      

mo d and sv A B mod

D Icon relationship object queries

This typ e of queries allow the users to retrieve all ob jects with a sp ecied icon in

Figure For example to retrieve the ob jects whose spatial relationships with ob ject

A along xaxis and y axis are and j resp ectively a set of ob jects S will b e retrieved

y

x

where S f B j sov r mod and sov r mod g

AB AB

E Icon relationship queries

This typ e of queries allow the users to retrieve the spatial relation icon in Figure

with two sp ecied ob jects for example nd the spatial relationship icon of ob jects

A and B It is clear that the work is similar to the ab ove query class D

F Category relationship queries

This typ e of queries can answer the spatial category according to two given ob jects

To nd the spatial category with ob jects A and B category rules describ ed in Section



are used For example A contains B i sv A B mod

G Orthogonal direction queries

Finally the orthogonal spatial relationship b etween ob jects A and B can b e examined

The four orthogonal directional aggregates describ ed ab ove are used

Similarity Retrieval based on the PN Matrix Representa

tion

The target of similar retrieval is to retrieve the images that are similar to the query image

In this Section we will describ e the similar typ es and the corresp onding similarity retrieval

algorithm based on our PN Matrix strategy Basicallywe will showhow those similar



typ es which are dened in the D Cstring representation can b e determined based on



our PN Matrix representation Now the denitions of the similar typ es are describ ed

as follows

 

Denition Picture f is a typei unit pictureoff iff is a picturecontaining the two





y

x

B A and B are also contained B y Ar objects A and B representedasx Ar

AB AB

x

in f and the relationships between A and B in f arerepresentedas x Ar B andy

AB

y

Ar B then

AB





y y

x x

type C ateg or y r r C ateg or y r r

AB AB AB AB





y y

x x

r or r type type and r r

AB AB AB AB





y y

x x

type r r and r r

AB AB

AB AB

y

x

where Categoryr r denotes the relationship category of the spatial relationship as

AB AB

shown in Table

Denition Given a matrix M a matrix operator R is dened as fol lows



R

Let M M where M i j M i j M j i i m ji

  



Denition Given two matrices M and M a matrix operator is dened as fol lows

 



Let M M M where M i j M i j M i j j i n

   

Denition Given a m m PN Matrix T the corresponding category matrix C is dened

as fol lows C i j ifT i j T j i mod C i j ifT i j T j i mod



and T i j T j i mod C i j ifT i j T j i mod C i j

 

ifT i j T j i mod C i j ifT i j T j i mod

or T i j T j i mod i m ji That is C i j

if the relationship between objects v and v is of the disjoin join contain belong and

i j

overlap category respectively by fol lowing the category rules part



Based on these two new matrix op erators R and the following three algorithms

type type type are used to determine whether two pictures are of typ e typ e

typ e similarity resp ectivelygiven two PN Matrices T and T

 

Algorithm typ e

 R  R

T T T T

 

  A C A C B B

D D

f1 f2

Figure One example of pictures f and f

 

Fol lowing the category rules nd the category matrix C and C representing the two

 

pictures f and f respectively

 



C C C IfC is zero in the lower triangular matrix these two pictures areof

 

type similarity otherwise there is no match

Algorithm typ e

Algorithm type passed

T T T

 

 R

T T



If T is zero in the lower triangular matrix these two pictures areoftype similarity

otherwise there is no match

Algorithm typ e

T T T IfT is zero these two pictures areoftype similarity otherwise there

 

is no match

Now we use one example to showhow those algorithms work Consider the pictures

as shown in Figure

Step Find the spatial matrices S and S and the PN matrices T and T representing

   

the two pictures f and f resp ectively

 

A B C D

  

A



S B





C



D

A B C D

  

A



T B





C

D

A B C D

 

A



S B j





C

 

D

A B C D

 

A



T B





C

D

 

Step Calculate T and T

 

A B C D

A

 R

 

B T T







C

  

D

A B C D

A

 R

 

T T B





C

  

D

Step Following the category rules compute the corresp onding category matrices

ov er l ap where and mean the disj oin join contain bel ong and par t

relationship resp ectively

A B C D A B C D

A A

B B

C C

 

C C

D D

Step Checktyp e similaritySinceC in the lower triangular matrix these two

pictures are of typ e similarity

A B C D

A



B C C C

 

C

D

Step Checktyp e similaritySinceT in the lower triangular matrix these two

pictures are of typ e similarity

A B C D

A

R

T T T B

 

C

D

Step Checktyp e similarity Since T these two pictures are not of typ e

similarity

A B C D

A

B

T T T

 

C

D

A Comparison

In this subsection wemake a comparison of our prop osed strategy and the previous pro

p osed representation strategies whichisasshown in Table

Table A Comparison

The rst item considers the data structure to represent the spatial relationships Only

the DLT matrix representation and the PN matrix representation record the spatial rela

tionships in matrices In this way ecient matrix op erations are applied instead of complex

string comparison strategies used for string representation The second item considers the

contents of representation Only DLT matrix representation records the relationship in D

space instead of the relationship along the xaxis and the y axis recorded in other prop osed

strategies Recording spatial relationship in D space makes pictorial query easy however

the DLT matrix strategy records only spatial relationships The third item concerns

ab out whether a cutting mechanism is applied Cutting mechanisms are applied in D

Gstring and D Cstring representation strategies to handle overlapping ob jects Since

cuttings make an ob ject into more than one comp onents more symb ols will b e used which

results in a requirement of large storage space and complex query pro cessing strategies

Although the D string and DLT matrix representations do not need cutting mechanisms

less spatial relationships could b e recorded While in our prop osed strategywedonot

need the cutting mechanisms but we still can handle the cases of overlapping ob jects The

forth item shows the numb er of spatial relationships in each of prop osed strategies Only

spatial relationships can b e recorded in the D string and DLT matrix representations

Although the D Gstring and D Cstring representation strategies can record spatial

relationships they need cutting mechanisms Our prop osed strategy is the only one that

can record spatial relationships without cutting

The fth item concerns ab out whether it is easy or dicult to do spatial reasoning

where spatial reasoning means the inference of a consistent set of spatial relationships

among the ob jects in an image In general if a cutting mechanism is used in the represen

tation then it is hard to do spatial reasoning as describ ed in Section The sixth item

considers the way to pro cess a pictorial query where a pictorial query allows the users to

query images with a sp ecied spatial relationship The D string representation strategy

pro cesses a pictorial query by using string matching The D Cstring representation strat

egy pro cess pictorial queries by using complex rank rules and computing rules The DLT

matrix representation strategy pro cesses pictorial queries by using matrix minus op erations

Our prop osed strategy pro cess pictorial queries by using the mo dule op eration whichis

more ecient as compared to the strategies used in the other representations The last

item shows the numb er of similar typ es which a representation strategy can distinguish

Basically our prop osed strategy can distinguish the same three similar typ es dened in the

D Csting representation strategy

Conclusion

Picture indexing techniques are needed to make ease pictorial information retrieval from

a pictorial database In this pap er wehave prop osed an ecient iconic indexing scheme

called PrimeNumberbased Matrix PN Matrix for symb olic pictures whichcombines the

advantages of the D Cstring and the DLT matrix In th prop osed scheme wehave

designed each spatial op erator a unique value whichisaproductofsomeprimenumbers

and derived ve category rules Since those category rules are mo duleop erationbased

they are ecient enough as compared to the previous approaches Wehave also prop osed

a PrimeNumb erbased Matrix data structure to represent pictorial data in which the

relationship b etween two ob jects is recorded obviously Consequently spatial reasoning can

b e done very straightforwardlyFor answering a pictorial query some mo dule op erations or

memb ership checking of a set of numb ers have b een applied In similarity retrieval some

new matrix op erations have b een applied A picture is dened to b e ambig uous if there

exists more than one dierent reconstructed picture from its representation How to handle

the problem of an ambiguous picture is one future research direction Furthermore how

to eciently handle large amounts of image databases is also an imp ortant future research

direction

References

S K Chang C W D C Dimitro and T Arndt An intelligent image database system

IEEE Trans on Software Eng May

S K Chang Q Y Shi and C W Yan Iconic indexing by D strings IEEE Trans on

Pattern Analysis and Machine Intel ligence PAMI May

G Costagliola F Ferrucci G Tortora and M Tucci Corresp ondences nonredundantD

strings IEEE Trans on Know ledge and Data Engineering April

S Y Lee and F J Hsu D Cstring a new spatial knowledge representation for image

database systems Pattern Recognition Oct

P W Huang and Y R Jean Using D C string as spatial knowledge representation for

image database systems Pattern Recognition Sept

S Y Lee M K Shan and WPYang Similarity retrieval of iconic image database Pattern

Recognition June

S S Liaw A new scheme for retrieval similar pictures in a pictorial database Journal of

Computers Dec

E Jungert Extended symb olic pro jections as a knowledge structure for spatial reasoning

th BPRA Conf on Pattern Recognition Springer Cambridge March

E Jungert and S K Chang An algebra for symb olic image manipulation and transformation

T S Kunii ed pp NorthHolland Amsterdam

S K Chang E Jungert and Y Li Representation and retrieval of symb olic pictures using

generated D strings Technical Report Universal of Pittsburgh PA

C C Chang Spatial matchretrieval of symb olic pictures Journal of Information Science

and Engineering Dec

C C Chang and J C Lin A fast spatial match accessing scheme for symb olic pictures

Journal of Electrical Engineering June

C C Chang and H C Wu A mo duleoriented signature extraction to retrieve symb olic

pictures Journal of Computer Dec

S Y Lee and F J Hsu Spatial reasoning and similarity retrieval of images using D Cstring

knowledge representation Pattern Recognition March