PREPRINT Please dont distribute To app ear in Vittorio Castelli and Lawrence Bergman eds Image Databases Search and Retrieval of Digital Imagery John Wiley and Sons Database Supp ort for Multimedia Applications Michael OrtegaBinderb erger Kaushik Chakrabarti University of Illinois at UrbanaChampaign Sharad Mehrotra University of California at Irvine August Intro duction Advances in high p erformance computing communication and storage technologies as well as emerging largescale multimedia applications have made the design and development of multi media information systems one of the most challenging and imp ortant directions of research and development within computer science The payos of a multimedia infrastructure are tremendous it enables many multibilli on dollarayear application areas Examples are medical information sys tems electronic commerce digital libraries like multimedia data rep ositories for training educa tion broadcast and entertainment sp ecial purp ose databases such as facengerprint databases for security and geographical information systems storing satellite images maps etc An integral comp onent of the multimedia infrastructure is a multimedia database management system Such a system supp orts mechanisms to extract and represent the content of multimedia ob jects provides ecient storage of the content in the database supp orts contentbased queries over multimedia ob jects and provides a seamless integration of the multimedia ob jects with the traditional information stored in existing databases A multimedia database system consists of multiple comp onents which provide the following functionalities Multimedia Ob ject Representation techniquesmo dels to succinctly represent b oth structure and content of multimedia ob jects in databases Content Extraction mechanisms to automaticallysemiautomatically extract meaningful features that capture the content of multimedia ob jects and that can b e indexed to supp ort retrieval Multimedia Information Retrieval techniques to match and retrieve multimedia ob jects based on the similarity of their representation ie similaritybased retrieval Multimedia Database Management extensions to data management technologies of in dexing and query pro cessing to eectively supp ort ecient contentbased retrieval in database management systems Many of the ab ove issues have b een extensively addressed in other chapters of this b o ok Our fo cus in this chapter is on how contentbased retrieval of multimedia ob jects can b e integrated into database management systems as a primary access mechanism In this context we rst explore the PREPRINT Please dont distribute supp ort provided by existing ob jectoriented and ob jectrelational systems for building multimedia applications We then identify limitations of existing systems in supp orting contentbased retrieval and summarize approaches prop osed in the literature to address these limitations We b elieve that this research will culminate in improved data management pro ducts that supp ort multimedia ob jects as rstclass ob jects capable of b eing eciently stored and retrieved based on their internal content The rest of the chapter is organized as follows In Section we describ e a simple mo del for contentbased retrieval of multimedia ob jects which is widely implemented and commonly supp orted by commercial vendors We use this mo del throughout the chapter to explain the issues that arise in integrating contentbased retrieval into database management systems DBMSs In Section we explore how the evolution of relational databases into ob jectoriented and ob ject relational systems which supp ort complex data typ es and userdened functions facilitates building multimedia applications We apply the analysis framework of Section to the Oracle the Informix and the IBM DB database systems in Section The chapter then identies limitations of existing stateoftheart data management systems from the p ersp ective of supp orting multimedia applications Finally Section outlines a set of research issues and approaches that we b elieve are crucial for the development of database technology providing seamless supp ort for complex multimedia information A Mo del for ContentBased Retrieval Traditionally contentbased retrieval from multimedia databases was supp orted by describing mul timedia ob jects with textual annotations Textual information retrieval tech niques were then used to search for multimedia information indirectly using the annotations Such a textbased approach suers from numerous limitations including the imp ossi bility of scaling it to large data sets due to the high degree of manual eort required to pro duce the annotations the diculty of expressing visual content eg texturepatterns or shap e in an image using textual annotations and the sub jectivity of manually generated annotations To overcome several of these limitations a visual featurebased approach has emerged as a promising alternative as is evidenced by several prototyp e and commercial systems In a visual featurebased approach a multimedia ob ject is represented using visual prop erties for example a digital photograph may b e represented using color texture shap e and textual features Typically a user formulates a query by providing examples and the system returns the most similar ob jects in the database The retrieval consists of ranking the similarity b etween the featurespace representations of the query and of the images in the database The query pro cess can therefore b e describ ed by dening the mo dels for ob jects queries and retrieval Ob ject Mo del A multimedia ob ject is represented as a collection of extracted features Each feature may have multiple representations capturing it from dierent p ersp ectives For instance the color his togram descriptor represents the color distribution in an image using value counts while the color moments descriptor represents the color distribution in an image using statistical pa rameters eg mean variance and skewness Asso ciated with each representation is a similarity function that determines the similarity b etween two descriptor values Dierent representations capture the same feature from dierent p ersp ectives The simultaneous use of dierent represen tations often improves retrieval eectiveness but it also increases the dimensionality of the PREPRINT Please dont distribute Query O ith Ob ject i H H W imp ortance of the ith i H W W 1 2 H Ob ject relative to H Hj the other query Ob jects O O W imp ortance of Feature j i;j 1 2 of Ob ject i relative to W W W W Feature j of other Ob jects 11 12 21 22 R R R Representation of i;j Feature j of Ob ject i R R R R 11 12 21 22 Figure Query Mo del search space which reduces retrieval eciency and has the p otential for intro ducing redundancy which can negatively aect eectiveness Each feature space eg a color histogram space can b e viewed as a multidimensional space in which a feature vector representing an ob ject corresp onds to a p oint A metric on the feature space can b e used to dene the dissimilarity b etween the corresp onding feature vectors Distance 1 values are then converted to similarity values Two p opular conversion formulae are s d and 2 d where s and d resp ectively denote similarity and distance With the rst formula s exp 2 if d is measured using the Euclidean distance function s b ecomes the cosine similarity b etween the vectors while if d is measured using the Manhattan distance function s b ecomes the histogram intersection similarity b etween them While cosine similarity is widely used in keywordbased do cument retrieval histogramintersection similarity is common for color histograms A numb er of image features and feature matching functions are further describ ed in Chapters to Query Mo del The query mo del sp ecies how a query is constructed and structured Much like multimedia ob jects a query is also represented as a collection of features One dierence is that a user may simultaneosly use multiple exampleob jects in which case the query can b e represented in either of the following two ways Featurebased representation The query is represented as a collection of features Each feature contains a collection of feature representations with multiple values The values corresp ond to the feature descriptors of the ob jects Ob jectbased representation A query is represented as a collection of ob jects and each ob ject consists of a collection of feature descriptors In either case each comp onent of a query is asso ciated with a weight indicating its relative imp ortance Figure shows a structure of a query tree in an ob jectbased mo del In the gure the query structure consists of multiple ob jects O and each ob ject is represented as a collection of multiple i feature values R ij 1 The conversion formula assumes that the space is normalized to guarantee that the maximum distance b etween p oints is equal to PREPRINT Please dont distribute Retrieval Mo del The retrieval mo del determines the similarity b etween a query tree and ob jects in the database The leaf level of the tree corresp onds to feature representations A similarity function sp ecic to a given representation is used to evaluate the similarity b etween a leaf no de R and the ij corresp onding feature representation of the ob jects in the database Assume for example that the leaf no des of a query tree corresp ond to two dierent color representations color histogram and color moments While histogram intersection may b e used