Data Exchange with Data-Metadata Translations ¤ y z z Mauricio A. Hernandez´ Paolo Papotti Wang-Chiew Tan IBM Almaden Research Center Universita` Roma Tre UC Santa Cruz [email protected] [email protected] [email protected] ABSTRACT of source-to-target tuple generating dependencies (s-t tgds) [6] or Data exchange is the process of converting an instance of one schema (Global-and-Local-As-View) GLAV mappings [7, 12]. They have into an instance of a different schema according to a given speci- also been extended to specify the relation of pairs of instances of fication. Recent data exchange systems have largely dealt with the nested schemas in [8, 18]. case where the schemas are given a priori and transformations can Data exchange systems [9, 13, 14, 21, 22] have been developed only migrate data from the first schema to an instance of the sec- to (semi-) automatically generate the mappings and the transfor- ond schema. In particular, the ability to perform data-metadata mation code in the desired language by mapping schema elements translations, transformation in which data is converted into meta- in a visual interface. These frameworks alleviate the need to fully data or metadata is converted into data, is largely ignored. This understand the underlying transformation language (e.g. XQuery) paper provides a systematic study of the data exchange problem and language-specific visual editor (e.g. XQuery editor). Further- with data-metadata translation capabilities. We describe the prob- more, some of these systems allow the same visual specification of lem, our solution, implementation and experiments. Our solution is mapping schema elements to be used to generate a skeleton of the a principled and systematic extension of the existing data exchange transformation code in diverse languages (e.g., Java, XSLT). framework; all the way from the constructs required in the visual Past research on data exchange, as well as commercial data ex- interface to specify data-metadata correspondences, which natu- change systems , have largely dealt with the case where the schemas rally extend the traditional value correspondences, to constructs re- are given a priori and transformations can only migrate data from quired for the mapping language to specify data-metadata transla- the first instance to an instance of the second schema. In partic- tions, and algorithms required for generating mappings and queries ular, data-metadata translations are not supported by these sys- that perform the exchange. tems. Data-metadata translations are transformations that convert data/metadata in the source instance or schema to data/metadata in the target instance or schema. Such capabilities are needed in many 1. INTRODUCTION genuine data exchange scenarios that we have encountered, as well Data exchange is the process of converting an instance of one as in data visualization tools, where data are reorganized in differ- schema, called the source schema, into an instance of a different ent ways in order to expose patterns or trends that would be easier schema, called the target schema, according to a given specifica- for subsequent data analysis. tion. This is an old problem that has renewed interests in recent A simple example that is commonly used in the relational setting years. Many data exchange related research problems were inves- to motivate and illustrate data-metadata translations is to “flip” the tigated in the context where the relation between source and target StockTicker(Time, Company, Price) table so that company names instances is described in a high-level declarative formalism called appear as column names of the resulting table [15]. This is akin to schema mappings (or mappings) [10, 16]. A language for map- the pivot operation [23] used in spreadsheets such as Excel. After pings of relational schemas that is widely used in data exchange, as a pivot on the company name and a sort on the time column, it well as data integration and peer data management systems, is that becomes easier to see the variation of a company’s stock prices and also compare against other companies’ performance throughout the ¤Work partly funded by U.S. Air Force Office for Scientific Re- day (see the table on the right below). search under contract FA9550-07-1-0223. y Time Symbol Price Time MSFT IBM Work done while visiting UC Santa Cruz, partly funded by an 0900 MSFT 27.20 0900 27.20 - IBM Faculty Award. 0900 IBM 120.00 0900 - 120.00 zWork partly funded by NSF CAREER Award IIS-0347065 and 0905 MSFT 27.30 0905 27.30 - NSF grant IIS-0430994. Observe that the second schema cannot be determined a priori Permission to make digital or hard copies of portions of this work for personal or classroom use is granted without fee provided that copies since it depends on the first instance and the defined transformation. Permissionare not made to copy or distributed without fee for all orprofit part or of thiscommercial material isadvantage granted provided and Such schemas are called dynamic output schemas in [11]. Con- thatthat the copies copies bear are notthis madenotice or and distributed the full for citation direct commercialon the first advantage, page. ceivably, one might also wish to unpivot the right table to obtain theCopyright VLDB copyrightfor components notice and of thethis title work of theowned publication by others and than its date VLDB appear, the left one. Although operations for data-metadata translations andEndowment notice is must given be that honored. copying is by permission of the Very Large Data have been investigated extensively in the relational setting (see, for BaseAbstracting Endowment. with credit To copy is permitted. otherwise, To or copy to republish, otherwise, to to post republish, on servers instance, [24] for a comprehensive overview of related work), this orto topost redistribute on servers to lists,or to requires redistribute a fee and/orto listsspecial requires permission prior specific from the subject is relatively unexplored for data exchange systems in which publisher,permission ACM. and/or a fee. Request permission to republish from: VLDBPublications ‘08, August Dept., 24-30, ACM, 2008, Inc. Auckland, Fax New+1 Zealand(212) 869-0481 or source or target instances are no longer “flat” relations and the re- [email protected]. 2008 VLDB Endowment, ACM 000-0-00000-000-0/00/00. lationship between the schemas is specified with mappings. PVLDB '08, August 23-28, 2008, Auckland, New Zealand Copyright 2008 VLDB Endowment, ACM 978-1-60558-305-1/08/08 260 Extending the data exchange framework with data-metadata trans- Source: Rcd Target: Rcd (a) lation capabilities for hierarchical or XML instances is a highly Sales : SetOf Rcd CountrySales : SetOf Rcd country country non-trivial task. To understand why, we first need to explain the region Sales : SetOf Rcd data exchange framework of [18], which essentially consists of style style three components: shipdate shipdate units units ² A visual interface where value correspondences, i.e., the re- price id lation between elements of the source and target schema can be manually specified or (semi-)automatically derived with a for $s in Source.Sales (b) exists $c in Target.CountrySales, $cs in $c.Sales schema matcher. Value correspondences are depicted as lines where $c.country = $s.country and $cs.style = $s.style and between schema element in the visual interface and it provides $cs.shipdate = $s.shipdate and $cs.units = $s.units and an intuitive description of the underlying mappings. $c.Sales = SK[$s.country] ² A mapping generation algorithm that interprets the schemas “For every Sales tuple, map it to a CountrySales tuple where Sales are grouped by country in that tuple.” and values correspondences into mappings. (c) ² A query generation algorithm that generates a query in some Sales CountrySales country Sales language (e.g., XQuery) from the mappings that are generated country region style shipdate units price USA East Tee 12/07 11 1200 USA style shipdate units id in the previous step. The generated query implements the spec- USA East Elec. 12/07 12 3600 Tee 12/07 11 ID1 ification according to the mappings and is used to derive the USA West Tee 01/08 10 1600 Elec. 12/07 12 ID2 target instance from a given source instance. (Note that the UK West Tee 02/08 12 2000 Tee 01/08 10 ID3 framework of [14] is similar and essentially consists of only the country Sales UK style shipdate units id second and third components.) Tee 02/08 12 ID4 Adding data-metadata translation capabilities to the existing data exchange framework requires a careful and systematic extension to Figure 1: Data-to-Data Exchange all three components described above. The extension must capture traditional data exchange as a special case. It is worth pointing out that the visual interface component described above is not peculiar 2.1 Data-to-Data Translation (Data Exchange) to [18] alone. Relationship-based mapping systems [20] consist of Data-to-data translation corresponds to the traditional data ex- a visual interface in which value correspondences are used to in- change where the goal is to materialize a target instance according tuitively describe the transformation between a source and target to the specified mappings when given a source instance. In data-to- schema and in addition to [18], commercial mapping systems such data translation, the source and target schemas are given a priori. as [13, 21, 22] all fall under this category. Hence, our proposed ex- Figure 1 shows a typical data-to-data translation scenario. Here, tensions to the visual interface could also be applied to these map- users have mapped the source-side schema entities into some tar- ping systems as well. The difference between mapping systems get side entities, which are depicted as lines in the visual interface.