DOCSLIB.ORG

Managing Data Quality, Transformations, and Loading Using an XML-Driven Engine

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Managing Data Quality, Transformations, and Loading Using an XML-Driven Engine Managing Data Quality, Transformations, and Loading Using an XML-Driven Engine Bill SAVAGE, Ying QIN, Michal ZMUDA, David HALL, Laxminarayana GANAPATHI, Sheping LI RTI International, 3040 Cornwallis Road, Research Triangle Park, NC 27709 {bsavage, yingqin, mzmuda, dhall, lganapathi, shepli}@rti.org and Atif HASAN, Thomas BAKER Alpha-Gamma Technologies, Inc., 4700 Falls of Neuse Road, Suite 350, Raleigh, NC 27609 {tbaker, ahasan}@alpha-gamma.com ABSTRACT support the Breast and Colon Cancer Family Registries Computer systems commonly require data to be loaded program. A core component of the Support Center is the into a database from external sources. Incoming data data integration process that loads monthly data must be examined to ensure they meet acceptable quality submissions from multiple cancer research centers in levels, are complete, and conform to required formats. Australia, Canada, and the United States. Submissions Data must often be integrated from multiple sources, contain study data defined by multiple data dictionaries increasing the complexity of the process. Data validation that contain listings of all data elements contained within and loading processes have been traditionally referred to each data file, including name, description, length, data as Extract-Transform-Load (ETL); alternately known as type, and often complex validation rules. The data are data integration. In this paper we present a new data transmitted in multiple file formats such as fixed fields, integration system developed to assemble, validate, and comma-separated values (CSV), and tab delimited. Every load data, with support for complex and diverse submitted data file must be fully documented, validated, validation requirements. This reusable, customizable tracked, and then loaded into an Oracle 10g database to system offers improvements over many systems in support further analysis and research. several areas. The system is based on Extensible Markup We discovered that the submitted data records, the Language (XML) constructs that define: data structures validation requirements, and even the file structures are in use by multiple submitting sources (data dictionaries); subject to frequent and sometimes unpredictable change. complex validation rules; mappings to target objects; and The dynamic nature of the situation informed the RTI target object structures. Java program components team’s decision to implement a data integration solution implement a self-contained data integration engine that based on easily configurable data dictionary can be a standalone system or embedded within a web specifications, including validation rules, that drive the site. data integration process. Keywords: Data Dictionary, Data Integration, Data The team determined that not only would machine- Validation, ETL, Java, XML. readable specifications support the complex needs of the process, but would also enable generation of user 1. INTRODUCTION documentation, or other forms for use by external Many computer systems depend on a process to load data systems. This dynamic, configurable approach is most into a database from external systems or sources. responsive to all needs of the project, including the Incoming data must be assembled, checked for evolution of study data and validation requirements completeness and conformance to required formatting, across and within the centers that submit the data. and finally checked to ensure they meet defined quality To meet the data integration requirements, the team had specifications. If the data must be integrated from to consider either developing a solution based on using multiple heterogeneous sources, processing complexity is an existing data integration package or developing a significantly increased. The data validation and loading custom solution using open source technologies where process is often referred to as Extract-Transform-Load possible. (ETL) or data integration. We have created a new data integration system that supports complex source 2. EVALUATION OF TECHNOLOGIES assemblies and validation requirements, using standard, The development team identified and investigated open source technologies. existing software that it could potentially use, and In 2005, RTI International (RTI) began developing the considered both open source and commercial packages. National Cancer Institute’s Informatics Support Center to The review process was informal, and involved several team members. The primary evaluation criteria included be deployed to different operating environments if requirements for: needed. · a highly configurable system; 3. SYSTEM DESIGN & DEVELOPMENT · the ability to plug into a Java-based web portal; Throughout development, the team followed industry- · response time suitable for interactive sessions; standard procedures to manage development in an environment with changing requirements. Data model · operation based on a data dictionary specified in a development utilized Computer Associates’ ERwin® standard format/language; Data Modeler product. The open source Concurrent · reading fixed length and delimited file formats; and Versions System (CVS) system was chosen as the source · supporting both standard and highly customized control system, and the open source Bugzilla system was types of data validation. used to manage and track issues. Software testing used Leading commercial systems such as IBM’s DataStage Selenium, JUnit, and DbUnit. [1] and Informatica’s PowerCenter [2] were considered. Frameworks for the data integration engine include J2SE Open source systems such as Enhydra Octopus [3], 1.5, Spring and Hibernate. Spring is an increasingly clover.ETL [4], and Xineo XIL [5] were also reviewed. important Java integration technology, often the choice We determined that currently available open source for stand-alone and web applications. It has many systems often lacked the ability to handle fixed-length architectural benefits, including effectively organizing file formats and sufficiently customized data validation. middle-tier objects, reducing the proliferation of Commercial systems were generally unable to plug into singletons, eliminating the need to use a variety of Java in the required manner. For some, the client custom properties for file formats, and using a consistent software was only available for the Windows operating XML-based approach. Hibernate is a high performance system, and the software was, in general, prohibitively object-relational persistence and query service. expensive. Data Integration Engine Processing Based on these and other considerations, we decided to Our design decouples the system’s base functionality develop a new data integration package with XML code from specific rules concerning data loading and utilized throughout for data dictionary specifications, validation constraints. We implemented this design using validation rules, and source-to-target mappings. XML XML metadata files of our design. These files govern was selected based on its wide acceptance throughout the engine operation and provide flexibility to handle computing industry and its flexibility in supporting many changing requirements. The formats of data sources and different types of specifications. targets, data file parsing information, source-to-target The selection of XML as the runtime specification mappings, and the data validation and error reporting medium recognizes its suitability and wide acceptance for information are all defined in XML. use in such situations. As noted in [6], XML supports The engine first reads all available XML files into document structures that can be nested to any level of memory as a singleton object named XMLTemplate. As complexity. This makes it especially suitable for source files are read into memory, they are handled by representing complex validation rules, which are the appropriate parser according to the extension of the sometimes best represented as hierarchies of assertions. input file name (.csv, .txt or .dat). The parsed data is The system design placed heavy emphasis on use of stored in memory as a two-dimensional array. metadata to specify many aspects of process behavior. Methods of the XMLTemplate object are then called to This mirrors many successful systems that specify perform data conversion and data validation. Data are metadata, especially using XML, to document the then stored in an object and passed to the database structure and meaning of data and their inter- module. relationships, as discussed in [7], [8], [9]. The database module acquires both the data and the The system implementation relies on metadata constructs loading instructions stored in the XMLTemplate object, (XML-based) for expressing data transformation and and then uses this information to update existing records, integration processes in heterogeneous database insert new records, or mark records as no longer active in environments, as discussed in [10], [11], [12]. the database. Java was chosen as the implementation language because When records are inserted or updated, an MD5 checksum of its use within similar RTI projects, its ability to handle is generated from the new record values and stored with complex data loading and validation, and its suitability the record. The system uses the checksum values to for direct connectivity within the project web portal. determine whether a submitted record represents a Because Java is platform independent, the package can change from the existing record; this also supports generation of file submission statistics.
Load more

Recommended publications
  • SQL Server Column Store Indexes Per-Åke Larson, Cipri Clinciu, Eric N
    SQL Server Column Store Indexes Per-Åke Larson, Cipri Clinciu, Eric N. Hanson, Artem Oks, Susan L. Price, Srikumar Rangarajan, Aleksandras Surna, Qingqing Zhou Microsoft {palarson, ciprianc, ehans, artemoks, susanpr, srikumar, asurna, qizhou}@microsoft.com ABSTRACT SQL Server column store indexes are “pure” column stores, not a The SQL Server 11 release (code named “Denali”) introduces a hybrid, because they store all data for different columns on new data warehouse query acceleration feature based on a new separate pages. This improves I/O scan performance and makes index type called a column store index. The new index type more efficient use of memory. SQL Server is the first major combined with new query operators processing batches of rows database product to support a pure column store index. Others greatly improves data warehouse query performance: in some have claimed that it is impossible to fully incorporate pure column cases by hundreds of times and routinely a tenfold speedup for a store technology into an established database product with a broad broad range of decision support queries. Column store indexes are market. We’re happy to prove them wrong! fully integrated with the rest of the system, including query To improve performance of typical data warehousing queries, all a processing and optimization. This paper gives an overview of the user needs to do is build a column store index on the fact tables in design and implementation of column store indexes including the data warehouse. It may also be beneficial to build column enhancements to query processing and query optimization to take store indexes on extremely large dimension tables (say more than full advantage of the new indexes.
    [Show full text]
  • When Relational-Based Applications Go to Nosql Databases: a Survey
    information Article When Relational-Based Applications Go to NoSQL Databases: A Survey Geomar A. Schreiner 1,* , Denio Duarte 2 and Ronaldo dos Santos Mello 1 1 Departamento de Informática e Estatística, Federal University of Santa Catarina, 88040-900 Florianópolis - SC, Brazil 2 Campus Chapecó, Federal University of Fronteira Sul, 89815-899 Chapecó - SC, Brazil * Correspondence: [email protected] Received: 22 May 2019; Accepted: 12 July 2019; Published: 16 July 2019 Abstract: Several data-centric applications today produce and manipulate a large volume of data, the so-called Big Data. Traditional databases, in particular, relational databases, are not suitable for Big Data management. As a consequence, some approaches that allow the definition and manipulation of large relational data sets stored in NoSQL databases through an SQL interface have been proposed, focusing on scalability and availability. This paper presents a comparative analysis of these approaches based on an architectural classification that organizes them according to their system architectures. Our motivation is that wrapping is a relevant strategy for relational-based applications that intend to move relational data to NoSQL databases (usually maintained in the cloud). We also claim that this research area has some open issues, given that most approaches deal with only a subset of SQL operations or give support to specific target NoSQL databases. Our intention with this survey is, therefore, to contribute to the state-of-art in this research area and also provide a basis for choosing or even designing a relational-to-NoSQL data wrapping solution. Keywords: big data; data interoperability; NoSQL databases; relational-to-NoSQL mapping 1.
    [Show full text]
  • Data Dictionary a Data Dictionary Is a File That Helps to Define The
    Cleveland | v. 216.369.2220 • Columbus | v. 614.291.8456 Data Dictionary A data dictionary is a file that helps to define the organization of a particular database. The data dictionary acts as a description of the data objects or items in a model and is used for the benefit of the programmer or other people who may need to access it. A data dictionary does not contain the actual data from the database; it contains only information for how to describe/manage the data; this is called metadata*. Building a data dictionary provides the ability to know the kind of field, where it is located in a database, what it means, etc. It typically consists of a table with multiple columns that describe relationships as well as labels for data. A data dictionary often contains the following information about fields: • Default values • Constraint information • Definitions (example: functions, sequence, etc.) • The amount of space allocated for the object/field • Auditing information What is the data dictionary used for? 1. It can also be used as a read-only reference in order to obtain information about the database 2. A data dictionary can be of use when developing programs that use a data model 3. The data dictionary acts as a way to describe data in “real-world” terms Why is a data dictionary needed? One of the main reasons a data dictionary is necessary is to provide better accuracy, organization, and reliability in regards to data management and user/administrator understanding and training. Benefits of using a data dictionary: 1.
    [Show full text]
  • USER GUIDE Optum Clinformatics™ Data Mart Database
    USER GUIDE Optum Clinformatics Data Mart Database 1 | P a g e TABLE OF CONTENTS TOPIC PAGE # 1. REQUESTING DATA 3 Eligibility 3 Forms 3 Contact Us 4 2. WHAT YOU WILL NEED 4 SAS Software 4 VPN 5 3. ABSTRACTS, MANUSCRIPTS, THESES, AND DISSERTATIONS 5 Referencing Optum Data 5 Optum Review 5 4. DATA USER SET-UP AND ACCESS INFORMATION 6 Server Log-In After Initial Set-Up 6 Server Access 6 Establishing a Connection to Enterprise Guide 7 Instructions to Add SAS EG to the Cleared Firewall List 8 How to Proceed After Connection 8 5. BEST PRACTICES FOR DATA USE 9 Saving Programs and Back-Up Procedures 9 Recommended Coding Practices 9 6. APPENDIX 11 Version Date: 27-Feb-17 2 | P a g e Optum® ClinformaticsTM Data Mart Database The Optum® ClinformaticsTM Data Mart is an administrative health claims database from a large national insurer made available by the University of Rhode Island College of Pharmacy. The statistically de-identified data includes medical and pharmacy claims, as well as laboratory results, from 2010 through 2013 with over 22 million commercial enrollees. 1. REQUESTING DATA The following is a brief outline of the process for gaining access to the data. Eligibility Must be an employee or student at the University of Rhode Island conducting unfunded or URI internally funded projects. Data will be made available to the following users: 1. Faculty and their research team for projects with IRB approval. 2. Students a. With a thesis/dissertation proposal approved by IRB and the Graduate School (access request form, see link below, must be signed by Major Professor).
    [Show full text]
  • Activant Prophet 21
    Activant Prophet 21 Understanding Prophet 21 Databases This class is designed for… Prophet 21 users that are responsible for report writing System Administrators Operations Managers Helpful to be familiar with SQL Query Analyzer and how to write basic SQL statements Objectives Explain the difference between databases, tables and columns Extract data from different areas of the system Discuss basic SQL statements using Query Analyzer Use Prophet 21 to gather SQL Information Utilize Data Dictionary This course will NOT cover… Basic Prophet 21 functionality Seagate’s Crystal Reports Definitions Columns Contains a single piece of information (fields) Record (rows) One complete set of columns Table A collection of rows View Virtual table created for easier data extraction Database Collection of information organized for easy selection of data SQL Query A graphical user interface used to extract Analyzer data SQL Query Analyzer Accessed through Microsoft SQL Server SQL Server Tools Enterprise Manager Perform administrative functions such as backing up and restoring databases and maintaining SQL Logins Profiler Run traces of activity in your system Basic SQL Commands sp_help sp_help <table_name> select * from <table_name> Select <field_name> from <table_name> Most Common Reporting Areas Address and Contact tables Order Processing Inventory Purchasing Accounts Receivable Accounts Payable Production Orders Address and Contact tables Used in conjunction with other tables These tables hold every address/contact in the
    [Show full text]
  • Uniform Data Access Platform for SQL and Nosql Database Systems
    Information Systems 69 (2017) 93–105 Contents lists available at ScienceDirect Information Systems journal homepage: www.elsevier.com/locate/is Uniform data access platform for SQL and NoSQL database systems ∗ Ágnes Vathy-Fogarassy , Tamás Hugyák University of Pannonia, Department of Computer Science and Systems Technology, P.O.Box 158, Veszprém, H-8201 Hungary a r t i c l e i n f o a b s t r a c t Article history: Integration of data stored in heterogeneous database systems is a very challenging task and it may hide Received 8 August 2016 several difficulties. As NoSQL databases are growing in popularity, integration of different NoSQL systems Revised 1 March 2017 and interoperability of NoSQL systems with SQL databases become an increasingly important issue. In Accepted 18 April 2017 this paper, we propose a novel data integration methodology to query data individually from different Available online 4 May 2017 relational and NoSQL database systems. The suggested solution does not support joins and aggregates Keywords: across data sources; it only collects data from different separated database management systems accord- Uniform data access ing to the filtering options and migrates them. The proposed method is based on a metamodel approach Relational database management systems and it covers the structural, semantic and syntactic heterogeneities of source systems. To introduce the NoSQL database management systems applicability of the proposed methodology, we developed a web-based application, which convincingly MongoDB confirms the usefulness of the novel method. Data integration JSON ©2017 Elsevier Ltd. All rights reserved. 1. Introduction solution to retrieve data from heterogeneous source systems and to deliver them to the user.
    [Show full text]
  • Product Master Data Management Reference Guide
    USAID GLOBAL HEALTH SUPPLY CHAIN PROGRAM Procurement and Supply Management PRODUCT MASTER DATA MANAGEMENT REFERENCE GUIDE Version 1.0 February 2020 The USAID Global Health Supply Chain Program-Procurement and Supply Management (GHSC-PSM) project is funded under USAID Contract No. AID-OAA-I-15-0004. GHSC-PSM connects technical solutions and proven commercial processes to promote efficient and cost-effective health supply chains worldwide. Our goal is to ensure uninterrupted supplies of health commodities to save lives and create a healthier future for all. The project purchases and delivers health commodities, offers comprehensive technical assistance to strengthen national supply chain systems, and provides global supply chain leadership. GHSC-PSM is implemented by Chemonics International, in collaboration with Arbola Inc., Axios International Inc., IDA Foundation, IBM, IntraHealth International, Kuehne + Nagel Inc., McKinsey & Company, Panagora Group, Population Services International, SGS Nederland B.V., and University Research Co., LLC. To learn more, visit ghsupplychain.org DISCLAIMER: The views expressed in this publication do not necessarily reflect the views of the U.S. Agency for International Development or the U.S. government. Contents Acronyms ....................................................................................................................................... 3 Executive Summary ...................................................................................................................... 4 Background
    [Show full text]
  • Master Data Management: Dos & Don’Ts
    Master Data Management: dos & don’ts Keesjan van Unen, Ad de Goeij, Sander Swartjes, and Ard van der Staaij Master Data Management (MDM) is high on the agenda for many organizations. At Board level too, people are now fully aware that master data requires specific attention. This increasing attention is related to the concern for the overall quality of data. By enhancing the quality of master data, disrup- tions to business operations can be limited and therefore efficiency is increased. Also, the availability of reliable management information will increase. However, reaching this goal is not an entirely smooth process. MDM is not rocket science, but it does have specific problems that must be addressed in the right way. This article describes three of these attention areas, including several dos and don’ts. C.J.W.A. van Unen is a senior manager at KPMG Management Consulting, IT Advisory. [email protected] A.S.M de Goeij RE is a manager at KPMG Management Consulting, IT Advisory. [email protected] S. Swartjes is an advisor at KPMG Risk Consulting, IT Advisory. [email protected] A.J. van der Staaij is an advisor at KPMG Risk Consulting, IT Advisory. [email protected] Compact_ 2012 2 International edition 1 Treat master data as an asset Introduction Do not go for gold immediately Every organization has to deal with master data, and seeks Although it may be very tempting, you should not go for ways to manage the quality of this specific group of data in the ‘ideal model’ when (re)designing the MDM organiza- an effective and efficient manner.
    [Show full text]
  • Getting the Data You Can't
    The Marriage of VFE, SQL, Visual Studio John W. Fitzgerald, MD Gregory P. Sanders, MD CHUG 2014 Fall Conference GETTING THE DATA YOU CAN’T GET: WHEN MEL JUST ISN’T ENOUGH The Problem . Built in data symbols are powerful but may not provide the data you need MEDS_AFTER() PROB_LIST_CHANGES() ALL_NEW() ….etc . Mldefs.txt may be available to address your needs but are undocumented, difficult to locate, and may not survive version changes. There are tables with no MEL connections . Complex relationships are generally not available Our Approach . Create SQL Centricity data access statements in Visual Studio, Visual Scripts as an .exe . Store in Centricity Client . Pass arguments to the .exe from VFE . Return results from .exe call . Parse the results as required. Non-trivial Set of Skills Required . VFE . Data Symbols . Centricity table structure and relationships . Model solution in Crystal, Access . SQL statement construction . Visual Studio or equivalent . Lots of debugging THE CLIENT . Folder on each local machine . EXE sits in Client folder for easy access . Code is run locally . Client calls EXE and passes its authentication to it . Client has access to network (ie, SQL server) . EXE must be pushed out to all Clients Dataflow: Call & Query Centricity SQL Centricity calls EXE EXE queries SQL Database EXE Dataflow: Compile & Return Centricity SQL EXE returns data to Centricity SQL returns data to EXE EXE Inside the EXE . Capture Passed Data . Compile SQL Statement . Connect to Database . Organize Queried Data . Pass Back to Centricity Capture Passed Data . Output Data . File name and path designated with “ /o “ primer . Input Data . Data passed to EXE from Centricity .
    [Show full text]
  • MDM Physical Data Dictionary
    IBM InfoSphere Master Data Management InfoSphere Master Data Management Version 11.3 MDM Physical Data Dictionary GI13-2668-01 IBM InfoSphere Master Data Management InfoSphere Master Data Management Version 11.3 MDM Physical Data Dictionary GI13-2668-01 Note Before using this information and the product that it supports, read the information in “Notices and trademarks” on page 195. Edition Notice This edition applies to version 11.3 of IBM InfoSphere Master Data Management and to all subsequent releases and modifications until otherwise indicated in new editions. © Copyright IBM Corporation 1996, 2014. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Chapter 1. Tables...........1 CDEMEMATCHFUNCTIONTP ........46 ACCESSDATEVAL ............1 CDEMEMATCHWORDTP .........47 ADDACTIONTYPE ............2 CDENDREASONTP ...........48 ADDRESS ...............2 CDENTITYLINKSTTP ...........49 ADDRESSGROUP ............5 CDFREQMODETP ............50 AGREEMENTSPECVAL ..........5 CDGENERATIONTP ...........50 BANKACCOUNT ............6 CDHIGHESTEDUTP ...........51 BILLINGSUMMARY............7 CDHOLDINGTP ............52 CAMPAIGN ..............9 CDIDSTATUSTP.............53 CAMPAIGNASSOCIAT ..........10 CDIDTP ...............53 CATEGORY ..............11 CDINACTREASONTP...........54 CATEGORYNLS .............12 CDINCOMESRCTP............55 CATEQUIV ..............13 CDINDUSTRYTP ............56 CATHIERARCHY ............14 CDLANGTP ..............56 CATHIERARCHYNLS ...........15 CDLINKREASONTP
    [Show full text]
  • The Database/Data Dictionary Interface
    PANEL : THE DATABASE/DATA DICTIONARY INTERFACE Larry KERSCHBERG (Chairperson)-Un iversity of South Carolina, Paul FEHDER-IBM Santa Teresa Lab, Georqe GAGNAC-Cincom Systems , Inc., John MXIIRE and llichael JAKES-Software A.G. of North America, Inc., Shamkant NAVATHE-University of Florida, Edgar SIBLEY-Alpha Omega Group, Inc. The purpose of this panel is to review the state- . Privacy, integrity, and security of-the-art in Data Dictionary systems and to issues. explore future capabilities and architectures for such systems. In particular, the interface . How can the dictionary bottleneck between Data Dictionarv svstems and Database be ameliorated? Flanaaement systems wili be discussed in relation to the following issues: 2. The Role of Data Dictionaries in Information Resources Management 1. Architecture of Data Dictionary Systems As data dictionaries are endowed with Current thinking regarding data dictionaries increased intelligence and enhanced capa- indicates that the dictionary is to be a bilities, they will play a key role in repository of information about an enter- information resource management. We wish prise. Thus, this repository, or knowledge to explore several aspects of this role, base, not only contains information about namely: the "semantics" of data stored in databases, but also knows about enterprise concepts A. Tools for enterprise administration. such as organizational structure and dyna- mics, forms, memos and other documents B. Application development tools. together with their organizational flows, and the description of information resources C. Managing computer hardware and software such as hardware and software systems. resources. In this framework we consider the following D. Communication mechanisms among data issues: dictionaries. A.
    [Show full text]
  • A Dictionary of DBMS Terms
    A Dictionary of DBMS Terms Access Plan Access plans are generated by the optimization component to implement queries submitted by users. ACID Properties ACID properties are transaction properties supported by DBMSs. ACID is an acronym for atomic, consistent, isolated, and durable. Address A location in memory where data are stored and can be retrieved. Aggregation Aggregation is the process of compiling information on an object, thereby abstracting a higher-level object. Aggregate Function A function that produces a single result based on the contents of an entire set of table rows. Alias Alias refers to the process of renaming a record. It is alternative name used for an attribute. 700 A Dictionary of DBMS Terms Anomaly The inconsistency that may result when a user attempts to update a table that contains redundant data. ANSI American National Standards Institute, one of the groups responsible for SQL standards. Application Program Interface (API) A set of functions in a particular programming language is used by a client that interfaces to a software system. ARIES ARIES is a recovery algorithm used by the recovery manager which is invoked after a crash. Armstrong’s Axioms Set of inference rules based on set of axioms that permit the algebraic mani- pulation of dependencies. Armstrong’s axioms enable the discovery of minimal cover of a set of functional dependencies. Associative Entity Type A weak entity type that depends on two or more entity types for its primary key. Attribute The differing data items within a relation. An attribute is a named column of a relation. Authorization The operation that verifies the permissions and access rights granted to a user.
    [Show full text]