DOCSLIB.ORG

Managing Data in Motion This Page Intentionally Left Blank Managing Data in Motion Data Integration Best Practice Techniques and Technologies

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Managing Data in Motion This Page Intentionally Left Blank Managing Data in Motion Data Integration Best Practice Techniques and Technologies Managing Data in Motion This page intentionally left blank Managing Data in Motion Data Integration Best Practice Techniques and Technologies April Reeve AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO Morgan Kaufmann is an imprint of Elsevier Acquiring Editor: Andrea Dierna Development Editor: Heather Scherer Project Manager: Mohanambal Natarajan Designer: Russell Purdy Morgan Kaufmann is an imprint of Elsevier 225 Wyman Street, Waltham, MA 02451, USA Copyright r 2013 Elsevier Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions. This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein). Notices Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods or professional practices, may become necessary. Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information or methods described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility. To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein. Library of Congress Cataloging-in-Publication Data Application submitted British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library ISBN: 978-0-12-397167-8 For information on all MK publications visit our website at www.mkp.com Printed in the USA 131415161710987654321 For my sons Henry who knows everything and, although he hasn’t figured out exactly what I do for a living, advised me to “put words on paper” and David who is so talented, so much fun to be with, and always willing to go with me to Disney. This page intentionally left blank Contents Foreword..................................................................................................................xv Acknowledgements............................................................................................... xvii Biography................................................................................................................xix Introduction.............................................................................................................xxi PART 1 INTRODUCTION TO DATA INTEGRATION Chapter 1 The Importance of Data Integration ............................ 3 The natural complexity of data interfaces ........................................3 The rise of purchased vendor packages ............................................4 Key enablement of big data and virtualization.................................5 Chapter 2 What Is Data Integration? .......................................... 7 Data in motion ...................................................................................7 Integrating into a common format—transforming data....................7 Migrating data from one system to another......................................8 Moving data around the organization ...............................................9 Pulling information from unstructured data....................................11 Moving process to data ...................................................................12 Chapter 3 Types and Complexity of Data Integration..................15 The differences and similarities in managing data in motion and persistent data ...........................................................................15 Batch data integration......................................................................16 Real-time data integration ...............................................................16 Big data integration .........................................................................17 Data virtualization ...........................................................................17 Chapter 4 The Process of Data Integration Development ...........................................................19 The data integration development life cycle...................................19 Inclusion of business knowledge and expertise..............................20 PART 2 BATCH DATA INTEGRATION Chapter 5 Introduction to Batch Data Integration.......................25 What is batch data integration?.......................................................25 Batch data integration life cycle .....................................................26 vii viii Contents Chapter 6 Extract, Transform, and Load ....................................29 What is ETL?...................................................................................29 Profiling ...........................................................................................30 Extract..............................................................................................30 Staging .............................................................................................31 Access layers ...................................................................................32 Transform.........................................................................................33 Simple mapping ..........................................................................33 Lookups.......................................................................................33 Aggregation and normalization ..................................................33 Calculation ..................................................................................34 Load .................................................................................................34 Chapter 7 Data Warehousing ...................................................37 What is data warehousing?..............................................................37 Layers in an enterprise data warehouse architecture......................38 Operational application layer .....................................................38 External data ...............................................................................38 Data staging areas coming into a data warehouse .....................39 Data warehouse data structure....................................................40 Staging from data warehouse to data mart or business intelligence ...................................................................40 Business Intelligence Layer........................................................40 Types of data to load in a data warehouse .....................................41 Master data in a data warehouse ................................................41 Balance and snapshot data in a data warehouse ........................42 Transactional data in a data warehouse .....................................43 Events..........................................................................................43 Reconciliation .............................................................................43 Interview with an expert: Krish Krishnan on data warehousing and data integration............................................44 Chapter 8 Data Conversion ......................................................51 What is data conversion? ................................................................51 Data conversion life cycle...............................................................51 Data conversion analysis .................................................................52 Best practice data loading ...............................................................52 Improving source data quality.........................................................53 Contents ix Mapping to target ..........................................................................53 Configuration data .........................................................................54 Testing and dependencies..............................................................55 Private data ....................................................................................55 Proving...........................................................................................56 Environments .................................................................................56 Chapter 9 Data Archiving .......................................................59 What is data archiving?.................................................................59 Selecting data to archive ...............................................................60 Can the archived data be retrieved?..............................................60 Conforming data structures in the archiving environment ...........61 Flexible data structures..................................................................61
Load more

Recommended publications
  • Data Mapping: Definition, Tools, and Techniques
    Data Mapping: Definition, Tools, and Techniques Enterprise data is getting more dispersed and voluminous by the day, and at the same time, it has become more important than ever for businesses to leverage data and transform it into actionable insights. However, enterprises today collect information from an array of data points, and they may not always speak the same language. To integrate this data and make sense of it, data mapping is used which is the process of establishing relationships between separate data models. What is Data Mapping? In simple words, data mapping is the process of mapping data fields from a source file to their related target fields. For example, in Figure 1, ‘Name,’ ‘Email,’ and ‘Phone’ fields from an Excel source are mapped to the relevant fields in a Delimited file, which is our destination. Data mapping tasks vary in complexity, depending on the hierarchy of the data being mapped, as well as the disparity between the structure of the source and the target. Every business application, whether on-premise or cloud, uses metadata to explain the data fields and attributes that constitute the data, as well as semantic rules that govern how data is stored within that application or repository. For example, Microsoft Dynamics CRM contains several data objects, such as Leads, Opportunities, and Competitors. Each of these data objects has several fields like Name, Account Owner, City, Country, Job Title, and more. The application also has a defined schema along with attributes, enumerations, and mapping rules. Therefore, if a new record is to be added to the schema of a data object, a data map needs to be created from the data source to the Microsoft Dynamics CRM account.
    [Show full text]
  • Enterprise Information Integration: Successes, Challenges and Controversies
    Enterprise Information Integration: Successes, Challenges and Controversies Alon Y. Halevy∗(Editor), Naveen Ashish,y Dina Bitton,z Michael Carey,x Denise Draper,{ Jeff Pollock,k Arnon Rosenthal,∗∗Vishal Sikkayy ABSTRACT Several factors came together at the time to contribute to The goal of EII Systems is to provide uniform access to the development of the EII industry. First, some technolo- multiple data sources without having to first loading them gies developed in the research arena have matured to the into a data warehouse. Since the late 1990's, several EII point that they were ready for commercialization, and sev- products have appeared in the marketplace and significant eral of the teams responsible for these developments started experience has been accumulated from fielding such systems. companies (or spun off products from research labs). Sec- This collection of articles, by individuals who were involved ond, the needs of data management in organizations have in this industry in various ways, describes some of these changed: the need to create external coherent web sites experiences and points to the challenges ahead. required integrating data from multiple sources; the web- connected world raised the urgency for companies to start communicating with others in various ways. Third, the 1. INTRODUCTORY REMARKS emergence of XML piqued the appetites of people to share data. Finally, there was a general atmosphere in the late 90's Alon Halevy that any idea is worth a try (even good ones!). Importantly, University of Washington & Transformic Inc. data warehousing solutions were deemed inappropriate for [email protected] supporting these needs, and the cost of ad-hoc solutions were beginning to become unaffordable.
    [Show full text]
  • Scientific Data Mining in Astronomy
    SCIENTIFIC DATA MINING IN ASTRONOMY Kirk D. Borne Department of Computational and Data Sciences, George Mason University, Fairfax, VA 22030, USA [email protected] Abstract We describe the application of data mining algorithms to research prob- lems in astronomy. We posit that data mining has always been fundamen- tal to astronomical research, since data mining is the basis of evidence- based discovery, including classification, clustering, and novelty discov- ery. These algorithms represent a major set of computational tools for discovery in large databases, which will be increasingly essential in the era of data-intensive astronomy. Historical examples of data mining in astronomy are reviewed, followed by a discussion of one of the largest data-producing projects anticipated for the coming decade: the Large Synoptic Survey Telescope (LSST). To facilitate data-driven discoveries in astronomy, we envision a new data-oriented research paradigm for astron- omy and astrophysics – astroinformatics. Astroinformatics is described as both a research approach and an educational imperative for modern data-intensive astronomy. An important application area for large time- domain sky surveys (such as LSST) is the rapid identification, charac- terization, and classification of real-time sky events (including moving objects, photometrically variable objects, and the appearance of tran- sients). We describe one possible implementation of a classification broker for such events, which incorporates several astroinformatics techniques: user annotation, semantic tagging, metadata markup, heterogeneous data integration, and distributed data mining. Examples of these types of collaborative classification and discovery approaches within other science disciplines are presented. arXiv:0911.0505v1 [astro-ph.IM] 3 Nov 2009 1 Introduction It has been said that astronomers have been doing data mining for centuries: “the data are mine, and you cannot have them!”.
    [Show full text]
  • Data Quality Through Data Integration: How Integrating Your IDEA Data Will Help Improve Data Quality
    center for the integration of IDEA Data Data Quality Through Data Integration: How Integrating Your IDEA Data Will Help Improve Data Quality July 2018 Authors: Sara Sinani & Fred Edora High-quality data is essential when looking at student-level data, including data specifically focused on students with disabilities. For state education agencies (SEAs), it is critical to have a solid foundation for how data is collected and stored to achieve high-quality data. The process of integrating the Individuals with Disabilities Education Act (IDEA) data into a statewide longitudinal data system (SLDS) or other general education data system not only provides SEAs with more complete data, but also helps SEAs improve accuracy of federal reporting, increase the quality of and access to data within and across data systems, and make better informed policy decisions related to students with disabilities. Through the data integration process, including mapping data elements, reviewing data governance processes, and documenting business rules, SEAs will have developed documented processes and policies that result in more integral data that can be used with more confidence. In this brief, the Center for the Integration of IDEA Data (CIID) provides scenarios based on the continuum of data integration by focusing on three specific scenarios along the integration continuum to illustrate how a robust integrated data system improves the quality of data. Scenario A: Siloed Data Systems Data systems where the data is housed in separate systems and State Education Agency databases are often referred to as “data silos.”1 These data silos are isolated from other data systems and databases.
    [Show full text]
  • Preview MS Access Tutorial (PDF Version)
    MS Access About the Tutorial Microsoft Access is a Database Management System (DBMS) from Microsoft that combines the relational Microsoft Jet Database Engine with a graphical user interface and software- development tools. It is a part of the Microsoft Office suite of applications, included in the professional and higher editions. This is an introductory tutorial that covers the basics of MS Access. Audience This tutorial is designed for those people who want to learn how to start working with Microsoft Access. After completing this tutorial, you will have a better understating of MS Access and how you can use it to store and retrieve data. Prerequisites It is a simple and easy-to-understand tutorial. There are no set prerequisites as such, and it should be useful for any beginner who want acquire knowledge on MS Access. However it will definitely help if you are aware of some basic concepts of a database, especially RDBMS concepts. Copyright and Disclaimer Copyright 2018 by Tutorials Point (I) Pvt. Ltd. All the content and graphics published in this e-book are the property of Tutorials Point (I) Pvt. Ltd. The user of this e-book is prohibited to reuse, retain, copy, distribute or republish any contents or a part of contents of this e-book in any manner without written consent of the publisher. We strive to update the contents of our website and tutorials as timely and as precisely as possible, however, the contents may contain inaccuracies or errors. Tutorials Point (I) Pvt. Ltd. provides no guarantee regarding the accuracy, timeliness or completeness of our website or its contents including this tutorial.
    [Show full text]
  • Informatica Cloud Data Integration
    Informatica® Cloud Data Integration Microsoft SQL Server Connector Guide Informatica Cloud Data Integration Microsoft SQL Server Connector Guide March 2019 © Copyright Informatica LLC 2017, 2019 This software and documentation are provided only under a separate license agreement containing restrictions on use and disclosure. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica LLC. U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation is subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License. Informatica, the Informatica logo, Informatica Cloud, and PowerCenter are trademarks or registered trademarks of Informatica LLC in the United States and many jurisdictions throughout the world. A current list of Informatica trademarks is available on the web at https://www.informatica.com/trademarks.html. Other company and product names may be trade names or trademarks of their respective owners. Portions of this software and/or documentation are subject to copyright held by third parties. Required third party notices are included with the product. See patents at https://www.informatica.com/legal/patents.html. DISCLAIMER: Informatica LLC provides this documentation "as is" without warranty of any kind, either express or implied, including, but not limited to, the implied warranties of noninfringement, merchantability, or use for a particular purpose.
    [Show full text]
  • Oracle Warehouse Builder Concepts Guide
    Oracle® Warehouse Builder Concepts 11g Release 2 (11.2) E10581-02 August 2010 Oracle Warehouse Builder Concepts, 11g Release 2 (11.2) E10581-02 Copyright © 2000, 2010, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007).
    [Show full text]
  • What to Look for When Selecting a Master Data Management Solution What to Look for When Selecting a Master Data Management Solution
    What to Look for When Selecting a Master Data Management Solution What to Look for When Selecting a Master Data Management Solution Table of Contents Business Drivers of MDM .......................................................................................................... 3 Next-Generation MDM .............................................................................................................. 5 Keys to a Successful MDM Deployment .............................................................................. 6 Choosing the Right Solution for MDM ................................................................................ 7 Talend’s MDM Solution .............................................................................................................. 7 Conclusion ..................................................................................................................................... 8 Master data management (MDM) exists to enable the business success of an orga- nization. With the right MDM solution, organizations can successfully manage and leverage their most valuable shared information assets. Because of the strategic importance of MDM, organizations can’t afford to make any mistakes when choosing their MDM solution. This article will discuss what to look for in an MDM solution and why the right choice will help to drive strategic business initiatives, from cloud computing to big data to next-generation business agility. 2 It is critical for CIOs and business decision-makers to understand the value
    [Show full text]
  • Using ETL, EAI, and EII Tools to Create an Integrated Enterprise
    Data Integration: Using ETL, EAI, and EII Tools to Create an Integrated Enterprise Colin White Founder, BI Research TDWI Webcast October 2005 TDWI Data Integration Study Copyright © BI Research 2005 2 Data Integration: Barrier to Application Development Copyright © BI Research 2005 3 Top Three Data Integration Inhibitors Copyright © BI Research 2005 4 Staffing and Budget for Data Integration Copyright © BI Research 2005 5 Data Integration: A Definition A framework of applications, products, techniques and technologies for providing a unified and consistent view of enterprise-wide business data Copyright © BI Research 2005 6 Enterprise Business Data Copyright © BI Research 2005 7 Data Integration Architecture Source Target Data integration Master data applications Business domain dispersed management (MDM) MDM applications integrated internal data & external Data integration techniques data Data Data Data propagation consolidation federation Changed data Data transformation (restructure, capture (CDC) cleanse, reconcile, aggregate) Data integration technologies Enterprise data Extract transformation Enterprise content replication (EDR) load (ETL) management (ECM) Enterprise application Right-time ETL Enterprise information integration (EAI) (RT-ETL) integration (EII) Web services (services-oriented architecture, SOA) Data integration management Data quality Metadata Systems management management management Copyright © BI Research 2005 8 Data Integration Techniques and Technologies Data Consolidation centralized data Extract, transformation
    [Show full text]
  • Transforming MARCXML Records Using XSLT
    Transforming MARCXML Records Using XSLT Violeta Ilik Head, Digital Systems & Collection Services Digital Innovations Librarian Galter Health Sciences Library Northwestern University Clinical and Translational Sciences Institute Chicago, IL March 4, 2015 Teun Hocks Violeta Ilik vivoweb.org Hosted by ALCTS, Association for Library Collections and Technical Services + ALCTS - Library Resources & Technical Services n Ilik, V., Storlien, J., and Olivarez, J. (2014). “Metadata Makeover: Transforming MARC Records Using XSLT.” Library Resources and Technical Services 58 (3): 187-208. “The knowledge gained from learning information technology can be used to experiment with methods of transforming one metadata schema into another using various software solutions” Violeta Ilik vivoweb.org Hosted by ALCTS, Association for Library Collections and Technical Services + Why Catalogers? n Survey conducted by Ma in 2007 revealed that the metadata qualifications and responsibilities required knowledge of MARC, crosswalks, XML, OAI … n Analysis of cataloging position description performed by Park, Lu, and Marion reveal that advances in technology have created a new realm of desired skills, qualifications and responsibilities for catalogers. n Terry Reese’s MarcEdit is a best friend to everyone in technical services n Tosaka stresses the importance of metadata transformation to enable reuse Violeta Ilik vivoweb.org Hosted by ALCTS, Association for Library Collections and Technical Services + Recommendations: n Cataloging departments need to be proactive in
    [Show full text]
  • Declarative Access to Filesystem Data New Application Domains for XML Database Management Systems
    Declarative Access to Filesystem Data New application domains for XML database management systems Alexander Holupirek Dissertation zur Erlangung des akademischen Grades Doktor der Naturwissenschaften (Dr. rer. nat.) Fachbereich Informatik und Informationswissenschaft Mathematisch-Naturwissenschaftliche Sektion Universität Konstanz Referenten: Prof. Dr. Marc H. Scholl Prof. Dr. Marcel Waldvogel Tag der mündlichen Prüfung: 17. Juli 2012 Abstract XML and state-of-the-art XML database management systems (XML-DBMSs) can play a leading role in far more application domains as it is currently the case. Even in their basic configuration, they entail all components necessary to act as central systems for complex search and retrieval tasks. They provide language-specific index- ing of full-text documents and can store structured, semi-structured and binary data. Besides, they offer a great variety of standardized languages (XQuery, XSLT, XQuery Full Text, etc.) to develop applications inside a pure XML technology stack. Benefits are obvious: Data, logic, and presentation tiers can operate on a single data model, and no conversions have to be applied when switching in between. This thesis deals with the design and development of XML/XQuery driven informa- tion architectures that process formerly heterogeneous data sources in a standardized and uniform manner. Filesystems and their vast amounts of different file types are a prime example for such a heterogeneous dataspace. A new XML dialect, the Filesystem Markup Language (FSML), is introduced to construct a database view of the filesystem and its contents. FSML provides a uniform view on the filesystem’s contents and allows developers to leverage the complete XML technology stack on filesystem data.
    [Show full text]
  • X3ML Mapping Framework for Information Integration in Cultural Heritage and Beyond
    Int J Digit Libr (2017) 18:301–319 DOI 10.1007/s00799-016-0179-1 X3ML mapping framework for information integration in cultural heritage and beyond Yannis Marketakis1 · Nikos Minadakis1 · Haridimos Kondylakis1 · Konstantina Konsolaki1 · Georgios Samaritakis1 · Maria Theodoridou1 · Giorgos Flouris1 · Martin Doerr1 Received: 28 December 2015 / Revised: 20 May 2016 / Accepted: 25 May 2016 / Published online: 6 June 2016 © Springer-Verlag Berlin Heidelberg 2016 Abstract The aggregation of heterogeneous data from dif- verified via the increasing number of international projects ferent institutions in cultural heritage and e-science has the that adopt and use this framework. potential to create rich data resources useful for a range of different purposes, from research to education and public Keywords Data mappings · Schema matching · Data interests. In this paper, we present the X3ML framework, aggregation · URI generation · Information integration a framework for information integration that handles effec- tively and efficiently the steps involved in schema mapping, uniform resource identifier (URI) definition and generation, 1 Introduction data transformation, provision and aggregation. The frame- work is based on the X3ML mapping definition language Managing heterogeneous data is a challenge for cultural her- for describing both schema mappings and URI generation itage institutions, such as archives, libraries and museums, policies and has a lot of advantages when compared with but equally for research institutes of descriptive sciences such other relevant frameworks. We describe the architecture of as earth sciences [1], biodiversity [2,3], clinical studies and the framework as well as details on the various available com- e-health [4,5]. These institutions host and maintain various ponents.
    [Show full text]