DOCSLIB.ORG

Next Generation Data Integration

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Next Generation Data Integration TDWI RESE A RCH TDWI BEST PRACtiCES REPORT SECOND QUARTER 2011 NEXT GENERATION DATA InTEGRATION By Philip Russom tdwi.org Research Sponsors DataFlux IBM Informatica SAP Syncsort Talend SECOND QUARTER 2011 TDWI BEST PRACtiCES REPORT NEXT GENERATION DATA InTEGRATION By Philip Russom Table of Contents Research Methodology and Demographics 3 Introduction to Next Generation Data Integration 4 Ten Rules for Next Generation Data Integration . 4 Why Care About NGDI Now? . 6 Leading Generational Changes for Data Integration 7 Expanding Into More DI Techniques . 7 Users’ Data Integration Tool Portfolios . 9 DI Tool and Platform Replacements . 10 Data Types Being Integrated . 13 Data Integration Architecture . 14 Organizational Issues for NGDI 17 Organizational Structures for DI Teams . 17 Unified Data Management . .20 Collaborative Data Integration . 22 Catalog of NGDI Practices, Tools, and Platforms 23 Potential Growth versus Commitment for DI Options . 24 Trends for Next Generation Data Integration Options . .26 Vendor Products and Platforms for NGDI 28 Recommendations 30 © 2011 by TDWI (The Data Warehousing InstituteTM), a division of 1105 Media, Inc. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. E-mail requests or feedback to [email protected]. Product and company names mentioned herein may be trademarks and/or registered trademarks of their respective companies. tdwi.org 1 NE X T Generation Data Integration About the Author PHILIP RUSSOM is a well-known figure in data warehousing and business intelligence, having published more than 500 research reports, magazine articles, opinion columns, speeches, Webinars, and more. Today, he’s TDWI Research Director for Data Management at The Data Warehousing Institute (TDWI), where he oversees many of TDWI’s research-oriented publications, services, and events. Before joining TDWI in 2005, Russom was an industry analyst covering BI at Forrester Research, Giga Information Group, and Hurwitz Group. He also ran his own business as an independent industry analyst and BI consultant and was a contributing editor with leading IT magazines. Before that, Russom worked in technical and marketing positions for various database vendors. You can reach him at [email protected]. About TDWI TDWI, a division of 1105 Media, Inc., is the premier provider of in-depth, high-quality education and research in the business intelligence and data warehousing industry. TDWI is dedicated to educating business and information technology professionals about the best practices, strategies, techniques, and tools required to successfully design, build, maintain, and enhance business intelligence and data warehousing solutions. TDWI also fosters the advancement of business intelligence and data warehousing research and contributes to knowledge transfer and the professional development of its Members. TDWI offers a worldwide Membership program, five major educational conferences, topical educational seminars, role-based training, onsite courses, certification, solution provider partnerships, an awards program for best practices, live Webinars, resourceful publications, an in-depth research program, and a comprehensive Web site: tdwi.org. About the TDWI Best Practices Reports Series This series is designed to educate technical and business professionals about new business intelligence technologies, concepts, or approaches that address a significant problem or issue. Research for the reports is conducted via interviews with industry experts and leading-edge user companies and is supplemented by surveys of business intelligence professionals. To support the program, TDWI seeks vendors that collectively wish to evangelize a new approach to solving business intelligence problems or an emerging technology discipline. By banding together, sponsors can validate a new market niche and educate organizations about alternative solutions to critical business intelligence issues. Please contact TDWI Research Director Philip Russom ([email protected]) to suggest a topic that meets these requirements. Acknowledgments TDWI would like to thank many people who contributed to this report. First, we appreciate the many users who responded to our survey, especially those who responded to our requests for phone interviews. Second, our report sponsors, who diligently reviewed outlines, survey questions, and report drafts. Finally, we would like to recognize TDWI’s production team: Jennifer Agee, Rod Gosser, and Denelle Hanlon. Sponsors DataFlux, IBM, Informatica, SAP, Syncsort, and Talend sponsored the research for this report. 2 TDWI RESE ARCH Research Methodology and Demographics Research Methodology and Position Demographics Corporate IT professional 67% Consultants 26% Report Scope. Data integration (DI) has changed so quickly Business sponsors/users 7% and completely in recent years that it scarcely resembles older definitions. For example, some people still think of DI as merely ETL for data warehousing or data movement utilities Industry for database administration. Those basic tasks and use cases Financial services 17% are still prominent in DI practice. Yet, DI practices and tools Consulting/professional services 16% have broadened into many more techniques and use cases. Insurance 9% While it’s good to have options, it’s hard to track them and Software/Internet 8% determine in which situations they are ready for use. The Telecommunications 6% purpose of this report is to accelerate users’ understanding Healthcare 5% of the many new products and options that have entered DI Manufacturing (non-computers) 5% practices in recent years. It will also help readers map newly Retail/wholesale/distribution 4% available technologies, products, and practices to real-world Government: federal 4% use cases. Education 3% Survey Methodology. In November 2010, TDWI sent an Pharmaceuticals 3% invitation via e-mail to the data management professionals Media/entertainment/publishing 3% in its database, asking them to complete an Internet- Utilities 3% based survey. The invitation was also distributed via Web Other 14% sites, newsletters, and publications from TDWI and other firms. The survey drew responses from almost 350 survey (“Other” consists of multiple industries, each represented by 2% or less of respondents.) respondents. From these, we excluded incomplete responses and respondents who identified themselves as academics or vendor employees. The resulting completed responses of 323 Geography respondents form the core data sample for this report. United States 51% Survey Demographics. The wide majority of survey Europe 25% respondents are corporate IT professionals (67%), whereas the Asia 8% remainder consists of consultants (26%) or business sponsors/ Australia 4% users (7%). We asked consultants to fill out the survey with a Canada 4% recent client in mind. Africa 2% Central or South America 2% The financial services (17%) and consulting (16%) industries Middle East 1% dominate the respondent population, followed by insurance 3% (9%), software (8%), telecommunications (6%), and other Other industries. Most survey respondents reside in the U.S. (51%) or Europe (25%). Respondents are fairly evenly distributed Company Size by Revenue across all sizes of companies and other organizations. Less than $100 million 22% Other Research Methods. In addition to the survey, TDWI $100–500 million 14% Research conducted many telephone interviews with $500 million–$1 billion 11% technical users, business sponsors, and recognized data $1–5 billion 16% management experts. TDWI also received product briefings $5–10 billion 9% from vendors that offer products and services related to the More than $10 billion 18% best practices under discussion. Don’t know 10% Based on 323 survey respondents. tdwi.org 3 NE X T Generation Data Integration Introduction to Next Generation Data Integration All aspects of DI have Data integration (DI) has undergone an impressive evolution in recent years. Today, DI is a rich set improved significantly of powerful techniques, including ETL (extract, transform, and load), data federation, replication, of late synchronization, changed data capture, data quality, master data management, natural language processing, business-to-business data exchange, and more. Furthermore, vendor products for DI have achieved maturity, users have grown their DI teams to epic proportions, competency centers regularly staff DI work, new best practices continue to arise (such as collaborative DI and agile DI), and DI as a discipline has earned its autonomy from related practices such as data warehousing and database administration. This report brings the To help user organizations understand and embrace all that next generation data integration reader up to date on (NGDI) now offers, this report catalogs and prioritizes the many new options for DI. This report DI’s many changes literally redefines data integration, showing that its newest generation is an amalgam of old and new techniques, best practices, organizational approaches, and home-grown or vendor-built functionality. The report brings readers up to date by discussing relatively recent (and ongoing) evolutions of DI that make it more agile, architected, collaborative, operational, real-time, and scalable. It points to new platforms for DI tools (open source, cloud, SaaS, and unified data management) and DI’s growing coordination with related best practices in data management (especially data quality, metadata and master data management, data integration
Load more

Recommended publications
  • Managing Data in Motion This Page Intentionally Left Blank Managing Data in Motion Data Integration Best Practice Techniques and Technologies
    Managing Data in Motion This page intentionally left blank Managing Data in Motion Data Integration Best Practice Techniques and Technologies April Reeve AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO Morgan Kaufmann is an imprint of Elsevier Acquiring Editor: Andrea Dierna Development Editor: Heather Scherer Project Manager: Mohanambal Natarajan Designer: Russell Purdy Morgan Kaufmann is an imprint of Elsevier 225 Wyman Street, Waltham, MA 02451, USA Copyright r 2013 Elsevier Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions. This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein). Notices Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods or professional practices, may become necessary. Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information or methods described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility.
    [Show full text]
  • Oracle Warehouse Builder Concepts Guide
    Oracle® Warehouse Builder Concepts 11g Release 2 (11.2) E10581-02 August 2010 Oracle Warehouse Builder Concepts, 11g Release 2 (11.2) E10581-02 Copyright © 2000, 2010, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007).
    [Show full text]
  • Using ETL, EAI, and EII Tools to Create an Integrated Enterprise
    Data Integration: Using ETL, EAI, and EII Tools to Create an Integrated Enterprise Colin White Founder, BI Research TDWI Webcast October 2005 TDWI Data Integration Study Copyright © BI Research 2005 2 Data Integration: Barrier to Application Development Copyright © BI Research 2005 3 Top Three Data Integration Inhibitors Copyright © BI Research 2005 4 Staffing and Budget for Data Integration Copyright © BI Research 2005 5 Data Integration: A Definition A framework of applications, products, techniques and technologies for providing a unified and consistent view of enterprise-wide business data Copyright © BI Research 2005 6 Enterprise Business Data Copyright © BI Research 2005 7 Data Integration Architecture Source Target Data integration Master data applications Business domain dispersed management (MDM) MDM applications integrated internal data & external Data integration techniques data Data Data Data propagation consolidation federation Changed data Data transformation (restructure, capture (CDC) cleanse, reconcile, aggregate) Data integration technologies Enterprise data Extract transformation Enterprise content replication (EDR) load (ETL) management (ECM) Enterprise application Right-time ETL Enterprise information integration (EAI) (RT-ETL) integration (EII) Web services (services-oriented architecture, SOA) Data integration management Data quality Metadata Systems management management management Copyright © BI Research 2005 8 Data Integration Techniques and Technologies Data Consolidation centralized data Extract, transformation
    [Show full text]
  • Data Profiling and Data Cleansing Introduction
    Data Profiling and Data Cleansing Introduction 9.4.2013 Felix Naumann Overview 2 ■ Introduction to research group ■ Lecture organisation ■ (Big) data □ Data sources □ Profiling □ Cleansing ■ Overview of semester Felix Naumann | Profiling & Cleansing | Summer 2013 Information Systems Team 3 DFG IBM Prof. Felix Naumann Arvid Heise Katrin Heinrich project DuDe Dustin Lange Duplicate Detection Data Fusion Data Profiling project Stratosphere Entity Search Johannes Lorey Christoph Böhm Information Integration Data Scrubbing project GovWILD Data as a Service Data Cleansing Information Quality Web Data Linked Open Data RDF Data Mining Dependency Detection ETL Management Anja Jentzsch Service-Oriented Systems Entity Opinion Ziawasch Abedjan Recognition Mining Tobias Vogel Toni Grütze HPI Research School Dr. Gjergji Kasneci Zhe Zuo Maximilian Jenders Felix Naumann | Profiling & Cleansing | Summer 2013 Other courses in this semester 4 Lectures ■ DBS I (Bachelor) ■ Data Profiling and Data Cleansing Seminars ■ Master: Large Scale Duplicate Detection ■ Master: Advanced Recommendation Techniques Bachelorproject ■ VIP 2.0: Celebrity Exploration Felix Naumann | Profiling & Cleansing | Summer 2013 Seminar: Advanced Recommendation Techniques 5 ■ Goal: Cross-platform recommendation for posts on the Web □ Given a post on a website, find relevant (i.e., similar) posts from other websites □ Analyze post, author, and website features □ Implement and compare different state-of-the-art recommendation techniques … … Calculate (,) (i.e., the similarity between posts and ) … Recommend top-k posts ? … Overview 7 ■ Introduction to research group ■ Lecture organization ■ (Big) data □ Data sources □ Profiling □ Cleansing ■ Overview of semester Felix Naumann | Profiling & Cleansing | Summer 2013 Dates and exercises 8 ■ Lectures ■ Exam □ Tuesdays 9:15 – 10:45 □ Oral exam, 30 minutes □ Probably first week after □ Thursdays 9:15 – 10:45 lectures ■ Exercises ■ Prerequisites □ In parallel □ To participate ■ First lecture ◊ Background in □ 9.4.2013 databases (e.g.
    [Show full text]
  • Data Profiling: Designing the Blueprint for Improved Data Quality Brett Dorr, Dataflux Corporation, Cary, NC Pat Herbert, Dataflux Corporation, Cary, NC
    SUGI 30 Data Warehousing, Management and Quality Paper 102-30 Data Profiling: Designing the Blueprint for Improved Data Quality Brett Dorr, DataFlux Corporation, Cary, NC Pat Herbert, DataFlux Corporation, Cary, NC ABSTRACT Many business and IT managers face the same problem: the data that serves as the foundation for their business applications (including customer relationship management (CRM) programs, enterprise resource planning (ERP) tools, and data warehouses) is inconsistent, inaccurate, and unreliable. Data profiling is the solution to this problem and, as such, is a fundamental step that should begin every data-driven initiative. This paper explores how data profiling can help determine the structure and completeness of data and, ultimately, improve data quality. The paper also covers the types of analysis that data profiling can provide as well as how data profiling fits into an overall data management strategy. INTRODUCTION Organizations around the world are looking for ways to turn data into a strategic asset. However, before data can be used as the foundation for high-level business intelligence efforts, an organization must address the quality problems that are endemic to the data that’s available on customers, products, inventory, assets, or finances. The most effective way to achieve consistent, accurate, and reliable data is to begin with data profiling. Data profiling involves using a tool that automates the discovery process. Ideally, this automation will help uncover the characteristics of the data and the relationships between data sources before any data-driven initiatives (such as data warehousing or enterprise application implementations) are executed. THE CASE FOR DATA PROFILING Not so long ago, the way to become a market leader was to have the right product at the right time.
    [Show full text]
  • Rapid Data Quality Assessment Using Data Profiling
    Rapid Data Quality Assessment Using Data Profiling David Loshin Knowledge Integrity, Inc. www.knowledge-integrity.com © 2010 Knowledge Integrity, Inc. 1 www.knowledge-integrity.com (301)754-6350 David Loshin, Knowledge Integrity Inc. David Loshin, president of Knowledge Integrity, Inc, (www.knowledge-integrity.com), is a recognized thought leader and expert consultant in the areas of data governance, data quality methods, tools, and techniques, master data management, and business intelligence. David is a prolific author regarding BI best practices, either via the expert channel at www.b-eye-network.com, “Ask The Expert” at Searchdatamanagement.techtarget.com, as well as numerous books on BI and data quality. His most recent book, “Master Data Management,” has been endorsed by data management industry leaders, and his valuable MDM insights can be reviewed at www.mdmbook.com. David can be reached at [email protected]. MDM Component Model © 2010 Knowledge Integrity, Inc. 2 www.knowledge-integrity.com (301)754-6350 1 Business-Driven Information Requirements Driver Benefit Information Requirement Increased revenue, increased share, cross- Unified master customer data, Customer sell/up-sell, segmentation, targeting, retention, matching/linkage, centralized analytics, Intelligence customer satisfaction, ease of doing business quality data, eliminate redundancy Compliance, privacy, risk management, accurate Data quality, semantic consistency Risk & response to audits, prevent fraud across business processes, consistency, Compliance
    [Show full text]
  • Data Quality Fundamentals
    Data Quality Fundamentals David Loshin Knowledge Integrity, Inc. www.knowledge-integrity.com © 2010 Knowledge Integrity, Inc. 1 www.knowledge-integrity.com (301)754-6350 Agenda The Data Quality Program Data Quality Assessment Using Data Quality Tools Data Quality Inspection, Monitoring, and Control © 2010 Knowledge Integrity, Inc. 2 www.knowledge-integrity.com (301)754-6350 1 THE DATA QUALITY PROGRAM © 2010 Knowledge Integrity, Inc. 3 www.knowledge-integrity.com (301)754-6350 Data Quality Challenges Consumer data validation of supplied data provides little value unless supplier has an incentive to improve its product Data errors introduced within the enterprise drain resources for scrap and rework, yet the remediation process seldom results in long-term improvements Reacting to data integrity issues by cleansing the data does not improve productivity or operational efficiency Ambiguous data definitions and lack of data standards prevents most effective use of centralized “source of truth” and limits automation of workflow Proper data and application techniques must be employed to ensure ability to respond to business opportunities Centralization of integrated reference data opens up possibilities for reuse, both of the data and the process © 2010 Knowledge Integrity, Inc. 4 www.knowledge-integrity.com (301)754-6350 2 Addressing the Problem To effectively ultimately address data quality, we must be able to manage the Identification of customer data quality expectations Definition of contextual metrics Assessment of levels of data quality Track issues for process management Determination of best opportunities for improvement Elimination of the sources of problems Continuous measurement of improvement against baseline © 2010 Knowledge Integrity, Inc. 5 www.knowledge-integrity.com (301)754-6350 Data Quality Framework Data quality Measurement Policies Procedures expectations Governance Standards Monitor Training Performance © 2010 Knowledge Integrity, Inc.
    [Show full text]
  • Creating the Golden Record
    CreatingCreating thethe GoldenGolden RecordRecord BetterBetter DataData throughthrough ChemistryChemistry Donald J. Soulsby metaWright.com AgendaAgenda • The Golden Record • Master Data • Discovery • Integration • Quality • Master Data Strategy DAMADAMA –– LinkedInLinkedIn GroupGroup C. Lwanga Yonke - Information Quality Practitioner ......SpewakSpewak advocatedadvocated usingusing datadata dependencydependency toto determinedetermine thethe idealideal sequencesequence inin whichwhich applicationsapplications shouldshould bebe developeddeveloped andand implemented:implemented: “Develop“Develop thethe applicationsapplications thatthat createcreate datadata beforebefore thosethose thatthat needneed toto useuse thatthat data”data” (p.10).(p.10). ArchitectureArchitecture AdvocatesAdvocates WilliamWilliam SmithSmith – Entity Lifecycle CliveClive FinkelsteinFinkelstein - Information Engineering – CRUD RonRon RossRoss -- ResourceResource LifeLife CycleCycle AnalysisAnalysis CRUDCRUD inin aa PerfectPerfect WorldWorld CanonicalCanonical SynthesisSynthesis Broadly speaking, materials scientists investigate two types of phenomena. Both are based on the microstructures of materials: … ii. How do these microstructures influence the properties of the material (such as strength, electrical conductivity, or high frequency electromagnetic absorption)? http://www.its.caltech.edu/~matsci/WhatIs2.html Business VS Development Life Cycles Zachman Framework for Enterprise Architecture WHERE WHAT WHEN HOW WHO WHY CONTEXTUAL List of List of List of List of List
    [Show full text]
  • The Importance of a Single Platform for Data Integration and Quality Management
    helping build the smart and agile business The Importance of a Single Platform for Data Integration and Quality Management Colin White BI Research March 2008 Sponsored by Business Objects The Importance of a Single Platform for Data Integration and Quality Management TABLE OF CONTENTS DATA INTEGRATION AND QUALITY: UNDERSTANDING THE PROBLEM 1 The Evolution of Data Integration and Quality Software 1 Building a Single Data Services Architecture 3 Applications 3 Service-Oriented Architecture Layer 4 Data Services Techniques 5 Data Services Management and Operations 5 Choosing Data Services Products 6 BUSINESS OBJECTS DATA SERVICES PLATFORM 7 BusinessObjects Data Services XI 3.0 7 Getting Started: Success Factors 9 Brand and product names mentioned in this paper may be the trademarks or registered trademarks of their respective owners. BI Research The Importance of a Single Platform for Data Integration and Quality Management DATA INTEGRATION AND QUALITY: UNDERSTANDING THE PROBLEM Companies are fighting a constant battle to integrate business data and content while managing data quality in their organizations. Compounding this difficulty is the growing use of workgroup computing and Web technologies, the storing of more data and content online, and the need to retain information longer for compliance reasons. These trends are causing data volumes to increase dramatically. The growing number Rising volumes are not the only cause of data integration and quality issues, of data sources is however. The growing numbers of disparate systems that produce and distribute data causing data and content also add to the complexity of the data integration and quality integration problems management environment. Business mergers and acquisitions only exacerbate the situation.
    [Show full text]
  • Unifying the Practices of Data Profiling, Integration, and Quality (Dpiq) by Philip Russom Senior Manager, TDWI Research the Data Warehousing Institute
    TDWI MONOGRAPH SERIES OCTOBER 2007 Unifying the Practices of Data Profiling, Integration, and Quality (dPIQ) By Philip Russom Senior Manager, TDWI Research The Data Warehousing Institute SPONSORED BY TDWI Monograph Unifying the Practices of Data Profiling, Integration, and Quality (dPIQ) Table of Contents Defining dPIQ...................................................................................................................................3 Cycles and Dependencies in Data Profiling, Integration, and Quality..............................................5 The Unified dPIQ Cycle ...................................................................................................................8 Recommendations...........................................................................................................................12 About the Author PHILIP RUSSOM is the senior manager of TDWI Research for TDWI, where he oversees many of TDWI’s research-oriented publications, services, awards, and events. Prior to joining TDWI in 2005, Russom was an industry analyst covering BI at Forrester Research, Giga Information Group, and Hurwitz Group. He’s also run his own business as an independent industry analyst and BI consultant and was contributing editor with Intelligent Enterprise and DM Review magazines. Before that, Russom worked in technical and marketing positions for various database vendors. You can reach him at [email protected]. About Our Sponsor DataFlux enables organizations to analyze, improve, and control their data through
    [Show full text]
  • Data Profiling Model for Assessing the Quality Traits of Master Data Management
    International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8 Issue-6, March 2020 Data Profiling Model for Assessing the Quality Traits of Master Data Management Dilbag Singh, Dupinder Kaur Abstract: Enterprise Resource Planning (ERP) and Business Thus it is crucial to addresses these critical data entities Intelligence (BI) system demand progressive rules for (golden record) also known as master data, which is a single maintaining the valuable information about customers, truth version of record among different copies that belongs products, suppliers and vendors as data captured through to same person having different values. Master data is the different sources may not be of high quality due to human complete record of a human, like the data is about errors, in many cases. The problem encounters when this information is accessible across multiple systems, within same customers, patients, suppliers, partners, employees, and organization. Providing adequacy to this scattered data is a top other critical entities. For this purpose, a high quality and agenda for any organization as maintaining the data is consistent copy of master data becomes a primary task for complicated, as having high quality data. Master Data any organization. The processes and systems which Management (MDM) provides a solution to these problems by maintain this data are known as Master Data Management maintaining “a single reference of truth” with authoritative [5]. source of master data (Customer, products, employees etc). Master Data Management (MDM) is a holistic framework Master Data Management (MDM) is a highlighted concern now consisting of processes, tools and technologies which a day as valid data is the demand for strategic, tactical and provides coordinated master data in the enterprise.
    [Show full text]
  • Using Data Profiling, Data Quality, and Data Monitoring to Improve Enterprise Information
    SOFTWARE METRICS AND ANALYSIS As the amount of data an enterprise man- ages continues to grow, an organization must contend with two unique challenges: Using Data the sheer volume of that data and the multiple methods used to standardize this information across systems. These challenges often lead to a disparate, disconnected infor- Profiling, mation technology (IT) environment, where the information in any system can oper- ate in an isolated silo, creating multiple “versions of the truth.” In this article, the Data Quality, authors examine how one organization met its data management challenges by analyz- ing the data it receives, creating repeatable routines to improve the quality of that and Data data, and implementing controls to maintain high levels of data integrity. Through this combination of process improvement and technology, organizations can establish a Monitoring more team-based view of data that spans the IT and line-of-business environments—and create a quality-focused data management to Improve culture. Key words best practices, case study, data governance, Enterprise data management, data profiling, data quality Information BRETT DORR AND RICH MURNANE DataFlux Corporation INTRODUCTION Many companies have come to realize the substantial impact underlying data can have on every aspect of an organization. For the past 20 or 30 years, companies have purchased billions of dollars worth of applications designed to maximize the data they have in hopes of providing a data management structure for that information on an ongoing basis. These business applications are often designed to augment or administrate a business process or processes (for example, a customer relationship management (CRM) system manages sales and marketing activities).
    [Show full text]