The Importance of a Single Platform for Data Integration and Quality Management
Total Page:16
File Type:pdf, Size:1020Kb
helping build the smart and agile business The Importance of a Single Platform for Data Integration and Quality Management Colin White BI Research March 2008 Sponsored by Business Objects The Importance of a Single Platform for Data Integration and Quality Management TABLE OF CONTENTS DATA INTEGRATION AND QUALITY: UNDERSTANDING THE PROBLEM 1 The Evolution of Data Integration and Quality Software 1 Building a Single Data Services Architecture 3 Applications 3 Service-Oriented Architecture Layer 4 Data Services Techniques 5 Data Services Management and Operations 5 Choosing Data Services Products 6 BUSINESS OBJECTS DATA SERVICES PLATFORM 7 BusinessObjects Data Services XI 3.0 7 Getting Started: Success Factors 9 Brand and product names mentioned in this paper may be the trademarks or registered trademarks of their respective owners. BI Research The Importance of a Single Platform for Data Integration and Quality Management DATA INTEGRATION AND QUALITY: UNDERSTANDING THE PROBLEM Companies are fighting a constant battle to integrate business data and content while managing data quality in their organizations. Compounding this difficulty is the growing use of workgroup computing and Web technologies, the storing of more data and content online, and the need to retain information longer for compliance reasons. These trends are causing data volumes to increase dramatically. The growing number Rising volumes are not the only cause of data integration and quality issues, of data sources is however. The growing numbers of disparate systems that produce and distribute data causing data and content also add to the complexity of the data integration and quality integration problems management environment. Business mergers and acquisitions only exacerbate the situation. Data quality Most organizations use a variety of software products to handle the integration of management in many disparate data and content, and to manage data quality. Often, custom solutions are companies is required for complex and legacy data environments. Although data integration immature projects have grown rapidly, budget and time pressures often lead to data quality issues being ignored by project developers. The result is that data quality management has not kept pace with the growth of data integration projects, and its use in many companies is still immature. Vendors are now Business user complaints and compliance legislation are forcing IT groups to devote providing more energy and resources to solving data quality problems. Nevertheless, many data consolidated data quality projects are still implemented separately from those for data integration. One integration and data reason for this is that in the past data quality tools have been developed and marketed quality products by a different set of vendors than those that supply data integration products. This has led to fractured purchasing strategies and skills development in IT groups. Vendor acquisitions and mergers have led to consolidated solutions, but product integration issues still remain. A data services If companies are to manage the integration and quality of the ever-increasing architecture is information mountain in their organizations, they need to design and build a data required for services architecture that provides a single environment for enterprise-wide business enterprise-wide data data and content integration and quality management (see Figure 1). This paper integration and examines the evolution of the data integration and quality industry, and explains the quality management benefits of moving toward a single data services architecture. It outlines requirements for a software platform for supporting such an architecture, and, as an example, reviews the BusinessObjects™ Data Services XI Release 3 platform from Business Objects, an SAP company. THE EVOLUTION OF DATA INTEGRATION AND QUALITY SOFTWARE Although data integration and quality problems have been widespread in companies throughout the history of computing, they deteriorated noticeably when organizations moved away from centralized systems to using distributed processing involving BI Research 1 The Importance of a Single Platform for Data Integration and Quality Management client/server computing, and more recently, Web-based systems. While there is no question that the move toward distributed processing systems improved access to data, which in turn enhanced business user decision-making and action-taking, it nevertheless increased the complexity of data integration and quality management tasks in organizations. Figure 1. Data Services: a Single Environment for Data Integration and Quality Management Data warehousing Improvements in data integration and quality came with the introduction of data and BI projects have warehousing and business intelligence (BI). The business intelligence market has helped improve data seen tremendous growth, and for many organizations business intelligence has quality become a key asset that enables them to optimize BI operations to reduce costs and maintain a competitive advantage. Business intelligence applications in these companies have become mission-critical because of the important role they play in the decision-making process. This reliance will grow as companies move toward using business intelligence, not only for strategic and tactical decision-making, but also for driving daily and intraday business operations. Data integration and The use of data warehousing and business intelligence has led to a much better quality management understanding of how business data flows through the business and how it is used to is an enterprise-wide make decisions. This is especially true for legacy system data, which is often poorly problem documented. This understanding is helping organizations deploy other data integration and quality projects that may not be directly related to business intelligence. Master data management is an example here. The result is that more organizations are now viewing data integration and quality as an enterprise-wide problem, not just an issue to be solved when building a data warehouse and business intelligence applications. BI Research 2 The Importance of a Single Platform for Data Integration and Quality Management An enterprise-wide Although companies have increased their spending on data integration and quality data services products, a single enterprise-wide data services solution has often remained elusive environment has due to the complexity of the tasks involved, and also because of the lack of a remained elusive consistent approach to information management across the enterprise. The solution is to develop an enterprise data services architecture, deploy a single and open data services platform to support this architecture, fill any gaps in the platform with third-party or custom-built software, and gradually evolve existing data integration and data quality projects to support the new data services environment. Six key aspects of an The main characteristics of an enterprise-wide data services architecture are as enterprise-wide data follows: services architecture • A single environment for data integration and data quality management • A common developer user interface and workbench • A single set of source data and content acquisition adapters • Shared and reusable data integration and data quality cleansing transforms • A single operations console and runtime environment • Shared metadata and metadata management services Many benefits to Although it will take time for organizations to move toward a single data services having a single data environment that supports both data integration and data quality management there services environment are significant benefits to doing so: • Organizations are more effective and competitive because they have access to consistent and trusted data • IT architecture is simpler, which reduces IT maintenance and development costs • Development cycle time is reduced due to a common data integration and data quality management environment • Data standards are easier to enforce and maintain because data integration and data quality processes can be shared and reused across projects BUILDING A SINGLE DATA SERVICES ARCHITECTURE Figure 2 illustrates the key requirements for building a single enterprise-wide data services environment. These requirements fall into four main areas: applications, application interfaces, techniques, and management. Applications The applications component represents those business applications that require data services for improving data quality and integrating data and content. Business transaction processing, master data management and business intelligence are key examples here. The move toward a service-oriented architecture (SOA) based on Web services is adding applications such as business content management and business collaboration to the applications mix. BI Research 3 The Importance of a Single Platform for Data Integration and Quality Management Figure 2. Data Services Requirements Service-Oriented Architecture Layer Most data quality and integration projects involve batch applications that gather data from multiple sources, clean and integrate it, and then load the results into a target data file or database. With demand growing for lower-latency data and a services- based architecture, this model of data integration processing must be enhanced and made more dynamic. Developers need a Developers now want to build applications that can use data services interactively, set of dynamic