Using Data Profiling, Data Quality, and Data Monitoring to Improve Enterprise Information
Total Page：16
File Type：pdf, Size：1020Kb
SOFTWARE METRICS AND ANALYSIS As the amount of data an enterprise man- ages continues to grow, an organization must contend with two unique challenges: Using Data the sheer volume of that data and the multiple methods used to standardize this information across systems. These challenges often lead to a disparate, disconnected infor- Profiling, mation technology (IT) environment, where the information in any system can oper- ate in an isolated silo, creating multiple “versions of the truth.” In this article, the Data Quality, authors examine how one organization met its data management challenges by analyz- ing the data it receives, creating repeatable routines to improve the quality of that and Data data, and implementing controls to maintain high levels of data integrity. Through this combination of process improvement and technology, organizations can establish a Monitoring more team-based view of data that spans the IT and line-of-business environments—and create a quality-focused data management to Improve culture. Key words best practices, case study, data governance, Enterprise data management, data profiling, data quality Information BRETT DORR AND RICH MURNANE DataFlux Corporation INTRODUCTION Many companies have come to realize the substantial impact underlying data can have on every aspect of an organization. For the past 20 or 30 years, companies have purchased billions of dollars worth of applications designed to maximize the data they have in hopes of providing a data management structure for that information on an ongoing basis. These business applications are often designed to augment or administrate a business process or processes (for example, a customer relationship management (CRM) system manages sales and marketing activities). From that standpoint, much of the work on these applications has focused on just that—the processes that feed into a “data supply chain” (Dyché and Levy 2006, 12). However, an organization’s ability to make effective decisions is based largely on the data in front of decision makers. From executive-level decisions about mergers and acquisition (M&A) activity to a call-center representative making a split-second www.asq.org 9 Using Data Profiling, Data Quality, and Data Monitoring to Improve Enterprise Information decision about customer service, the information provide decision makers with actionable information an enterprise assembles on virtually every aspect of on potential disruptions and emerging threats to the organization—customers, prospects, products, employees, operating assets, and suppliers. inventory, finances, assets, or employees—can have iJET is an interesting use case for data management a significant effect on the organization’s ability to due to the amount of data it amasses on travelers and satisfy customers, reduce costs, improve productivity, trips each year. Along with data on client operating or mitigate risks. assets and suppliers, iJET tracks more than 14 million Good data hold the key to better decision making, client trips each year. Information on these trips is as they give an organization the background to assess received from thousands of different sources, such as business situations and locate trends that provide a employees, travel systems, and travel agencies. Due competitive edge. This is a “data payoff” that organiza- to the close relationship between its information and tions need to see, as the amount of business is estimated the services it provides, iJET is an information-driven to be doubling every 1.2 years (Lohr 2011). But an company. While manufacturing companies cultivate organization cannot simply cleanse and improve the and deliver products, or pharmaceutical companies quality of its data, and then expect these data to be research and refine drugs and therapeutics, iJET’s a static resource over time. Data are a fluid, dynamic, business model focuses on providing high-quality data and ever-changing resource, so even the most elegant on the travel process as well as dedicated services and data quality initiative is not a once-and-done activity support built on top of this information framework. (Fisher 2009, 128). To serve its network of organizations, iJET takes For example, companies acquire new customers, an employee-centric model. Worldcue Traveler allows products, inventory, and assets every day. Additionally, employees to log in and view their current and recent companies regularly add new data sources from new trips. Employees, corporate travel managers, and trading partners in the supply chain—or they add corporate security managers use this information for new customers through M&A activity or organic employee tracking, tax records, planning, and so on. growth. This dynamic data environment can lead Travel information is notoriously tricky, and iJET to a number of adverse business impacts, including faces an interesting mix of issues to harmonize and fraudulent disbursements, undue risk exposure, payroll synchronize this wealth of information. More than 50 overpayments, and underbilling (Loshin 2011, 7). To intelligence analysts and subject-matter experts have avoid these potential problems, a more expansive and access to 15,000 sources of intelligence that track repeatable data quality strategy is necessary, especially global threats that may impact employees, expatriates, in information-driven organizations. assets, and suppliers. The data flowing into iJET can come from more than 2500 data sources. CASE STUDY: DATA Outside of the sheer size and scope of the data that accumulate during millions of trips, the unique MANAGEMENT IN PRACTICE aspects of travel data pose their own set of issues: This article will examine the use of data manage- • Trips contain, on average, 3.8 segments (or legs). ment technologies by iJET International to better So, each trip is actually composed of several investigate and control a large and diverse set of subcomponents that belong to a master data travel information. iJET is an intelligence-driven element. provider of business resiliency and risk management solutions to nearly 500 multinational corporations and • Due to the number of times trips change and the governments, helping the organization prepare for and number of data sources iJET uses, information respond to global threats to its people, facilities, and on the same trip can arrive at iJET between supply chain assets. iJET’s Worldcue Global Control five and 10 times. Center technology solutions give decision makers The data are complex not just in the number of and organizations the real-time information needed elements related to a single trip (for example, multiple to anticipate and respond to business disruptions airline segments, hotel stays, rental cars), iJET also with a competitive edge. iJET’s Worldcue solutions faces a high degree of duplicate or redundant data 10 SQP VOL. 13, NO. 4/© 2011, ASQ Using Data Profiling, Data Quality, and Data Monitoring to Improve Enterprise Information elements. These characteristics lead to four data The presence and proliferation of nonstandard or management phases for iJET: poor-quality data can have an immediate impact on 1) Analyze the integrity of information through iJET’s ability to serve its client base. If an employee data profiling. Also known as data discovery, has an incorrect email or phone number, it will obvi- data profiling provides an overview of the ously be more difficult to reach that individual in an relative strengths and weaknesses of an emergency. If employees are assigned to the wrong organization’s data. This sets the stage for client or business unit, iJET will have more difficulty efforts to improve the quality of information responding to blanket requests from the organization later in the process. for details on the travel status of a group of employees. Without a standard way to reconcile that data, these 2) Manage inconsistent, incomplete, or incorrect errors can happen repeatedly, putting undue strain data. Relatively minor issues, like nonstandard on both the IT and business groups tasked with phone number patterns (919.555.1212 vs. managing data. 9195551212), often lead to data challenges The first step for iJET—and for any organization when trying to analyze data trends. In addition, that is undertaking a data management initiative—is missing fields or partial information complicate to conduct an accurate assessment of the data it has efforts to provide comprehensive service, while as well as new data sources as they become part of the also presenting a challenge for downstream corporate information environment. This phase of data data analysis efforts. discovery provides a blueprint of the logical steps of 3) Resolve duplicate records. Multiple instances a data management initiative, and it can become an of employees’ profiles or trips sourced from integral part of any ongoing data management practice. multiple systems make it difficult to provide Data profiling techniques generally fall into three an accurate picture of the risk associated groups: with a threat as well as make accurate deci- • Structure discovery: Do the patterns of the data sions to mitigate the risk associated with the match expected patterns? Do the data adhere threat. Resolving these duplicate records is to appropriate uniqueness and null value rules? the primary objective of iJET’s data program. • Content discovery: Are the data complete? Are 4) Conduct ongoing data monitoring and they accurate? Do they contain information analysis. Building off the previous stages, that is easily