Constructing a Meta Data Architecture
Total Page:16
File Type:pdf, Size:1020Kb
0-471-35523-2.int.07 6/16/00 12:29 AM Page 181 CHAPTER 7 Constructing a Meta Data Architecture This chapter describes the key elements of a meta data repository architec- ture and explains how to tie data warehouse architecture into the architec- ture of the meta data repository. After reviewing these essential elements, I examine the three basic architectural approaches for building a meta data repository and discuss the advantages and disadvantages of each. Last, I discuss advanced meta data architecture techniques such as closed-loop and bidirectional meta data, which are gaining popularity as our industry evolves. What Makes a Good Architecture A sound meta data architecture incorporates five general characteristics: ■ Integrated ■ Scalable ■ Robust ■ Customizable ■ Open 181 0-471-35523-2.int.07 6/16/00 12:29 AM Page 182 182 Chapter 7 It is important to understand that if a company purchases meta data access and/or integration tools, those tools define a significant portion of the meta data architecture. Companies should, therefore, consider these essential characteristics when evaluating tools and their implementation of the technology. Integrated Anyone who has worked on a decision support project understands that the biggest challenge in building a data warehouse is integrating all of the dis- parate sources of data and transforming the data into meaningful informa- tion. The same is true for a meta data repository. A meta data repository typically needs to be able to integrate a variety of types and sources of meta data and turn the resulting stew into meaningful, accessible business and technical meta data. For example, a company may have a meta data require- ment to show its business users the business definition of a field that appears on a data warehouse report. The company probably used a data modeling tool to construct the physical data models to store the data pre- sented in the report’s field. Let’s say the business definition for the field orig- inates from an outside source (i.e., it is external meta data) that arrives in a spreadsheet report. The meta data integration process must create a link from the meta data on the table’s field in the report to the business defini- tion for that field in the spreadsheet. When we look at the process in this way, it’s easy to see why integration is no easy feat. (Just consider creating the necessary links to all of the various types and sources of data and the myriad delivery forms that they involve.) In fact, integrating the data is probably the most complex task in the meta data repository implementation effort. Scalable If integration is the most difficult of the meta data architecture characteris- tics to achieve, scalability is the most important characteristic. A meta data repository that is not built to grow, and grow substantially over time, will soon become obsolete. Three factors are driving the current proliferation of meta data repositories: Continuing growth of decision support systems. As we discussed in Chapter 1, businesses are constantly demanding greater and greater func- tionality from their decision support systems. It is not unusual for both the size of a data warehouse database and the number of users accessing it to double in the first year of operation. As these decision support initia- 0-471-35523-2.int.07 6/16/00 12:29 AM Page 183 Constructing a Meta Data Architecture 183 tives continue to grow, the meta data repository must be able to expand to address the increasing functional requirements. Recognition of the value of enterprise-wide meta data. During the past three or four years, companies have begun to recognize the value that a meta data repository can bring to their decision support initiatives. Com- panies are now beginning to expand their repository efforts to include all of their information systems, not just decision support. I am aware of two Fortune 100 firms that are looking to initiate an enterprise-wide meta data solution. As soon as one of these major companies builds a repository to support all of its information systems, many others are likely to follow suit. Chapter 11, The Future of Meta Data, addresses the value of applying enterprise-wide meta data to corporate information systems. Increasing reliance on knowledge management. Knowledge manage- ment is a discipline that promotes the application of technology to identify- ing, capturing, and sharing all of a company’s information assets (e.g., documents, policies, procedures, databases, and the inherent knowledge of the company’s workforce). The concept of knowledge management is a good one: Capture the information assets and make them available throughout the enterprise. However, knowledge management is generating mixed reviews in the real world. Companies are just now beginning to understand that a meta data repository is the technical backbone that is necessary to implement a knowledge management effort. Software vendors and corporations alike are now expanding their meta data solutions to pro- vide a real-world approach to knowledge management. (Once again, Chap- ter 11, The Future of Meta Data, offers a detailed discussion of this topic.) META DATA: IT’S NOT JUST FOR DECISION SUPPORT A number of years ago I was speaking at a conference in Chicago about the value that meta data can bring to a decision support system. After the talk, a member of the audience approached me and asked why I limited my meta data discussion to only those topics under decision support, since meta data can support all of a company’s IT systems. I agreed that meta data can significantly aid a corporation’s IT systems, but explained that I did not address it during the talk because it was difficult enough to convince people that meta data can help a decision support system, let alone provide value to every information system in the company. My stance on this topic and my presentations have changed dramatically in the past few years. Now that people understand the value, they’re looking for the specifics of how to use enterprise-wide data most effectively and leverage it to their information systems. 0-471-35523-2.int.07 6/16/00 12:29 AM Page 184 184 Chapter 7 Robust As with any system, a meta data repository must have sufficient functional- ity and performance to meet the needs of the organization that it serves. The repository’s architecture must be able to support both business and techni- cal user reports and views of the meta data, as well as providing acceptable user access to these views. Some of the other functionality required from the meta data architecture includes: ■ Ability to handle time- or activity-generated events ■ Import/export capability ■ Support for data lineage ■ Security setup and authorization facilities ■ Archival and backup facilities ■ Ability to produce business and technical reports Customizable If the meta data processes are home-grown (i.e., built without the use of meta data integration or access tools), then customization is not a problem since the entire application is tailored for the specific business environ- ment. If, however, a company uses meta data tools to implement the repos- itory architecture (as most do), the tools need to be customized to meet the specific current and future needs of the meta data initiative. Customization is a major issue for companies that purchase prepackaged meta data solutions from software vendors. These solutions are generally so rigid in their architecture that they cannot fill the specific needs of any company. In the case of a meta data solution, one size definitely does not fit all! To be truly effective, these prepackaged solutions require a significant amount of customization to tailor them for each business environment. Open The technology used for the meta data integration and access processes must be open and flexible. For example, the database used to store the meta data is generally relational, but the meta data architecture should be suffi- ciently flexible to allow a company to switch from one relational database to another without massive architectural changes. Also, an open meta data repository enables a company to share meta data externally, and most important, make it accessible to all users. If, for exam- 0-471-35523-2.int.07 6/16/00 12:29 AM Page 185 Constructing a Meta Data Architecture 185 ple, a company decides to Web-enable all of its meta data reports, the processes for providing access to these reports should be able to use any standard Web browser. Key Elements of Meta Data Architecture In addition to the general characteristics of good architecture, all good data repositories share a set of key elements that are essential for success, regardless of the architectural approach used to build the repository. In short, all good repositories: ■ Are based on clear, well-defined management direction ■ Use the same front end as the data warehouse ■ Use the same entity and attribute naming standards throughout ■ Incorporate multiple sources of meta data ■ Include automated and reusable processes ■ Use a standardized integration process ■ Use a flexible meta model ■ Manage multiple versions of meta data ■ Incorporate update facilities ■ Use a component-based multitier architecture ■ Incorporate a security management scheme ■ Incorporate cross-tool meta data dependency and lineage Clear Management Direction A set of clear, well-defined repository requirements are critical to the suc- cess of the meta data project. While this may not seem like an architectural issue, it is. I have seen more than one repository effort in which manage- ment changes in direction caused severe changes in the repository archi- tecture. Probably the most extreme case of misdirection that I dealt with involved a company that, for many years, depended on UNIX-based hardware and a Sybase database.