Selecting a Data Warehouse Architecture David Septoff, SAS, Rockville, MD Kevin Simmons, SAS, Rockville, MD
Total Page:16
File Type:pdf, Size:1020Kb
Selecting a Data Warehouse Architecture David Septoff, SAS, Rockville, MD Kevin Simmons, SAS, Rockville, MD Abstract Data Warehouse projects have provided end-users with after implementation begins. Waiting until after extraordinary returns on investment. Maintaining those implementation begins usually increases the amount of extraordinary returns requires an architecture that is rework. Most developers will admit that it is more time robust, flexible and scalable. Careful thought must be consuming and difficult to do rework than to do the job given to the selection of the data warehouse and right the first time. For this reason, there is little doubt decision-support architecture. A mistake in this area about the value of a carefully architected data warehouse often leads to the deployment of an application that and decision-support environment. There has been, cannot provide long-term support for the data and however, a great deal of debate surrounding the "best" analytical needs of the end-user community. architecture to deploy. The selection of the "best" architecture will be driven by such factors as: available Selecting the most appropriate architecture requires technology, business requirements, commitment to the examination of a number of factors. Some of those development effort, skill level of the developers, and the factors are rooted in technical constraints, while others availability of resources. are rooted in how end-users are organized. There are also political issues that must be noted during selection There are several architectures, which can be supported of the architecture. The first step in selecting an via the SAS ® System as well as products from other architecture is to understand your decision-support vendors. These architecture options include: Enterprise needs. The second step is to understand the available Data Warehouse (BDW), Data Mart, Operational Data architectures. This paper covers some of the most Store (ODS) and Hybrid Data Warehouse (HOW). common architectures including: There are advantages as well as disadvantages for each architecture. • Enterprise Data Warehouse (BDW) • DataMart The goal of this paper is to briefly describe each • Operational Data Store (ODS) architecture, highlight its key components, and provide • Hybrid Data Warehouse (HOW). the reader with items to consider during the selection of the "best" architecture for their data warehouse project. This paper also offers some helpful hints for selecting the most appropriate architecture for your decision What is a Data Warehouse? support needs. The main purpose of the data warehouse is to provide This paper will be of interest to anyone who is currently end-users with access to data that can be used in in the planning stages of a data warehouse project. It is decision-support activities. Bill Inmon (the father of assumed that the reader understands the value of data data warehousing) defines the data warehouse as warehousing as it relates to decision-support activities. follows: "A data warehouse is a subject oriented, integrated, non-volatile, and time variant collection of data in support of management's decisions.") If you Introduction extend Inmon's definition to include a collection of data Data Warehouse development teams are not always given the time to carefully plan and evaluate design ) Inmon, W.H., Building the Data Warehouse, New decisions. Although this is not optimal, the data York: Wiley Computer Publishing, 1996, Second warehouse architecture can be determined, or modified Edition 32.5 that has physically separated from the source system and passed through quality assurance processes, you have the • ETL layer - Software is required to extract data SAS Institute's definition of a data warehouse. from the source systems, format the data for consistency and accuracy, integrate the data and To better understand Inmon's definition of the data load the data into the data warehouse. This warehouse, let's compare the data warehouse and an process is referred to as extraction, online transaction processing (OLTP) system. See Table transformation, and loading (ETL). Metadata 1. OLTPs are designed to collect and process (data about data) is a critical piece of this layer. transactional information that supports routine, day-to day business operations. OLTP processing includes the • Management layer - The management layer addition, update and deletion of extraordinary volumes facilitates day-to-day support activities for the of data. A common requirement for OLTPs is that data data warehouse. This support includes be processed with sub-second response time. Sub archiving data, backing up databases, second response time requires that developers focus on maintaining system security and job scheduling. achieving optimal performance and system throughput. Metadata is also a critical piece of this layer. Support for this type of computing environment usually involves data structures that are entity-relationship based • Exploitation layer - End-users need a facility to rather than subject oriented. This type of computing run queries against the data warehouse or data environment also puts pressure on developers to limit mart. The query tool is usually clientlserver data redundancy and maximize access speed for small based or web-based. Query tools need access subsets of data. to metadata as well as security information. The Enterprise Data Warehouse The Enterprise Data Warehouse (EDW) will usually support all (or most) of the corporation where there are requirements for flexible data access and usage across departments or business units. See Figure I. The EDW will be a common repository that is leveraged by multiple business units for decision-support activities and its design and construction will be based on the needs of the entire corporation. All the data in the EOW will be based on a common set of business rules, business dimensions, key performance indicators and Table 1: Data Warehouse and OLTP Comparison metrics. There is no requirement for the EDW to be centralized. Data Warehouse The term "enterprise" serves to communicate the depth and breadth of the data warehouse. The EOW can be Architectures centralized to a single location or distributed across the enterprise. In either case, the EOW must be managed What is a Data Warehouse by the Information Systems (IS) department. Management issues for the EDW should not be confused Architecture? with control issues. Distributed locations should be The architecture for a data warehouse consists of more allowed to control what data goes into the EDW, when it than software and the phYSical storage of data. It should is updated, and who has access to the data. Managing be viewed as an integration of technology and processes. the implementation of these decisions should fall to a A data warehouse architecture consists of several layers: group with responsibility for the corporate infrastructure, typically IS. • Conceptual layer - The conceptual layer includes industry and organizational aspects, Data for the EDW usually originates from several corporate vision, a clear deSCription of the operational systems. Via batch processes, the source problem to be solved and a deployment data is extracted from the operational systems, scrubbed, strategy. The primary output from this layer is integrated and transformed to meet the requirements of a list of data warehouse subjects and a data end-users. The data is then loaded into the repository model. and made available to the end-user community. 326 The major benefit for the independent data mart is that a The major benefit of the EDW architecture is that it single business unit or department could manage the facilitates access to an enterprise-wide view of the data. resources and personnel required for its deployment. The major disadvantage of the EDW architecture is that This typically has a minimal impact on IS resources and it can be very time consuming, resource intensive and can result in a very fast deployment. The major costly to deploy. Before selecting the type of disadvantages for the independent data mart are that it architecture, the data warehouse development team will only be accessible by the business unit for which it should be certain that access to enterprise-wide data is a was deployed and the lack of integration constrains the requirement. breadth of data to be analyzed. Before selecting this type of architecture, the data warehouse development team should be certain that access by a single business unit is a requirement. Enterprise Data Warehouse Dependent data marts are interconnected repositories that have been deployed in a distributed fashion. The dependent data marts may be deployed by individual business units or departments, but can be integrated to provide end-users with an enterprise-wide view of data. If the dependent data marts are designed properly, they may be the first steps toward an Enterprise Data Warehouse. A proper design mandates that the dependent data marts share a common set of business rules, business dimensions, key performance indicators and metrics. Source data for the dependent data mart is extracted from operational systems as well as systems that may be external to the business unit for which the data mart was constructed. Source data may also be extracted for an Enterprise Data Warehouse if one exists. Figure 1: Enterprise Data Warehouse The major benefit for the dependent data mart is that it supports an enterprise-wide view of data without The Data Mart requiring the IS resources