Oracle Data Integrator Best Practices for a Data Warehouse Oracle Best Practices March 2008 Oracle Data Integrator Best Practices for a Data Warehouse PREFACE ................................................................................................................ 7 PURPOSE .................................................................................................................. 7 AUDIENCE ................................................................................................................. 7 ADDITIONAL INFORMATION .......................................................................................... 7 INTRODUCTION TO ORACLE DATA INTEGRATOR (ODI) .......................................... 8 OBJECTIVES ............................................................................................................... 8 BUSINESS -RULES DRIVEN APPROACH .............................................................................. 8 Introduction to Business rules ........................................................................... 8 Mappings .......................................................................................................... 9 Joins .................................................................................................................. 9 Filters................................................................................................................. 9 Constraints ........................................................................................................ 9 TRADITIONAL ETL VERSUS E-LT APPROACH ..................................................................... 9 UNDERSTANDING ORACLE DATA INTEGRATOR (ODI) INTERFACES ...................................... 10 A BUSINESS PROBLEM CASE STUDY .............................................................................. 11 IMPLEMENTATION USING MANUAL CODING ................................................................... 13 IMPLEMENTATION USING TRADITIONAL ETL TOOLS ......................................................... 15 IMPLEMENTATION USING ODI’S E-LT AND THE BUSINESS -RULE DRIVEN APPROACH .............. 17 Specifying the Business Rules in the Interface................................................. 17 Business Rules are Converted into a Process .................................................. 18 BENEFITS OF E-LT COMBINED WITH A BUSINESS -RULE DRIVEN APPROACH .......................... 20 ARCHITECTURE OF ORACLE DATA INTEGRATOR (ODI) ......................................... 23 ARCHITECTURE OVERVIEW .......................................................................................... 23 GRAPHICAL USER INTERFACES ..................................................................................... 24 REPOSITORY ............................................................................................................ 24 SCHEDULER AGENT ................................................................................................... 26 METADATA NAVIGATOR ............................................................................................. 27 Oracle Data Integrator Best Practices for a Data Warehouse Page 2 USING ORACLE DATA INTEGRATOR IN YOUR DATA WAREHOUSE PROJECT ......... 28 ODI AND THE DATA WAREHOUSE PROJECT ................................................................... 28 ORGANIZING THE TEAMS ............................................................................................ 28 REVERSE -ENGINEERING , AUDITING AND PROFILING SOURCE APPLICATIONS .......................... 30 DESIGNING AND IMPLEMENTING THE DATA WAREHOUSE ’S SCHEMA .................................. 32 SPECIFYING AND DESIGNING BUSINESS RULES ................................................................ 33 BUILDING A DATA QUALITY FRAMEWORK ...................................................................... 38 DEVELOPING ADDITIONAL COMPONENTS ...................................................................... 39 PACKAGING AND RELEASING DEVELOPMENT .................................................................. 40 VERSIONING DEVELOPMENT ....................................................................................... 40 SCHEDULING AND OPERATING SCENARIOS ..................................................................... 41 MONITORING THE DATA QUALITY OF THE DATA WAREHOUSE ........................................... 41 PUBLISHING METADATA TO BUSINESS USERS ................................................................. 41 PLANNING FOR NEXT RELEASES ................................................................................... 42 DEFINING THE TOPOLOGY IN ORACLE DATA INTEGRATOR .................................. 44 INTRODUCTION TO TOPOLOGY ..................................................................................... 44 DATA SERVERS ......................................................................................................... 45 Understanding Data Servers and Connectivity ............................................... 45 Defining User Accounts or Logins for ODI to Access your Data Servers .......... 47 Defining Work Schemas for the Staging Area ................................................. 47 Defining the Data Server ................................................................................. 48 Examples of Data Servers Definitions ............................................................. 49 Teradata ...................................................................................................................49 Oracle .......................................................................................................................50 Microsoft SQL Server ...............................................................................................50 IBM DB2 UDB (v8 and higher) ..................................................................................50 IBM DB2 UDB (v6, v7) and IBM DB2 MVS ................................................................51 IBM DB2 400 (iSeries) ..............................................................................................51 Flat Files (Using ODI Agent)......................................................................................52 XML ..........................................................................................................................52 Microsoft Excel ........................................................................................................53 PHYSICAL SCHEMAS ................................................................................................... 53 CONTEXTS ............................................................................................................... 54 LOGICAL SCHEMAS .................................................................................................... 56 PHYSICAL AND LOGICAL AGENTS .................................................................................. 56 THE TOPOLOGY MATRIX ............................................................................................ 56 OBJECT NAMING CONVENTIONS .................................................................................. 59 DEFINING ODI MODELS ....................................................................................... 61 INTRODUCTION TO MODELS ........................................................................................ 61 MODEL CONTENTS .................................................................................................... 62 IMPORTING METADATA AND REVERSE -ENGINEERING ....................................................... 63 Introduction to Reverse-engineering............................................................... 63 Reverse-engineering Relational Databases..................................................... 64 Non Relational Models .................................................................................... 64 Flat Files and JMS Queues and Topics ......................................................................64 Fixed Files and Binary Files with COBOL Copy Books ...............................................65 XML ..........................................................................................................................65 LDAP Directories ......................................................................................................66 Other Non Relation Models .....................................................................................66 Troubleshooting Reverse-engineering ............................................................ 67 Oracle Data Integrator Best Practices for a Data Warehouse Page 3 JDBC Reverse-engineering Failure ...........................................................................67 Missing Data Types ..................................................................................................67 Missing Constraints ..................................................................................................68 CREATING USER -DEFINED DATA QUALITY RULES ............................................................. 68 ADDING USER -DEFINED METADATA WITH FLEX FIELDS .................................................... 70 DOCUMENTING MODELS FOR BUSINESS USERS AND DEVELOPERS ...................................... 71 EXPORTING METADATA ............................................................................................. 72 IMPACT ANALYSIS , DATA LINEAGE AND CROSS -REFERENCES ............................................