Front cover IBM® Information Management Software Smarter Business Dynamic Information with IBM InfoSphere Data Replication CDC Log-based for real-time high volume replication and scalability High throughput replication with integrity and consistency Programming-free data integration Chuck Ballard Alec Beaton Mark Ketchie Anzar Noor Frank Ketelaars Judy Parkes Deepak Rangarao Bill Shubin Wim Van Tichelen ibm.com/redbooks International Technical Support Organization Smarter Business: Dynamic Information with IBM InfoSphere Data Replication CDC March 2012 SG24-7941-00 Note: Before using this information and the product it supports, read the information in “Notices” on page ix. First Edition (March 2012) This edition applies to Version 6.5 of IBM InfoSphere Change Data Capture (product number 5724-U70). © Copyright International Business Machines Corporation 2012. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . ix Trademarks . x Preface . xi The team who wrote this book . xii Now you can become a published author, too! . xvi Comments welcome. xvii Stay connected to IBM Redbooks . xvii Chapter 1. Introduction and overview . 1 1.1 Optimized data integration . 2 1.2 InfoSphere architecture . 4 Chapter 2. InfoSphere CDC: Empowering information management. 9 2.1 The need for dynamic data . 10 2.2 Data delivery methods. 11 2.3 Providing dynamic data with InfoSphere CDC . 12 2.3.1 InfoSphere CDC architectural overview . 14 2.3.2 Reliability and integrity . 16 Chapter 3. Business use cases for InfoSphere CDC . 19 3.1 InfoSphere CDC techniques for transporting changed data . 20 3.1.1 Option 1: Database staging. 21 3.1.2 Option 2: Message queue (MQ) based integration . 22 3.1.3 Option 3: File-based integration . 23 3.1.4 Option 4: InfoSphere DataStage Direct Connect . 24 3.2 Data warehousing and business intelligence . 24 3.2.1 Active data warehousing . 24 3.2.2 Slowly changing dimensions . 26 3.3 Consolidation. 28 3.3.1 Consolidation: Sample implementation 1 . 29 3.3.2 Consolidation: Sample implementation 2 . 30 3.4 Distribution . 31 3.4.1 Distribution: Sample implementation 1 . 32 3.4.2 Distribution: Sample implementation 2 . 33 3.5 Database migration . 35 3.5.1 Database migration: Sample implementation . 35 3.6 Application integration . 37 3.6.1 Application integration: Sample implementation 1 . 37 © Copyright IBM Corp. 2012. All rights reserved. iii 3.6.2 Application integration: Sample implementation 2 . 38 3.6.3 Application integration: Sample implementation 3 . 39 3.7 Integration with master data management . 40 3.7.1 Integration with master data management: Sample implementation 41 3.8 Integration with IBM Information Server . 43 3.8.1 Integration with IBM Information Server: Sample implementation . 43 3.9 Operational business intelligence . 45 3.9.1 Operational business intelligence: Sample implementation . 45 Chapter 4. Solution topologies . 47 4.1 Unidirectional replication . 48 4.2 Cascading replication . 49 4.3 Bidirectional replication . 52 4.4 Consolidation replication . 54 4.5 Data distribution replication . 57 4.6 Hub-and-Spoke replication with propagation . 58 4.7 Destination . 62 4.7.1 JMS Message Queue . 63 4.7.2 Flat files. 64 4.7.3 DataStage . 67 4.7.4 Web services. 71 Chapter 5. InfoSphere CDC features and functionality . 75 5.1 Transformations . 77 5.1.1 Column functions . 78 5.1.2 Journal control fields . 80 5.1.3 Joining. 80 5.1.4 User exits for customizations . 82 5.1.5 Considerations for using transformational functionality. 83 5.2 Replication modes. 83 5.2.1 Refresh . 83 5.2.2 Continuous mirroring. 87 5.2.3 Scheduled end (net change) . 88 5.3 Filtering . 89 5.3.1 Row level. 89 5.3.2 Column level . 90 5.4 Apply methods. 91 5.4.1 Standard . 91 5.4.2 LiveAudit . 92 5.4.3 Adaptive Apply . 93 5.4.4 Summarization . 93 5.4.5 Row consolidation . 94 5.4.6 Soft deletes . 95 iv Smarter Business: Dynamic Information with IBM InfoSphere Data Replication CDC 5.4.7 Custom apply methods (user exits). 96 5.4.8 Flat files. 96 5.4.9 DataStage direct connect . 97 5.4.10 JMS message queues. 98 5.5 Conflict detection and resolution . 99 Chapter 6. Understanding the architecture. 103 6.1 Component overview. 104 6.1.1 InfoSphere CDC instances . 107 6.1.2 Interoperability between the InfoSphere CDC components . 110 6.2 Management Console fundamentals. 111 6.2.1 Access Manager Interface . 112 6.2.2 Configuration Interface . 114 6.2.3 Monitoring Interface . 117 6.2.4 InfoSphere CDC API . 119 6.2.5 Access Server fundamentals . 120 6.3 The InfoSphere CDC engine . ..