Building VLDB for BI Applications on OS/390: Case Study Experiences Seungrahn Hahn, Nancy S. Ferguson, Markku Koskiniemi, Tarkeshwar Shrimali International Technical Support Organization www.redbooks.ibm.com SG24-5609-00 SG24-5609-00 International Technical Support Organization Building VLDB for BI Applications on OS/390: Case Study Experiences January 2000 Take Note! Before using this information and the product it supports, be sure to read the general information in Appendix F, “Special Notices” on page 259. First Edition (January 2000) This edition applies to DB2 for OS/390 Version 5, Program Number 5655-DB2, SmartBatch for OS/390 V1R2, OS/390 eNetwork Communications Server V2R6, DominoGo Webserver V5R0M0 for use with the OS/390 Version 2 Release 6. Comments may be addressed to: IBM Corporation, International Technical Support Organization Dept. HYJ Mail Station P099 2455 South Road Poughkeepsie, NY 12601-5400 When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you. © Copyright International Business Machines Corporation 2000. All rights reserved. Note to U.S Government Users – Documentation related to restricted rights – Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp. Contents Figures....................................................ix Tables....................................................xiii Preface...................................................xv Howtousethisbook............................................xv The Team That Wrote This Redbook ................................xvi CommentsWelcome........................................... xvii Chapter 1. Introduction to VLDB ...............................1 1.1 The demand for VLDB data warehouses .......................1 1.2VLDBissuesandconsiderations.............................2 1.3Thecasestudies.........................................5 1.3.1IBMS/390TeraplexIntegrationCentermission..............6 1.3.2 Why choose S/390 for BI applications .....................7 1.4Banking-casestudyA.....................................7 1.4.1Objectivesofthetests.................................8 1.4.2BIapplicationsinthetests..............................8 1.4.3Testsystemconfiguration...............................9 1.5HealthInsurance-casestudyB.............................10 1.5.1Objectivesofthetests................................11 1.5.2BIapplicationsinthetest..............................11 1.5.3Testsystemconfiguration..............................12 1.6IBM-casestudyC.......................................13 1.6.1Objectivesofthetest.................................13 1.6.2Testsystemconfiguration..............................14 1.7Softwareusedinthecasestudies...........................15 1.7.1 Client tools available to use the data warehouse . ...........15 1.8 Naming conventions of this redbook..........................16 Chapter 2. Database design for VLDB..........................17 2.1 Logical DB design . ......................................17 2.2PhysicalDBdesign......................................20 2.3Partitioningstrategy......................................23 2.3.1 Partitioning methods. .................................24 2.4Indexconsiderations.....................................32 2.4.1 Design non-partitioned index for query parallelism ...........32 2.5Compression...........................................33 2.6DSTATSstatistics.......................................38 2.7DB2configuration........................................40 2.8 DASD storage management using DFSMS and RVA . ...........41 © Copyright IBM Corp. 2000 iii 2.8.1Workfiledatasets...................................43 2.9Casestudytests.........................................44 2.10 Case study A: physical database design .....................44 2.10.1 Deviations from the logical model . ......................45 2.10.2Partitioningconsiderations............................45 2.10.3 Denormalization . .................................47 2.10.4Datasetplacement..................................47 2.10.5Indexstrategy......................................49 2.10.6 Non-partitioned indexes . ...........................50 2.10.7 Buffer pool allocations ...............................50 2.11CasestudyB:physicalDBdesign..........................50 2.11.1 Database design considerations . ......................51 2.11.2 Physical database design considerations . ................52 2.11.3Partitioning........................................53 2.11.4PhysicalDB2objectsplacement........................54 2.11.5 Buffer pool allocation ................................54 2.12Summary.............................................55 Chapter 3. Data warehouse population.........................57 3.1Architecturalconsiderations................................58 3.1.1Initialload..........................................58 3.1.2 Refresh of data warehouse. ...........................63 3.1.3Criticalsuccessfactors(CSF)..........................65 3.1.4 Risk management. .................................66 3.1.5Applicationparallelism................................67 3.2 Implementing data population...............................71 3.2.1 DB2 Utility . ......................................71 3.2.2 LOAD and RUNSTATS parallelism . ......................72 3.3Casestudytests.........................................72 3.3.1Objectives.........................................72 3.4Initialload..............................................73 3.4.1Objectives.........................................73 3.4.2 Methodology. ......................................73 3.4.3Thecasestudytestresults.............................77 3.5 Monthly refresh of data warehouse...........................79 3.5.1Objectives.........................................79 3.5.2 Methodology. ......................................79 3.5.3Casestudytestresults................................82 3.6Concurrency............................................83 3.6.1Objectives.........................................83 3.6.2 Methodology. ......................................83 3.6.3Casestudytestresults................................84 3.7 1 TB database build ......................................87 iv Building VLDB for BI Applications on OS/390: Case Study Experiences 3.8Summary..............................................90 Chapter 4. VLDB utility maintenance ..........................91 4.1 Reorganization ..........................................93 4.2Backup................................................95 4.3Recovery..............................................96 4.4Summary..............................................97 Chapter 5. Data mart on S/390................................99 5.1Architecturalconsideration................................100 5.1.1Twophysicalsystems................................101 5.1.2Threeormore(multi-)physicalsystems..................102 5.1.3 Data warehouse and data mart coexistence considerations . 104 5.2Performance...........................................105 5.3 Population considerations ................................106 5.3.1 Create the database and its tables . .....................110 5.3.2Initialloadofdatamart...............................111 5.4 Data mart maintenance . ................................116 5.5 Resource management: intelligent resource sharing . ..........119 5.6Casestudytests........................................120 5.6.1 Data mart population ................................121 5.6.2 Intelligent resource sharing between DM and DW ..........125 5.7Summary.............................................130 Chapter 6. End-user access methodologies....................133 6.1Toolselectionconsiderations..............................133 6.2 Options for end-user data access . ..........................135 6.2.1 Web enablement . ................................135 6.2.2 Distributed connections to DB2.........................141 6.2.3OLAPtools........................................142 6.2.4 QMF for Windows. ................................146 6.3Casestudytests........................................146 6.3.1Webcasestudytests................................147 6.3.2OLAPcasestudytest................................155 6.4Summary.............................................160 Chapter 7. Workload management in data warehouse environment .163 7.1 Workload management issues . ..........................163 7.2 Workload Manager concepts ..............................165 7.3WLMstrategy..........................................168 7.3.1 Favoring short-running queries .........................168 7.3.2Favoringcriticalqueries..............................170 7.3.3 Preventing system resource monopolization...............171 7.3.4 Enabling continuous computing . .....................173 v 7.3.5 Data warehouse and data mart coexistence ...............174 7.3.6 Managing external sources of work .....................176 7.4 Implementing WLM for business intelligence . ...............177 7.4.1ConsiderationswhenimplementingWLM.................178 7.4.2 Business intelligence workload characteristics . ..........178 7.4.3 Thoughts on classification . ..........................179 7.4.4 Sample data warehouse analysis . .....................180 7.4.5Creatingaservicedefinition...........................181 7.4.6WLMpolicies......................................181 7.4.7 Workloads and service classes.........................182 7.4.8Settingdurations...................................188