Miami-Dade County
Total Page:16
File Type:pdf, Size:1020Kb
MIAMI-DADE COUNTY Cognos Business Analytics and IDMS Integration at Miami-Dade County Abstract of the Program Miami-Dade County led the way as the first company in the world to successfully extract information from a CA IDMS network database using IBM Cognos Business Analytics. Miami- Dade County stores the majority of their critical data, such as Criminal Justice, Payroll, Traffic and Building information in CA IDMS mainframe databases and traditionally extracts this data using Computer Associates CA IDMS tools (CA ADS ADS/O and IDMS/Cobol/IDMS). Through the collaboration of CA Computer Associates, IBM, and Miami-Dade County, a driver was created for IBM Cognos to access this data directly by using Business Analytics tools without the need for exporting the data into a contemporary distributed environment. The problem or need for the program IDMS (Integrated Database Management System) is primarily a network (CODASYL) database management system for mainframes. Only CA tools could access IDMS data. IDMS mainframe technology did not have the capability to be accessed directly by contemporary business analytics software solutions. This limitation required data extraction, file transfer protocols, and script loading processes to expose IDMS, IMS, and DB2 mainframe data to contemporary technologies, including business analytics tools like IBM Cognos Analytics. Multiple data warehouses and operational data stores are loaded from the primary data sources originating from the IDMS mainframe platform and other relational data sources. More than 50 production IDMS databases are actively supported and used county-wide at Miami- Dade County. Access to data by the distributed world such as making the data available on an Oracle or MSSQL database requires adding multiple intermediate steps, using multiple technologies, and multiple points of failure to migrate data and make it available to analytic technologies. Description of the program After Computer Associates and IBM developed an API or driver capable of directly connecting IDMS Release 19 mainframe database data to IBM Cognos BI 10.2.2, a proof of concept project was undertaken. First, IDMS needed an upgrade to version 19 in production to expose the connectivity functionality to IBM Cognos BI 10.2.2. The effort to upgrade IDMS to version 19 started on 2/23/16 and completed on 6/14/15. The next step was to bring up an IBM Cognos BI version 10.2.2 environment. This project required a brand new installation, configuration, testing, and deployment of Linux and Windows servers for Cognos 10.2.2 environments: Internet production (public facing), 1 MIAMI-DADE COUNTY Intranet production, Development/Training (for both internet and intranet production) and Sandbox for configuration changes and system testing. Each of the four Cognos environments consists of 2 Linux guests, and each Linux guest contains 4 different components (3 basic Cognos, 1 Cognos utilities). Each Linux guest interfaces with 12 different Windows web gateways which also needed installation and configuration. The total number of components installed and configured were 32 Linux components and 38 Windows web gateways. The Sandbox environment, which allowed the proof of concept to complete, was the first environment brought up. This effort started on 5/15/2015 and completed on 8/5/2015. Once the Sandbox’s infrastructure and middleware (Cognos) installation and configuration completed, the Cognos-IDMS connectivity was possible. The next step was to establish successful connectivity using the IDMS driver. This effort started on 8/5/15. At first, the connectivity allowed immediate access to multiple records within a single table in IDMS. Further testing revealed that the driver could not manage multiple table relationships within IDMS. Miami-Dade County resolved this issue by establishing an IDMS relational layer with views that allowed the connection to access all tables defined within the view and expose a single record point for all related information. IDMS is a network database. In order to expose the same information in a relational mode, a SQL schema layer for use on a NONSQL DB was created. Using IDMS in a relational functionality, we defined data views for all records required from the mainframe. An IDMS analyst defines the relational view and submits that for inclusion to the IDMS database administrator who then builds the view and grants security permissions to the IBM Cognos authorized user. Using the IBM Cognos Framework Manager software, the administrators create the metadata model that defines how reports will view the data in Cognos. The following step was to federate the same IDMS data to existing related information residing in Oracle databases or data warehouses by using related primary keys. This was accomplished by adding the Oracle database to the metadata model and joining the data based on primary keys. The tasks included: 1. Upgrade IDMS to release 19.0. 2. Install and configure IBM Cognos BI ver 10.2.2 3. Obtain and install CA-IDMS driver on Cognos 10.2.2. 4. Connect to an IDMS network database. 5. Create Cognos Framework Manager Metadata model and package for Cognos 10.2.2. 6. Create reports against an IDMS database. 7. Connect to an IDMS dictionary. 8. Build scripts for IDMS SQL View/Schema to Criminal Justice data. 9. Build views from the scripts to Criminal Justice data. 10. Access metadata view schema and publish Cognos package for Criminal Justice data. 11. Access (Report on) Criminal Justice IDMS data via Cognos. 2 MIAMI-DADE COUNTY 12. Federate Criminal Justice IDMS data accessed through Cognos with an Oracle data source containing related complementary data. Matching primary keys on both datasets were used to join the data. The above tasks were carried out using the Sandbox (system test environment) Cognos 10.2.2 environment, and were completed on 10/19/15. After the completion of the proof of concept, all Cognos 10.2.2 environments were launched: the Cognos 10.2.2 Development/Training environment and Cognos 10.2.2 Internet Production environments launched on 2/19/16, and the Cognos 10.2.2 Intranet Production environment launched on 3/11/16. By 3/11/16, the Cognos-IDMS connectivity was enabled on all production and development/training systems. Following is a diagram that depicts a high-level representation of the configurations for each environment. Each of the 4 environments are configured in the same standard configuration for ease of operations, maintenance and problem resolution. 3 MIAMI-DADE COUNTY COGNOS ON LINUX SYSTEM Z Single Sign on Single Sign On Gateway NetScaler 12000 Gateway PRIMARY-Application SECONDARY-Application tier Linux on Z tier Linux on Z Audit DB Oracle 11gR2 - AIX VVM VVM Repository 1 Repository 2 Content Store Oracle 11gR2 - AIX Oracle 11gR2 - AIX Oracle 11gR2 - AIX Multiple domains: All Microsoft Windows Server Active Directory Domains 1. MIAMIDADE Domain 2. FIRE Domain 3. MDPD Domain 4. Aviation Domain 5. AOC Smtp.miamidade.gov 4 MIAMI-DADE COUNTY Use of Technology Both the IDMS 19 database system and Cognos 10.2.2 environments are utilized in this project. The IDMS database system is running on Mainframe z/OS 1.13. The Enterprise Analytics Cognos platform utilizes the following technology: Load-balanced web servers: Windows 2012 R2 servers running Microsoft IIS web servers are load-balanced using a NetScaler device. These web servers use SSL and Single Sign-on for secured Cognos connection and automatic sign on for a user accessing the environment. System z servers: Each server runs on Red Hat Linux for System z, and has Cognos 10.2.2 FP4 installed with: two installations of the Report Servers Application Tier component which run the reports, one installation of Content Managers (which access the report and create the SQL for the reports), VVM (Virtual View Manager to access SQL Server databases) and Transformer (to update data cubes). These components are load balanced, and fail-over of the Content Managers is handled by the Cognos software. System databases: The Cognos installation uses three (3) system Oracle databases to run the Cognos software. LDAP (Lightweight Directory Access Protocol) – application for accessing directory services over an IP network. LDAP combined with the web gateways provide single sign-on for users across the county on different Active Directory domains: Miami Dade, MDPD (Police), Aviation, Fire, AOC (Administrative Office of the Courts) and System Admin (for Cognos Administrators). MS Exchange – mail server so that the Cognos software can email reports to the users. Linux scripting: Linux scripts and jobs are used to streamline the sequential tasks required for operations, monitoring system status and stability. The Cost of the Program Prior to this project, the IDMS and IBM Cognos enterprise licenses were purchased and the environments were installed with a previous version. Upgrading/re-installing the environments was planned prior to the implementation of this project. Therefore, the actual project cost does not include this cost. The following are costs for a county to reproduce an enterprise environment. Below are the costs associated with the IBM Cognos BI environments. A county attempting to replicate this program would already have an IDMS database system and infrastructure in place due to the fact that this solution is intended to connect to and perform analysis on a legacy system. Any contemporary attempt to capture data would place the data on a distributed environment that 5 MIAMI-DADE COUNTY would not include