VSAM to DB2 Conversion for JBHunt

By Raguraman B.

Date written(MM/DD/YY): 23 Jun 1999 Projects involved: JBHOCONV Target readers: Those who are involved in re-engineering of programs from VSAM to DB2

I hereby declare that this document is based on my personal experiences and / or experiences of my project members. To the best of my knowledge, this document does not contain any material that infringes the copyrights of any other individual or organization including the customers of Infosys.

Name of the Author. Raguraman B.

Abstract (Mention your contribution clearly):

General technical issues in Reengineering when Primary database is converted from VSAM to DB2 are explained in this document.

The project was executed for the client J.B.Hunt, a Truck Corporation in the US. The existing order maintenance software uses a set of VSAM files to manage data. The client decided to move the database to DB2 in a phased manner so that business is not interrupted in anyway.

The BOK gives details on the issues under the following topics:

1. Why DB2 was preferred over VSAM? 2. Smooth transition from VSAM to dB2. 3. Performance issues in DB2. 4. Issues in Easytrieve Programs due to DB2. 5. Issues in Project Management.

Keywords (Include the technologies used):

VSAM, DB2, COBOL, Easytrieve

H/W Platform: Mainframe

S/W Environment: MVS, COBOL, CICS, JCL, Easytrieve, VSAM, DB2 Appln. Type:

Project Type: Re-engineering

Check all the life cycle processes and support functions for which this BOK is relevant and check their respective boxes. (to check a box, double click the box and change the default value to "checked")

Corporate Planning

Marketing

Human Resource and Development

Training/Education and Research

Computer and Communication Services

Documentation Group

Finance and Administration

Travel and Cultural Issues

Purchase

STP

Project Initiation

Requirement Specification

Design

Build Reviews/Inspection and Testing

Acceptance

Maintenance

Quality Assurance and Productivity

Tools

Methodologies/Techniques

Project Management

Miscellaneous

1. Introduction

General technical issues in Reengineering when Primary database is converted from VSAM to DB2.

The existing order maintenance software uses a set of VSAM files to manage data. The client decided to move the database to DB2 in a phased manner so that Business is not interrupted in anyway.

The various issues are under the following topics.

1. Why DB2 was preferred over VSAM? 2. Smooth transition from VSAM to dB2. 3. Performance issues in DB2. 4. Issues in Easytrieve Programs due to DB2. 5. Issues in Project Management.

2. Section two

1. Why DB2 was preferred over VSAM?

The primary reason for having the data in DB2 is to integrate various data from various divisions on a common database. This lays the foundation for Data warehousing and Data mining takes a wider role in giving pertinent information to the management on its business trends.

DB2 among other things gives the advantage of Relational Data management, designed for security and performance. The data design becomes more flexible and copious amount of business logic additions as required by J.B. Hunt’s demanding customers is incorporated with relatively less labor without modification to the primary design.

The relational data management’s effectiveness is especially when having alternate index on more than 5 fields in VSAM data sets. Alternate index not only consumes memory unjustified by JBH’s business, also takes its toll on performance which is critical in On- line transactions. To circumvent this problem, JBH maintained a lot of alternate files instead of Alternate indices and these alternate files are inserted/updated whenever Main file is inserted/updated. This was enabled through an I/O module called ROCK. With DB2 this elaborate maintenance becomes redundant and upon careful usage of data by programs, we can create index on table’s columns which is more efficient and more application friendly.

2. Smooth Transition from VSAM to DB2.

This is achieved by first installing an I/O module which will get activated whenever a VSAM write/rewrite happens. Depending on the data mapping between VSAM and DB2 (On which Infosys was not involved), these I/O modules will insert/update onto corresponding DB2 tables. This has an advantage that, existing programs needn’t be touched and just an insertion of this I/O module into ROCK will do the job. This process is both relatively simple and has zero impact on the way existing programs are running.

Successful installation of this and periodic jobs that will check the sync between the data in VSAM and DB2 will ensure that the data is in sync at both VSAM and DB2 all times.

Phase 1 of the conversion involves the inquiry programs which refer to the Order maintenance database only for look ups, to be modified to look up from corresponding DB2 tables as per the mapping directly instead of VSAM database. Phase 1 for Update is slightly trickier in the sense that the update happens for both VSAM and DB2. The update of DB2 happens through the program and separate I/O modules and not through the ROCK.

Thus after phase 1 completion a program will ensure that the VSAM and DB2 database are perfectly in sync. Note that with half the programs converted for Phase 1 the other half remains unaffected as they still operate on both VSAM and DB2 through ROCK.

Phase 2 starts after the database is stabilised for performance and VSAM file is completely retired. VSAM access in the programs is duly commented and DB2 portion is left intact. Since Phase 1 is fully completed before Phase 2 all inquiries will happen only from DB2 and the conversion is deemed complete at this stage.

3. Performance issues in DB2.

The major difference in VSAM and DB2 regarding the Locking is while the VSAM has a row level lock, DB2 has a Page level lock. The primary key for Order maintenance being an Order number which is a 7 digit number, it is quite natural in business that successive order numbers will always be processed upon at a given time. This leads to a fact that several transactions will operate upon a single page, which when locked, forces other transactions to wait for the page. This problem is solved by inverting the order number and storing it in DB2. i.e, if VSAM order number is 1234567, DB2’s order number will be 765432100. This way successive order numbers get loaded in different pages and thereby increases the Pages loaded and thus gives split second response. This is important as a myriad of transactions is started by JBH’s customers/ onsite personnel through Internet, Delphi and APPC.

Secondly, Browse on VSAM’s alternate files is more efficient than a browse on DB2 as this due to relational DBMS will involve a join on table and a SORT on desired fields. This is solved for Online and Batch in different manner.

For online programs, fields involved in Browse are de-normalized and indexed in DB2 to avoid costly joins and unions. Not so recent orders are moved into another identical table which is warehoused. i.e, those tables are optimized for storage as they do not have indices. Online programs which predominantly use the recent orders thus provide a much better performance.

For Batch Programs the situation is slightly different. There are around 50 jobs that will browse the entire VSAM data set and make some report. JBH’s business constrains that all the orders for the past 2 years be browsed. Since data in warehoused tables are not optimized for performance, these Programs take extraordinarily longer time to complete and usually end up with SQLCODE -904 for resource being not available.

The above problem when analysed was found out that all the 50 jobs try to open the cursor and all 50 jobs process all the orders. A solution worked out was to unload the DB2 table into a Flat file using Platinum’s fast unload utility. A sample JCL to do this is appended in the end for reference. This is similar to File aid in other installations. One job will run prior to all other 50 jobs and will unload data from DB2 tables. Only those rows that were updated on that day (by looking at the last updated timestamp) would be unloaded. This job, since it deals with only that day's insertions/updates takes a relatively shorter time to run. Once the flat file is updated for today’s activities and merged with previous day’s output will yield a master flat file. The browse on VSAM data sets is converted to Browse on this master flat file. Since browse on Seq. Data sets is far more efficient than browse on indexed files, the jobs ran faster than before.

4. Easytrieve Programs.

This presented with a slightly trickier problem. The version of Easytrieve in JBH was CA-Easytrieve Plus 6.2. The nature of the easytrieve Program flow is recursive. It was necessary to code a cursor and fetch even for a simple select. Where a cursor was required, a simple select on the job statement would suffice.

There is a problem when a Easytrieve program having a DB2 call cannot call a COBOL program with DB2 call. This is because, the DB2’s loadlib acting on Easytrieve Program is different from that of COBOL. This necessitated writing new modules or incorporating the COBOL call logic inside Easytrieve itself. This is because, before conversion Easytrieve Program will not have any DB2 call whereas after conversion it would have.

5. Issues in Project management.

As described in the previous sections, a project like this depends on a lot of existing I/O modules and due to the vast nature of the system, it takes time to fully understand the role of each I/O module and its impact on new modules in both functionality and performance. Training new project members is extremely time consuming and very frustrating to new comers as they are faced with a myriad of routines and they have to get adjusted in a short span of time.

Uncertainty in VSAM to DB2 Mapping which was outside the scope of Infosys dealt a severe blow to both schedule and quality of the deliverables as we do not have positive control over the data design. This involved a considerable effort spent on reworks and took its toll on the personnel doing final inspection before delivery to JBH.

This also created frequent changes to existing modules and the bulk of Programs were released to us for conversion in the last 3 months of the project. This created a sense of undue urgency and needless to say had adversely affected smooth delivery of programs.

Also, in the wake of Year 2000 problem, this project of database reengineering got a relatively low priority from JBH which slowed down the process of inspection, delivery and clearance of issues.

JCL for fast unload from DB2 onto flat files.

//STEP0050 EXEC PGM=PTLDRIVM,COND=(0,NE),

// PARM='EP=UTLGLCTL/DB2P',

//* UNLOAD UTILITY PTLDRIVM SPECIFY DB2 SUBSYSTEM IN PARM

// REGION=7M

//STEPLIB DD DSN=SYS3.PLATINUM.PROD.LOADLIB,DISP=SHR

// DD DSN=SYS2.DB2.DB2P.DSNLOAD,DISP=SHR

//* DB2’S LOADLIB DSN410.RUNLIB.LOAD IN OTHER INSTLN.

//PTILIB DD DSN=SYS3.PLATINUM.PROD.LOADLIB,DISP=SHR

// DD DSN=SYS2.DB2.DB2P.DSNLOAD,DISP=SHR

//* LOADLIB OF PLATINUM UTILITIES

//PTIPARM DD DSN=SYS3.PLATINUM.PROD.PARMLIB,DISP=SHR

//PTIXMSG DD DSN=SYS3.PLATINUM.PROD.XMESSAGE,DISP=SHR

//PTIMSG DD SYSOUT=*

//SYSOUT DD SYSOUT=*

//SYSUDUMP DD SYSOUT=*

//SYSREC01 DD DSN=MOE.PSD.MOEP1200.ORDEXTR.DAILY.TORDER,

// DISP=(NEW,CATLG,DELETE),

// SPACE=(CYL,(10,5),RLSE),

// DCB=(RECFM=FB,LRECL=4,BLKSIZE=0)

//* ABOVE DECLN FOR DAILY FLAT FILE. THIS WILL

//* CONSEQUENTLY MERGED WITH PREVIOUS DATE’S TO

//* FROM MASTER FLAT FILE //SYSIN DD *

FASTUNLOAD

OUTPUT-FORMAT DSNTIAUL

SQL-ACCESS ONLY

SELECT ORD_I

FROM ALI.TORDER

WHERE CUR_ORD_STT_C LIKE 'C%'

AND REQ_TYP_C <> 'EQUIPMENT'

AND LST_UPD_S BETWEEN

'1999-06-14-21.30.11.643059'

AND '1999-06-15-00.01.01.852149'

/*

//* SYSIN GIVES THE QUERY THE RESULT OF WHICH WILL

//* BE UNLOADED INTO THE DAILY FLAT FILE.

References

1.

For further details contact:

Type in Name & Address here.

Phone: email: