Project Code: MXC-W061

DML to SLX Integration at JPMorgan Chase

A Major Qualifying Project Report Submitted to the faculty of Worcester Polytechnic Institute In partial fulfillment of the requirements for the Degree of Bachelor of Science

Submitted By:

______Michael J. Kristan Megan R. Slonski

Project Center: Wall St. New York, NY B Term 2006

Sponsoring Agency: JPMorgan Chase & Co.

Submitted To: Project Advisors:

______Michael Ciaraldi Arthur Gerstenfeld

On-Site Liaisons: Marisa Giliberti Danielle Rumore Brian Kenney

Abstract

This MQP aimed to assist JPMorgan Chase with the successful completion of the high priority DML to SLX Integration project. In the pre-design phase, work was carried out to map data fields between the two systems and to analyze the anticipated effect of this project on server performance. Concurrently, a thorough project plan was created to be used in a critical path analysis. Struggles in creating this led to the recommendation of a process for better managing projects.

This document is not intended for public distribution. i Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Authorship

This Major Qualifying Project is a collaborative effort of both team members. In compiling research and writing this paper each of us dedicated a significant amount of time and effort. There was an overall equal contribution to the executive summary, introduction, and background section made by both team members. Megan Slonski was mainly responsible to the sections pertaining to the critical path analysis and project management process improvements. Michael Kristan was primarily responsible for the sections regarding capacity planning and data mapping.

This document is not intended for public distribution. ii Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Executive Summary

As the 17th largest corporation in the United States, JPMorgan Chase & Co. provides a wide range of banking and financial services to customers worldwide. One such service is known as securities lending. Securities lending is the market practice whereby securities are temporarily transferred from a lender to a borrower. For the period of the loan, the lender is protected against default through receipt of collateral from the borrower. JPMorgan acts as the facilitator in this transaction by matching lenders to qualified borrowers. JPMorgan also invests the collateral received on the loan and splits the return on investment between all three parties.

In order to maintain its outstanding reputation, JPMorgan Chase & Co. must utilize new technology, maximize efficiency of its internal processes, and continually improve the services provided to its clients. Understanding the importance of these key business strategies, JPMorgan executives decided that the software Securities Lending

Xpress (SLX) should be the sole front-end trading platform used to initiate all securities lending loans globally. The reason for this decision is that SLX has checks built in that ensure compliance to audit and government regulations. Also SLX is built on the most up-to-date, flexible platform when compared to the other systems. This will allow future changes to be made more quickly and at a lower cost. In this vision the other systems, including International Securities Lending System / Dan McLune (DML), would only be used as loan maintenance systems and would cease to perform any loan initiation activity.

The current state of the business requires that several incremental projects be completed in order to turn this vision into a reality. One such project is to integrate DML and SLX so that information pertaining to loan maintenance activity in DML is passed in the form

This document is not intended for public distribution. iii Permission must be obtained from JPMorgan Chase & Co. to disclose this document. of a message to SLX throughout the day and overnight using a feedback loop. Only then

can SLX use this information to accurately update lender limit and borrower credit line

utilizations.

The work done on this Major Qualifying Project (MQP) aimed to assist JPMorgan

with the successful completion of this important project. Specifically, work was done to

accomplish four main goals. The first two relate to pre-design activities that needed be

completed prior to the design and build phases of the project. Success of the pre-design

activities would contribute to the successful implementation of the feedback loop. The

first goal of this MQP was to assist in mapping and matching the data fields between the two systems to ensure that SLX receives all of the information it needs from DML to properly update its database. The second goal was to analyze the anticipated effects of the feedback loop on servers and mainframes so that additional resources could be introduced and performance would not be compromised. The third goal of this MQP was to gather the information needed to conduct a critical path analysis that would examine ways to expedite the delivery of the project. From this a fourth goal was established, which was to recommend possible improvements to project management practices that could result in shorter project durations, fewer delays, and more accurate estimates of project cost and delivery dates for this and future projects.

The objective of the data mapping exercise was to examine the fields that DML has defined in a loan snapshot and compare that to the data fields SLX needs to receive to properly update borrower credit and lender limit utilizations. One of the challenges was that the fields in DML do not directly map to the fields in SLX. Therefore, an in-depth analysis needed to be conducted to examine how the data fields from DML should be

This document is not intended for public distribution. iv Permission must be obtained from JPMorgan Chase & Co. to disclose this document. manipulated so that they are in a useable format for SLX. As part of the data mapping exercises, sample loans were observed from the production environment. Unfortunately a problem occurred when taking the raw loan data from DML and inputting it into a

Microsoft Access database that would allow it to be analyzed. This raw data is known as a loan master file. There were more loan records in the loan master file than Microsoft

Access allowed to be imported at once. To get around this, a program was developed that would split the file into multiple, smaller files that could be imported into Microsoft

Access one at a time. With this accomplished, the JPMorgan team could begin to analyze how DML fields would map to SLX fields.

The second objective, capacity planning, was to carry out a comprehensive analysis on the SLX servers and to gather enough information to make infrastructure decisions moving forward. The primary deliverable was a document that gives the reader an understanding of the resources required to run SLX in a production environment and to analyze the impact that the DML feedback loop will have on the systems. Information was gathered from many sources and compiled together to form a complete picture of the resources that SLX depends on. One of the primary data sources was a feedback loop that was formerly created as part of a project to integrate SLX with the domestic counterpart to DML known as SLMU. The plan was to investigate the current performance of the SLX servers with the SLMU feedback loop in use. The next step was to compare the number of SLMU messages to the anticipated number of DML messages to predict the effect that the DML feedback loop would have on SLX server performance.

The analysis showed that the anticipated volume of DML messages between

11AM and 1PM is so high that it will diminish the SLX server performance. The

This document is not intended for public distribution. v Permission must be obtained from JPMorgan Chase & Co. to disclose this document. performance will be diminished to such an extent that SLX will not be able to process

messages from DML in the time deemed acceptable by business executives. This will

hinder the booking of new loans and the updating of lender and borrower utilizations which will result in lost opportunity for JPMorgan.

The third objective of this MQP was to gather the information needed to conduct a critical path analysis that would examine ways to expedite the delivery of the DML to

SLX Integration Project. This project was important because it was deemed a high priority within the firm. A number of steps needed to be completed to gather the information that was required to utilize the critical path method in the most effective way.

The basis for this analysis would be a thorough and cohesive project plan, which was created as a product of this objective. Meetings were held with each member of the project team to determine all project tasks, time estimates for the completion of each task, and the network of task dependencies. Work was also done with the project delivery manager to compile a list of all resources that would contribute to the project and to determine the percentage of each resource’s time that would be allocated to this initiative.

All of this information was then incorporated into the project plan to be used in the future management of the project.

The difficulty experienced in compiling the information needed for a thorough project plan exposed the need for the fourth goal of this MQP, which was to recommend improvements to project management processes. The improvements were aimed at making the processes more efficient, streamlined, simple and/or error-proof. In order to make these improvements, a thorough investigation of project management practices was conducted surrounding the following activities: providing project cost and duration

This document is not intended for public distribution. vi Permission must be obtained from JPMorgan Chase & Co. to disclose this document. estimates to the Investment Council for project funding, creating an accurate project plan and using it keep the project on schedule, and tracking action items that arise during team meetings. This evaluation showed that there is a lot of room to make improvements to how projects are managed. The biggest observation is that there is not a defined process based on best practices in place in the Securities Lending Technology group. A different process is used for each new project. This results in confusion, poor practices being used, and time wasted by all team members learning a new system for each project. Therefore, we have developed a process for managing projects that encompasses the aforementioned aspects of project management to be used by all project managers. It is based on the inefficiencies observed with current practices and on what was learned about the project needs that project managers are responsible to meet. The implementation of this process is expected to result in shorter project duration, fewer delays, and a more accurate sense of project cost and end dates due to increased communication across team roles, greater accountability and awareness of due dates, and simpler, repeatable processes that are less prone to error.

This document is not intended for public distribution. vii Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Acknowledgements

We would like to thank Worcester Polytechnic Institute and our sponsor

JPMorgan Chase for giving us the opportunity to complete this project with the Securities

Lending Technology group at JPMorgan. Particularly we would like to thank Danielle

Rumore, Brian Kenney, Vitoriana Morais, Mike Salamina, Reynaldo Generoso, Besley

Nelson, Peter Ennis, and our liaison Marisa Giliberti for their continuous support and commitment to helping us make this a successful project and a meaningful learning experience.

Furthermore, we would like to thank our advisors Professor Arthur Gerstenfeld and Professor Mike Ciaraldi for their guidance and assistance in helping us develop our project and write this report.

.

This document is not intended for public distribution. viii Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Table of Contents

Abstract...... i Authorship...... ii Executive Summary...... iii Acknowledgements ...... viii Table of Contents ...... ix Table of Figures...... xi Table of Tables ...... xii 1. Introduction...... 1 2. Background ...... 5 2.1. JPMorgan Chase & Co...... 5 2.2. JPMorgan Worldwide Securities Services...... 6 2.3. Securities Lending ...... 7 2.4. JPMorgan’s Role in Securities Lending ...... 10 2.5. The Technology Behind JPMorgan Securities Lending ...... 13 2.5.1. Custody Management Systems – COSMIC & TITAN...... 13 2.5.2. Front-end Trading - SLX ...... 14 2.5.2.1. SLX Hardware ...... 14 2.5.2.2. SLX Trade Views ...... 14 2.5.3. Legacy Trading Application - DML...... 15 2.5.4. Legacy Trading Application - SLMU...... 15 2.6. Project Delivery Framework for JPMorgan Worldwide Securities Services ...... 16 2.7. SMLU to SLX Integration Project...... 17 3. Methodology ...... 20 3.1. Critical Path Analysis ...... 20 3.2. Project Management Process Improvements...... 23 3.3. Data Mapping Exercises ...... 23 3.3.1. Parsing the DML Loan Master ...... 24 3.3.1.1. Design ...... 25 3.3.1.2. Implementation ...... 25 3.3.2. Aggregation and Summarization of Data Fields...... 26 3.4. Capacity Planning Analysis ...... 27 3.4.1. Server Specifications ...... 27 3.4.2. Used Server Resources ...... 28 3.4.3. SLMU Message Statistics...... 28 4. Results and Analysis ...... 30 4.1. Critical Path Analysis ...... 30 4.2. Project Management Process Improvements...... 37 4.2.1. Investment Council Estimates and Project Plan Creation...... 38 4.2.1.1. Current Process ...... 38 4.2.1.2. Problems ...... 39 4.2.2. Use of Microsoft Project...... 43 4.2.3. Action Item Tracking...... 44 4.3. Data Mapping Exercises ...... 45 4.3.1. Parsing the DML Loan Master ...... 45

This document is not intended for public distribution. ix Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 4.3.2. Aggregation and Summarization of Data Fields...... 48 4.4. Capacity Planning Analysis ...... 48 4.4.1. Processor Analysis...... 48 4.4.2. Memory Analysis...... 49 4.4.3. Storage Capacity ...... 51 4.4.4. SLMU Message Statistics...... 52 4.4.5. DML Projections...... 54 5. Recommendations...... 56 5.1. Project Management Process Improvements...... 56 5.1.1. The Estimation Workbook...... 58 5.1.2. Microsoft Project and the Critical Path Method ...... 64 5.1.3. Action Item Tracking...... 67 5.1.4. Weekly Status Meetings ...... 70 5.2. IT Equipment Upgrades for SLX...... 71 Works Cited...... 73 Appendix A : Securities Lending Technology Systems Relationships...... 75 Appendix B : High Level Task Document for Project Plan Creation...... 76 Appendix C : Snapshots of the STMate Tool ...... 80 Appendix D : Screenshots of the same loan in DML and SLX...... 82 Appendix E : DML Loan Master Parser Source Code ...... 84 Appendix F : DML Loan Master Parser JUnit Test Case ...... 88 Appendix G : Microsoft Access SQL Queries ...... 89 Join DML/SLX Brokers...... 89 DML Brokers without matching SLX Records ...... 89 SLX Brokers without matching DML Records ...... 89 Unique LOAN-TYPE Values ...... 89 Unique SEC-LOC Values...... 89 Unique SEC-TYPE Values...... 89 Appendix H : Invitation to Final Presentation...... 90 Appendix I : Review of SLX Server Performance...... 91

This document is not intended for public distribution. x Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Table of Figures

Figure 1: PDF Project Phases (Project Delivery Framework, 2006)...... 16 Figure 2: Current Flow of Securities Lending Information (Kenney, 2006)...... 19 Figure 3: Design of DML Loan Master Parser (Java Trademark, 2006)...... 25 Figure 4: Flow Chart for Business Analyst...... 31 Figure 5: Flow Chart for Quality Assurance Team ...... 32 Figure 6: Flow Chart for DML Team ...... 33 Figure 7: Flow Chart for SLX Team...... 34 Figure 8: Flow Chart for SLX Mainframe Team...... 35 Figure 9: STMate Snapshot Showing Unnecessary Task List...... 41 Figure 10: Class Dependency Diagram for DMLLoanMasterParser ...... 46 Figure 11: Instability Metric for DMLLoanMasterParser (green dot)...... 46 Figure 12: Efferent Couplings for DMLLoanMasterParser...... 47 Figure 13: Class and Package Statistics for DMLLoanMasterParser...... 47 Figure 14: Production Server CPU Usage ...... 49 Figure 15: Memory Usage on APP06 (GTI Hobbit, 2006) ...... 50 Figure 16: Memory Usage on APP10 (GTI Hobbit, 2006) ...... 50 Figure 17: Memory Usage on DBP01 (GTI Hobbit, 2006)...... 50 Figure 18: /local/app/a_slx on APP06 frequently uses more than 90% of quota ...... 51 Figure 19: /app/a_slxftp on the JIP NAS is near capacity ...... 52 Figure 20: SLMU Messages in SLX...... 53 Figure 21: Comparing SLMU Performance Between Oct and Nov...... 54 Figure 22: SLMU and Anticipated DML Messages over a Trading Day...... 55 Figure 23: Project Management Process...... 57 Figure 24: Estimation Workbook- Instructions ...... 59 Figure 25: Estimation Workbook- Team Member Worksheet ...... 60 Figure 26: Estimation Workbook- Summary Sheet by Phase ...... 62 Figure 27: Estimation Workbook- Summary Sheet by Role ...... 63 Figure 28: Diagram of Two Layer Project...... 65 Figure 29: Action Item Tracking Sheet Instructions...... 68 Figure 30: Action Item Tracking Sheet for Action Items...... 69 Figure 31: Action Item Tracking Sheet for Risk Items...... 69

This document is not intended for public distribution. xi Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Table of Tables

Table 1: Acceptable Non-cash Collateral (Devasthali, 2006) ...... 7 Table 2: Key Team Members and Roles...... 22 Table 3: SLX Production Server Hardware (Ghelani, 2006)...... 28 Table 4: STMate Entry Based on Complexity of Repeated Tasks ...... 42 Table 5: Code Metrics for DMLLoanMasterParser.java ...... 45

This document is not intended for public distribution. xii Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 1. Introduction

As the 17th largest corporation in the United States, JPMorgan Chase & Co. provides a wide range of banking and financial services to customers worldwide. In order to maintain its outstanding reputation, JPMorgan Chase & Co. must utilize new

technology, maximize efficiency of its internal processes, and continually improve the

services provided to its clients. Understanding the importance of these key business

strategies, JPMorgan executives decided in 2004 that the software Securities Lending

Xpress (SLX) should be the sole front-end trading platform used to initiate all securities lending trades globally. The reason for this decision is that SLX has checks built in that ensure compliance to regulations and lender rules and limits. Also SLX is built on the most up-to-date, flexible platform when compared to the other systems. This will allow future changes to be made more quickly and at a lower cost. In this vision, the other systems, International Securities Lending System / Dan McLune (DML), Securities

Lending Menu (SLMU), and GlobalOne, would only be used as loan maintenance systems and would cease to perform any loan initiation activity. The current state of the business requires that several incremental projects be completed in order to turn this vision into a reality. One such project is to integrate DML and SLX so that information from DML is passed to SLX throughout the day and overnight.

To understand the need for this project, one must understand the current process

and the implications it had on successful securities lending transactions. At the time,

DML was the system responsible for international loan maintenance activities. It tracked actions such as marks, recalls and paybacks and used this information to immediately

update borrower credit line utilization and lender limit utilization. JPMorgan holds its

This document is not intended for public distribution. 1 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. borrowers in the securities lending program to various restrictions, based on their contract, dictating how much a particular borrower can have in outstanding loans at a time. This is known as their credit limit. As loans are repaid and new ones are taken out, the utilization of a borrower’s credit limit changes. DML tracks these activities and immediately updates borrower credit line utilization so that new loans are not initiated that will exceed the borrower’s credit limit. In a similar manner, JPMorgan custody clients work with JPMorgan agents to determine lender limit guidelines. These limits and restrictions are based on how much of their securities a lender wishes to have out on loan at any time and to whom the securities are lent. As international loans are modified throughout the day, DML tracks the activity and updates lender limit utilization so that

JPMorgan does not lend out more securities than the lender feels is too risky to have out at one time.

The problem lies in the fact that throughout the day DML does not feed this information to SLX, which is the most frequently used platform for initiating new loans.

Without knowledge of DML activities, and therefore without accurate intraday credit line utilization figures in SLX, two different unacceptable situations may result. Utilization may be understated in SLX, which can result in trades being booked that should not have been because there is actually insufficient available credit line. If such a trade is instructed to DML, it is rejected since DML does not have sufficient credit line available.

On the other hand, utilization could be overstated in SLX, which results in lost opportunity since trade initiation is not permitted if there is insufficient available credit line reflected.

This document is not intended for public distribution. 2 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. In both instances, the Database team must be contacted to manually adjust the available credit line in SLX. This is time-consuming for both the Database staff and the traders and in time critical situations can still result in loss of opportunity.

Therefore, the main objectives of the DML to SLX Integration Project are:

1. SLX to have accurate international utilization at start of day

2. SLX to have accurate intraday international utilization

3. traders able to view a current picture of each open loan in SLX rather than having to toggle between SLX and DML

4. users able to define at will which value should be used to impact utilization: either the required collateral or the DML Calculated Value (Kenney, 2006)

At a very high level this will be accomplished by creating a feedback loop for intraday updates and an overnight batch file. The feedback loop will create intraday messages in response to loan activity in DML. When SLX processes one of these messages, lender/borrower utilizations will be updated and a snapshot of the loan will be created and visible to SLX users. A key aspect in the success of this project is to make sure that SLX calculates risk factors and therefore fractional limits in exactly the same way that DML does so that both systems display the same utilizations and limits.

The work done on this Major Qualifying Project (MQP) aimed to assist JPMorgan with the successful completion of this important project. Specifically, work was done to accomplish four main goals. The first two relate to the pre-design activities that needed be completed prior to the design and build phases of the project. Success of the pre- design activities would contribute to the successful implementation of the feedback loop.

This document is not intended for public distribution. 3 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. The first goal of this MQP was to assist in mapping and matching the data fields between the two systems to ensure that SLX receives all of the information it needs to properly update its database. The second goal was to analyze the anticipated effects of the feedback loop on servers and mainframes so that additional resources could be introduced and performance would not be compromised. The third goal of this MQP was to gather the information needed to conduct a critical path analysis that would examine ways to expedite the delivery of the project. From this a fourth goal was established, which was to recommend possible improvements to project management practices that could result in shorter project durations, fewer delays, and more accurate estimates of project cost and delivery dates for this and future projects.

This document is not intended for public distribution. 4 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 2. Background

2.1. JPMorgan Chase & Co.

As the 17th largest U.S. corporation, JPMorgan Chase & Co. provides banking and financial services to customers worldwide. Its annual revenues for 2005 were 79.9

billion dollars, which is an impressive 10.6 percent increase from the previous year.

According to Fortune Magazine, JPMorgan Chase trails slightly behind Citigroup and

Bank of America, who rank 8th and 12th respectively on the Fortune 500 (Fortune, 2006).

With origins dating back to the 1700’s, JPMorgan has a rich legacy of more than

200 years of banking and financial services experience. J. P. Morgan Chase & Co., is the result of a merger between JPMorgan and Chase Manhattan in 2000, which then acquired

BankOne more recently in 2004 (History, 2006). JPMorgan Chase & Co. uses two brands to provide financial services to its customers. According to its 2005 Annual

Report, the Chase brand provides a full range of banking and asset management services to small businesses, corporations, and governments. Services include but are not limited to Consumer Banking, Home Finance, Credit Cards, Auto Finance, Education Finance,

Business Credit, Equipment Leasing, and Real Estate (2006). The JPMorgan brand has eight lines of business. Those businesses are Investment Banking, Research, Asset

Management, Private Banking, Worldwide Securities Services, Treasury Services,

Private Equity, and Private Client Services (Home, 2006). Through continued excellence in providing these services, JPMorgan has remained a prominent member of the financial industry and plays a major role on the international stage. JPMorgan Chase & Co is a component of the 30 member Dow Jones Industrial Average and has over 1.2 trillion

This document is not intended for public distribution. 5 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. dollars in assets. Both of these qualities are a reflection of the company’s long term stability. (Annual Report 2005, 2006)

2.2. JPMorgan Worldwide Securities Services

With 12.9 trillion dollars of securities under custody, the Worldwide Securities

Services (WSS) division of JPMorgan is the leader in the security services industry.

Through its consultative approach and proven experience, JPMorgan partners with clients to assess and attend to their individual needs. JPMorgan Worldwide Securities Services help clients to optimize efficiency, diminish risk, and increase revenue by providing a range of services which include: American Depositary Receipts, Broker Dealer Services,

Clearance and Collateral Management, Domestic & Global Custody, Fund Services,

Global Debt Services, Investment and Liquidity Solutions, Outsourcing Solutions,

Performance Measurement, Securities Lending, Securities Settlement and Clearing, and

Specialty Trust Services. With a presence in over 90 markets, JPMorgan Worldwide

Securities Services leverages its size and capabilities to help its global client base realize their financial goals. The business’ client base includes asset managers, pension funds, mutual funds, broker dealers, financial institutions and hedge funds (WSS, 2006).

JPMorgan Worldwide Securities Services is led by Executive Vice President

Michael Clark. After joining JPMorgan in 1994, Mr. Clark assumed leadership roles in the Institutional Trust Services business in 2000 and the Investor Services business in

2005, which evolved into the Worldwide Securities Services business. Under his leadership JPMorgan Worldwide Securities Services has continuously received top ranking in industry surveys. ISF Magazine named it “#1 Overall Cash Provider” in 2006

This document is not intended for public distribution. 6 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. and JPMorgan Worldwide Securities Services received the award for “Best Investor

Services” in the Fourth Annual Waters Ranking by Waters Magazine (WSS, 2006)

2.3. Securities Lending

Securities lending, a service provided by JPMorgan Worldwide Securities

Services, is the market practice whereby securities are temporarily transferred from a lender to a borrower. For the period of the loan, the lender is protected against default through receipt of collateral from the borrower, usually in the form of cash, although non- cash collateral is sometimes accepted. At the end of the loan term the securities are returned to the lender. Table 1 below displays the acceptable forms of non-cash collateral for securities lending.

Acceptable Collateral Cash Government Securities Equities REPO Commercial Paper C/D (Yankee/Domestic/Eurodollar) Time Deposits Medium Term Notes U.S. Agencies Master Notes Bank Notes Funding Agreement/GIC Asset Backed Securities Money Market Funds Table 1: Acceptable Non-cash Collateral (Devasthali, 2006)

During the period of the loan, both the title of the lent securities and the title of

the collateral securities transfer ownership between parties. Along with the titles come the rights associated with owning the security. The borrower now has the right to sell or

This document is not intended for public distribution. 7 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. lend the security to a third party, has voting privileges, and receives the economic benefits of owning the security, such as dividends and coupons. Commonly, the contract between the lender and borrower stipulates that the borrower make equivalent payments back to the lender for money received in this manner (Devasthali, 2006). Similarly, through contractual stipulations lenders may reserve the right to recall the lent securities in order to resume voting privileges. Lending firms such a JPMorgan act as intermediaries within securities lending and facilitate the lender/borrower relationship.

More information on JPMorgan’s role in securities lending is given in Section 2.4 below.

Lenders are typically JPMorgan custody clients such as insurance companies, mutual funds, endowments, and pension funds. The term custody client refers to a situation where the client owns a particular security, but the security is in the care of

JPMorgan. In this case JPMorgan is also responsible for maintaining the securities under custody, performing activities such as tracking stock splits, dividend payouts, stock price fluctuations, and mergers and acquisitions. Generally, a good lender has the following characteristics:

• a diversified portfolio with a variety of lendable securities

• a large enough securities portfolio to make lending worthwhile because there is a small margin of return associated with securities lending

• passive trading strategies resulting in few loan recalls when compared to actively managed portfolios.

This document is not intended for public distribution. 8 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. When the lender wants to sell a security in their portfolio that happens to be on loan to a borrower, the lender must recall the loan to make the sale. Therefore, passively managed portfolios make for better lenders (Devasthali, 2006).

Lenders benefit from securities lending because they receive incremental returns on their securities that would otherwise lie dormant in their investment accounts. It provides an opportunity for lenders to make money with very little risk. Lenders make money through securities lending in a few ways. First, if the collateral received on the loan is in the form of cash, JPMorgan invests that cash in a short-term investment. The profit made on the investment is then split between JPMorgan, the lender, and the borrower. If the collateral is in a non-cash form, the borrower pays a fee which is split between JPMorgan and the lender. A lender also makes money in instances when the lender is subject to high withholding taxes on dividends or interest, that the borrow is not subject to. Here the security would be transferred to the borrower, who would receive the dividend at a lower tax rate. The benefit from the difference in the tax rates is split between the lender and the borrower. Lastly, lenders can earn money through securities lending in the case when stock issuers offer stockholders the choice of receiving a cash dividend or reinvesting it in additional securities at a discount price. If the lender cannot take advantage of the discounted price because ownership of additional shares would make their holdings larger than permitted under investment guidelines, the securities can be sourced to a borrower who shares the economic benefit of the discounted prices with the lender (Devasthali, 2006).

In addition to the financial benefits described above, borrowers make money through securities lending because it allows them to cover the short position. A lot of

This document is not intended for public distribution. 9 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. money can be made from an activity called selling short. In this case, the person borrows shares of a stock through a securities lending transaction and then immediately sells them. If the price of the stock falls the borrower re-buys the shares at the lower price and repays the original lender to settle the loan. The difference in the selling price and the re- purchase price is the profit for the borrower minus fees associated with the securities lending transaction (Morris & Siegel, 1993). Similarly, Brian Kenney, a business analyst for JPMorgan, describes that a borrower can make money in a situation where a broker sells shares of a particular stock not yet owned, to a client to meet client demand. During the float period (after the sale but before the shares are delivered to the client) the broker borrows the shares through a securities lending transaction, hopefully at a lower price, to give to the original client. Again the price difference less lending fees is the profit for the broker (personal communication, October 24, 2006).

2.4. JPMorgan’s Role in Securities Lending

JPMorgan acts as a middleman between the lender and the borrower in securities lending transactions. It facilitates the lender/borrower relationship and maintains all transaction throughout the duration of the loan. As one of the top ranked custodians for securities worldwide, JPMorgan primarily performs discretionary lending where it is the custodian for the lendable assets. JPMorgan’s achievements in this area of business are reflected in its receipt of several prestigious rankings, such as custodian of the year from

ICFA Magazine in 2005 and number one securities lender and global custodian by

Finance Asia (WSS, 2006). Globally, there are only eight major players in the securities lending business. It takes a huge amount of capital to enter into the business and the amount of money made on each loan is marginal. Therefore, to be a successful securities

This document is not intended for public distribution. 10 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. lending agent one needs numerous clients each with a large portfolio of lendable securities.

JPMorgan credits its success in the securities lending industry to its highly refined securities lending program. JPMorgan has 390 lenders and 121 borrowers currently involved in its program. The resulting, vast inventory of lendable securities provides borrowers with a wide selection of securities from which to borrow. Likewise, with such a large number of borrowers, lenders’ securities are utilized more frequently. This maximizes lenders’ financial gain from participation in the program.

As an integral part of its securities lending program, JPMorgan works closely with the lender to define lending limits and rules that govern how their securities can be lent. An example is when a lender states that no more than 100 million dollars of securities can be lent out at a time. Another example is when a lender declares that they wish to hold back 25 percent of a certain holding from lending. These lending limits can be broad and apply to an entire portfolio, or they can be very detailed, with specific rules for each type of security. Furthermore, lenders have the option of restricting to whom their securities can be lent. However, JPMorgan thoroughly screens each borrower before allowing them to participate in the lending program. All of these lending limits are then used by JPMorgan to match lenders to appropriate borrower loan requests

(Devasthali, 2006).

Another aspect of the JPMorgan program is that it protects its lenders and ensures the risk associated with securities lending is marginal in three ways. First, JPMorgan offers a broad indemnification plan that protects lenders in the case that the borrower

This document is not intended for public distribution. 11 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. defaults on the loan or goes bankrupt. Second, JPMorgan requires borrowers to provide at least 102 percent of the value of the lent security in collateral, in order to account for fluctuations in the market values of the lent securities and the collateral. For example, in lower risk loans, JPMorgan will require that the borrower furnish JPMorgan with collateral that is equal to 102 percent of the value of the securities being borrowed.

According to Brian Kenney, an example of a lower risk loan is an established domestic brokerage such as Merrill Lynch borrowing a sum of US stock and providing US dollars as collateral. For higher risk loans, a rate of 105 percent may be charged. An example of this would be a trader in Australia acting on behalf of an Israeli brokerage borrowing a

Russian stock and providing euros and a letter of credit from a bank in Spain as collateral

(personal communication, October 24, 2006). Lastly, the risk associated with the investment of the cash collateral is the only risk that JPMorgan securities lending clients are exposed to. To minimize this risk, JPMorgan utilizes conservative investment strategies and develops strict investment guideline and restrictions with each of their lending clients. This allows lenders to dictate how much risk they are exposed to.

The JPMorgan securities lending program offers unique options to its lenders and borrowers that help to distinguish it from its competitors. JPMorgan custody clients who agree to participate in the securities lending program are not charged a fee for their participation. Bank of New York and JPMorgan are the only two securities lending service providers that allow borrowers the option of using non-cash forms of collateral against their loans. JP Morgan accepts U.S. Treasuries and Letters of Credit.

Additionally, JPMorgan recently introduced a second type of lending to its program called non-custody lending, where JPMorgan is the lending agent but does not have

This document is not intended for public distribution. 12 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. custody of the lendable assets. This is attractive because it opens the door for numerous new clients to participate in securities lending with JPMorgan.

2.5. The Technology Behind JPMorgan Securities Lending

JPMorgan relies heavily on versatile and reliable technology in order to be one of the major players in the securities lending business. To successfully facilitate securities lending transactions, JPMorgan uses a variety of programs to manage its inventory of securities and to initiate and maintain loans. These systems allow traders to obtain real time data regarding all transactions and holdings in the securities lending service. This project, along with other initiatives within JPMorgan, aims to integrate these systems in order to optimize the uses of each and streamline securities lending processes. A better understanding of these systems is vital in appreciating the relationships between them and the processes currently being used by the securities lending group. A diagram detailing the relationships between systems can be found in Appendix A.

2.5.1. Custody Management Systems – COSMIC & TITAN

JPMorgan has two systems in production that manage its inventory of lendable securities. The inventory of international securities resides in a system called COSMIC whereas the domestic securities in custody reside in TITAN. These repositories are responsible for managing who owns a particular security and if it is out on loan. The systems do not however know or care about the details of a loan. Access to COSMIC and TITAN are indirect and are handled through front-end trading platforms.

(Devasthali, 2006)

This document is not intended for public distribution. 13 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 2.5.2. Front-end Trading - SLX

The front-end trading platform used by the majority of traders at JPMorgan is

SLX. Currently, ninety-three percent of all US loans originated in SLX. SLX is one of the newest applications that is currently in production and offers traders easier access to data when compared to the legacy systems. SLX ensures compliance with auditing and federal trade guidelines. It also is designed to ensure that all lenders have an equal opportunity to lend their securities, which is required by government regulations. SLX is based on an Oracle/UNIX platform and was written in a proprietary language called

Tenfold.

2.5.2.1. SLX Hardware

SLX production systems are hosted on multiprocessor UNIX based IBM AIX nodes. The Global IT Services Group within JPMorgan offers tools and utilities that monitor CPU, memory, disk utilization, and network traffic for each node. Real time data is taken in five minute snapshots and is posted at midnight. Raw data is also archived for future access. (GTI Hobbit, 2006)

2.5.2.2. SLX Trade Views

On the omni level (also known as the trade level) the borrower sees the entire loan as one large package, regardless of the number of lenders associated with the loan. On the component level (also known as the position or sequence level) traders can look at a loan in more detail. If multiple lenders have securities lent in a particular loan the sequence level view gives details about each lender and the value of its securities associated with that loan. According to Brian Kenney, business analyst at JPMorgan

This document is not intended for public distribution. 14 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Worldwide Securities Services, the reason that the second level exists is because it allows for better accounting and tracking of each loan (personal communication, October 25,

2006).

2.5.3. Legacy Trading Application - DML

At the present time, there is a legacy application that coexists with SLX and is situated between SLX and COSMIC. This application is called DML and is used primarily by international traders that are not based in the US or in the UK. Although

DML can perform loan activity, upon completion of this project DML will only be responsible for loan maintenance. DML exists on a mainframe and can be accessed through mainframe session thin clients such as RUMBA. DML is a Sunguard application written in COBOL. DML uses DB2 and VSAM as a database platform. (Devasthali,

2006)

2.5.4. Legacy Trading Application - SLMU

SLMU is the counterpart to DML and is used for US-domestic securities lending loan maintenance. SLMU has recently been phased out as a trading platform in favor of

SLX. A feedback loop was created between SLX and SLMU to feed SLX with real time data related to updates in SLMU. SLMU still exists as an interface to TITAN but is no longer used to process trades. Like DML, SLMU is a COBOL/DB2 platform application.

(Devasthali, 2006)

This document is not intended for public distribution. 15 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 2.6. Project Delivery Framework for JPMorgan Worldwide Securities Services

The Project Delivery Framework (PDF) is a common and integrated project life cycle framework using the best practices from JPMorgan Chase. The framework consists of three tiers of 8 phases each, with required and recommended deliverables and processes per phase. The tier classification of a project, as Tier 1, Tier 2 or Tier 3, is based on project cost, risk and complexity. It is expected that all projects sponsored and led by teams within JPMorgan Worldwide Securities Services will adhere to the

PDF. The figure below shows the breakdown of project phases as required for all

JPMorgan Worldwide Securities Services projects.

Figure 1: PDF Project Phases (Project Delivery Framework, 2006)

As part of the PDF process the project manager and delivery manager needs to

meet with the JPMorgan Chase Investment Council (IC) to obtain approval and funding

three times for each project. One section of these presentations to the Investment Council

is an estimation of the cost and completion date of the project. At the first meeting with

the IC, the estimate, known as the E1 Estimate, needs to have a 50% confidence interval.

At the second meeting, the E2 Estimate is due and should have a confidence interval of

30%. At the final IC meeting, the E3 Estimates are presented which require a 10%

confidence interval. The accuracy of each subsequent estimate is tighter because it is

This document is not intended for public distribution. 16 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. gathered at a later date when more information about the tasks required to complete the project and resource constraints have been identified (Project Delivery Framework,

2006).

In the Securities Lending Technology Division a tool called STMate is used to gather each estimate. Each key team member on the project enters anticipated tasks and durations of each task into the worksheet. All entries are compiled in a summary sheet and a cost and project duration estimate is then generated. Sample snapshots of the tool with information from the E1 Estimate of this project can be viewed in Appendix B.

2.7. SMLU to SLX Integration Project

Prior to the DML to SLX Integration Project, JPMorgan undertook a very similar project to integrate SMLU and SLX. In order for SLX to become the solitary front-end trading platform, it must reflect accurate borrower credit line utilization. SLMU is used for domestic loan maintenance activities such as re-pricing and returns. The problem arises from the fact that these activities impact credit line utilization, but since SLMU did not relay information to SLX throughout the day, these impacts were not reflected in

SLX utilization figures. Therefore, the objectives of this project became:

• The reflection of accurate borrower credit line utilization in SLX related to loans maintained on SLMU. Utilization is to be reflected accurately in SLX both intraday and end of day and is therefore to be based on SLX loan initiation and cancellation as well as on information provided intraday and end of day by the SLMU to SLX feedback loop.

• The continued reflection of approximated borrower credit line utilization related to loans maintained on DML (all international markets except the UK) based on

This document is not intended for public distribution. 17 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. SLX loan initiation and cancellation as well as on information provided by the end of day DML MIS file

In order to accomplish these objectives, the project team created a feedback loop between SLX and SLMU. The result is that each time a loan is updated in SLMU a message is sent to SLX with this information. Borrower credit utilization and lender limit utilization are then updated in SLX within two minutes. This ensures that only loans that do not exceed lender/borrower limits are initiated and that clients’ participation in loan activity is maximized. Upon completion of the SLX to SLMU Integration

Project, the flow of information between the key systems in securities lending could be represented in Figure 2 below. Since the SLX/DML integration project occurred after this project, the diagram also represents the status quo of information flow prior to the completion of the DML to SLX Integration Project.

This document is not intended for public distribution. 18 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. MIS File (used for nominees other than DTC, FED Nightly and Crest)

SLX SLMU DML

INT’L US BORROWER BORROWER BORROWER BORROWER CREDIT LINE CREDIT LINE US FEEDBACK LOOP CREDIT LINE CREDIT LINE UTILIZATION UTILIZATION Intraday and Nightly UTILIZATION UTILIZATION

Intraday: Intraday: Intraday: Intraday:

New SLX Loans (+) New SLX Loans (+) New Loans (+) New Loans (+) SLX Cancels(-) SLX Cancels(-) Marks (+/-) Marks (+/-) New SLMU / Titan Returns (-) Returns (-) Loans (+) COAC Adj (+/-) COAC Adj (+/-) SLMU Returns (-) Cash Adj (+/-) Cash Adj (+/-) SLMU / Titan COAC Cancels (-) Adj (+/-)

Nightly: Nightly:

Recalc based on MIS Recalc based on SOD files Snapshots from SLMU

Input into SLX Intraday: Input into SLMU or DML Intraday: New Loans New SLX Loans Marks SLX Cancels Returns COAC Adj Cancels

Figure 2: Current Flow of Securities Lending Information (Kenney, 2006)

This document is not intended for public distribution. 19 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 3. Methodology

3.1. Critical Path Analysis

As mentioned previously, this project was very important to the JPMorgan

business because it laid the foundation for several new products and services to be

developed that would bring significant amounts of new revenue to the company. Without

this project, these new products and services could not be developed. With this in mind, there was a serious push from management to expedite the project. The critical path method was determined to be the superior tool for analyzing ways to expedite the project.

A critical path analysis, based on a comprehensive project plan, allows managers to

prioritize activities for the effective management of project completion. Often projects

don’t finish on time because of multi-tasking and lack of prioritization. The critical path

method works to identify which tasks are critical to the on-time completion of a project.

Priority and special attention are subsequently given to these tasks because if the

completion of a critical task is delayed then the whole project is delayed. If feasible,

additional resources should be allocated to critical tasks to expedite their completion

(Internet Center for Management and Business Administration, Inc.).

A number of steps needed to be completed to gather the information that was required to utilize the critical path method in the most effective way. The basis for this analysis was an accurate and thorough project plan. Three key pieces of information are required to begin constructing a project plan. The first is a listing of all tasks and activities required to complete the project. The second is estimates regarding the time needed to complete each task. The third is a list of predecessors for each task. A

This document is not intended for public distribution. 20 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. preliminary project plan had been constructed prior to the start of this project. However, as the project changed and developed, the original project plan needed to be modified substantially.

A big challenge in obtaining these three types of information was that the project was still far from completion, barely entering the design phase. At this point in the project there were still many more questions than answers about how the project goals would be accomplished. Each member in the project team, each with their own area of expertise, had unique thoughts as to which tasks were necessary to realize the objectives of the project. This led to disconnects in the overall picture of how the project would be completed. These disconnects prohibited the creation of a cohesive project plan.

To obtain the information needed to create a cohesive project plan, the first step was to create a document with the high level tasks and key deliverables of the project, using information from the original project plan. The document also contained a list of the four key objectives of the project and a note reminding resources that all tasks should add value to fulfilling either a direct objective of the project or one of the documents required by the Project Delivery Framework (PDF) process. This document was then distributed to the project team for review. A meeting was held to discuss the document where the agenda was to 1) brainstorm if all high level tasks were accounted for and 2) collectively come to an agreement on a single strategy for how the project would be completed. The document was then updated with information collected at the meeting.

The final version of the document can be seen in Appendix B.

This document is not intended for public distribution. 21 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. At that point all team members had the same vision of the strategy that would be used to complete the project. Subsequently, the data needed to update the project plan could be gathered. The major team members were determined and can be seen in the table below.

Team Member Specialty Area

Brian Kenney Business Analyst

Mike Salamina SLX Development

Peter Ennis SLX Mainframe Development

Rey Generoso DML Development

Besley Nelson Quality Assurance

Table 2: Key Team Members and Roles

Individual interviews were set up with each key team member to gather information

needed for creating the project plan and performing the critical path method. The agenda

for each interview contained the following points:

1. Document all activities that a particular individual needs to complete for the extent of the project.

2. Estimate the number of days each activity will take to complete if the owner is exclusively devoted to the tasks completion.

3. Identify the predecessors of each activity and the earliest possible start of each.

4. Receive suggestions for improving the STMate Tool used for making estimates regarding project cost and duration.

The High Level Task Document and each person’s STMate spreadsheet from the E1 estimates were used to guide the interviews.

This document is not intended for public distribution. 22 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 3.2. Project Management Process Improvements

Another objective of this project was to recommend ways in which project management processes could be improved. The improvements were aimed at making the processes more efficient, streamlined, simple and/or error-proof. To complete this objective the current process for managing projects needed to be understood and evaluated. This was partially accomplished by going through the experience of executing many steps of the current system to develop a project plan and gather information for estimates to present to the Investment Council. Throughout the execution of these activities, a list of notes about inefficiencies and possible sources of error inherent in the process was maintained. Information about current processes and ways to improve them was also gathered in two other key ways. Informal discussions were held with the project manager and the project team regarding their struggles and experiences with current practices. Lastly, research was conducted using the Microsoft Project help function to investigate proper usage of the tool and additional capabilities that were not previously being leveraged when managing JPMorgan projects.

3.3. Data Mapping Exercises

The goal of the data mapping exercise was to examine the fields that DML has defined in a loan snapshot and compare that to what data fields SLX needs to receive to properly update borrower credit and lender limit utilizations. One of the challenges was that the fields in DML do not directly map to the fields in SLX. Therefore, an in-depth analysis needed to be conducted to examine how the data fields from DML should be manipulated so that they are in a useable format for SLX. Once this was accomplished, to

This document is not intended for public distribution. 23 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. verify that the manipulations are correct, the team will create a proof of concept. This will entail creating a mock intraday message using the DML data fields as inputs that outputs the same information in the SLX format.

To the surprise of the development team, the data was too complex to create a one-to-one mapping such as saying the broker field in DML matched the broker field in

SLX. According to Danielle Rumore, calculations proved to be extremely difficult to trace because of the extensive amount of COBOL logic that was built around each field in DML over the past decade. (Personal Communication, November 15, 2006). Danielle also explained that up-to-date documentation does not exist for DML, so very few people know exactly what each field means. As part of the data mapping exercises, sample loans were observed from the production environment. Screenshots of one such loan can be found in Appendix D.

3.3.1. Parsing the DML Loan Master

The first encountered problem was taking the raw loan data from DML, known as

a Loan Master File, and putting it into a database that allowed it to be analyzed. The loan

master file contains over 220,000 records stored in a comma delimited file called a CSV

(Comma Separated Value) file. Due to software limitations in Microsoft Excel XP, only

the first 65,536 records could be displayed because row numbers are stored in a 16 bit

integer. Importing data into Microsoft Access using the Microsoft .txt Jet Engine was

also problematic because opening the text file for parsing caused a virtual memory

overflow on the workstation.

This document is not intended for public distribution. 24 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 3.3.1.1. Design

To get around the file size limitation, a program was developed that would split the CSV into multiple files that had less than 65,000 records. A parser was designed that would read in the CSV file one line at a time and immediately output it to a target CSV file. A counter would be in place to keep track of how many records were written to the target file. Once the user-defined limit was reached, the parser would send the data out to another file. A high level design is shown in Figure 3.

Figure 3: Design of DML Loan Master Parser (Java Trademark, 2006)

3.3.1.2. Implementation

To implement this parser, the Java programming language was used.

Development occurred in an Eclipse 3.2 development environment using the Sun Java

JS2E SDK v1.5. A third party CSV parser package called com.csvreader.CsvReader was

used in the implementation because of its high benchmark ratings. One of the benefits of using this package is that none of the data is buffered in memory, so it can process a CSV file of any size.

This document is not intended for public distribution. 25 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. The software uses nested for loops to scan the records. The outer loop reads one line at a time. The inner loop reads each column and writes it to the target output text file. If however, the number of columns in a row did not equal 125 (a hard coded constant equal to the expected number of columns a DML loan), the parser would output that entire row to a different file to be analyzed later.

Command line arguments were implemented to allow the user to specify the input file name, the output file name, and the maximum number of records that each output file is allowed to contain. The only known bugs associated with the program are the following:

• Using an integer value greater than Integer.MAX_VALUE for integers will cause problems when counting the number however that value in Java 1.5 is 2^31 – 1 which is larger than the number of integers allowed in Excel anyways.

• Using long filenames or name with spaces causes problems with the file name string. We suggest using the MS-DOS 8.3 file naming convention when referencing file names in the command line parameters.

The compiled .class file was then placed on a shared network drive so that any user that has the 5.0 release of the Java Runtime Environment (JRE) could parse a loan master. The source code for the program can be found in Appendix I.

3.3.2. Aggregation and Summarization of Data Fields

Once the data was imported into Microsoft Access, there was a need to run various queries on the data in order to find patterns or unique values. According to requirements and needs, the following queries were developed:

This document is not intended for public distribution. 26 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. • Queries on the SLX/DML broker tables

• Join DML/SLX Brokers • DML Brokers without matching SLX Records • SLX Brokers without matching DML Records

• Queries on the DML Loan Master

• Unique LOAN-TYPE values • Unique SEC-LOC values • Unique SEC-TYPE values • Filter out closed loans

These queries were written in Microsoft Access SQL Query format. The queries can been found in Appendix J. These queries along with the Microsoft Access database were then posted to a shared folder so that any person that wanted to run these queries could do so.

3.4. Capacity Planning Analysis

The goal of the capacity planning analysis was to carry out a comprehensive analysis on the SLX servers and to gather enough information to make infrastructure decisions moving forward. The primary deliverable was a document that gives the reader an understanding of the resources required to run SLX in a production environment and to analyze the impact that the DML feedback loop will have on the systems. Information was gathered from many sources and compiled together to form a complete picture of the resources that SLX depends on. The complete document can be found Appendix I.

3.4.1. Server Specifications

The first step in the capacity planning analysis was trying to understand what the

current servers had for hardware. Through documentation and utilities provided by

This document is not intended for public distribution. 27 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. JPMorgan’s Global Technology Infrastructure (GTI Hobbit, 2006). The primary focus is on the production system. General information about production server hardware can be found in Table 3. All SLX servers are IBM AIX servers with PowerPC processors

running a UNIX environment. The servers can also allocate a percentage of the total

number of physical processors to a specific application (in this case SLX).

APP06 APP10 DBP01 Total Number of Processors 4 2 8 Processors assigned to SLX 4 2 3.5 Processor Speed 1.7 GHz 1.7 GHz 1.7 GHz Processor Type 64 bit 64 bit 64 bit L2 Cache Size 1.5 MB 1.5 MB 1.5 MB Physical Memory 7.75 GB 7.75 GB 62 GB /dev/hd6 : 4 GB /dev/hd6 : 4 GB /dev/hd6 : 4 GB Known Paging Files /dev/paging00 : 6GB SLX Disk Drives 9 11 6 Table 3: SLX Production Server Hardware (Ghelani, 2006)

3.4.2. Used Server Resources

The next step was to look at the current performance of the servers. There were

two areas of particular interest, current processor usage and allocated memory. Both of

these pieces of information could be obtained from a JPMorgan website that provided

real-time server statistics. (GTI Hobbit, 2006) For the analysis, data was gathered for the

same week (October 15th – October 21st) to maintain consistency. The three areas of

focus are processor usage, memory allocation, and storage quotas.

3.4.3. SLMU Message Statistics

The final step was to merge in statistics from SLMU and projected DML activity

onto the plot of production environment. SLMU data was gathered from the SLX

This document is not intended for public distribution. 28 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. production environment query on the SLMU feedback loop. DML data was exported by the DML system administrators and emailed to requestors.

This document is not intended for public distribution. 29 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 4. Results and Analysis

4.1. Critical Path Analysis

The majority of the information gathered for the critical path analysis was

obtained during the interviews with key members of the project team. Each interview

focused on identifying the tasks that a particular resource would need to complete as part

of the project and what the task dependencies are. Some team members were able to provide task durations during these meetings. Timing was such that E2 Estimates, which are the second round estimates that should have a 30 percent confidence interval, were due a short time after these interviews. Therefore, these meetings served a dual purpose: to aid in the creation of a thorough project plan and to aid in filling out the STMate tool for the E2 Estimates. Task durations for those team members that did not provide them in the interviews were gathered from the E2 Estimate spreadsheets once they were completed. From these meetings, flow charts were created with the tasks for each team member to visually highlight predecessors and task dependencies. The flow charts for each resource can be viewed in the following figures.

This document is not intended for public distribution. 30 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 4: Flow Chart for Business Analyst

This document is not intended for public distribution. 31 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 5: Flow Chart for Quality Assurance Team

This document is not intended for public distribution. 32 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Define DML fields analysis for SLX DML Unit Test Backout Recovery DML Calculations E2 Estimates resources Plan/Strategy Plan 15 Days 3 Days (Data Mapping)

Brainstorm High Level Design w/ DML Team 2 Days DML - Rey Generoso

Detailed Design DD Logical Unit of DD Snapshot (DD) Work SOD File

DD Process Logic Design for Loan Master

Start Document 2 Documents 1. Traceability DD Prototyping Matrix 2. ECM 3. Backout DD Document Recovery Plan Update 4. Turnover Documents Document 1,3, 5 5. Implementation & Rollout Document DD Walkthrough

SOD- Intraday (I/O Module)- Intraday (I/O Module)- Programming & Create Module for Create Module for Logic of Copybook Snapshot Unit Work (Build, Test, Peer (Build, Test, Peer Code (Build, Test, Peer Code Code Review) Review) Review)

SOD- JCL Proc (Build, Test, Peer Intraday- Modify Existing Intraday- Modify Existing Code Review) Modules for Mark Process Modules for DML onlines for Japanese Assets (Build, Test, Peer Code (Build, Test, Peer Code Review) Review) Finalize All Documents SOD- CA7 Setup Scheduling (Build, Test, Peer Code Review) SIT Tests in DML

Revise Documents Support QA and 1,2,3,5 UAT Testing

Start Document 4

Figure 6: Flow Chart for DML Team

This document is not intended for public distribution. 33 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 7: Flow Chart for SLX Team

This document is not intended for public distribution. 34 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Data Mapping Completion (Information Brainstorm High SLX Mainframe- regarding outputs from Level Design Peter Ennis DML and necessary inputs to SLX)

Detailed Design Detailed Design Detailed Design SOD Batch Intraday Batch Intraday Online Program Specs Program Specs Program Specs

Enter Build Phase

Write Unit Test Write SIT Strategy Strategy

Build Jobs Build Copybook Infrastructure Build Copybook Common Logic CICS, MQ, VSAM File Layouts Module or DB2

Enter Testing

Build Intraday Online Program

Execute Unit Tests

Execute SIT

Build End-of- intraday Batch Program Build SOD Program

Support QA

Figure 8: Flow Chart for SLX Mainframe Team

This document is not intended for public distribution. 35 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. All of the information gathered from the interviews, the flow charts, and the E2

Estimates was then entered into Microsoft Project to create an initial project plan. The next step was to identify all of the resources that would be working on the project. All team members’ vacation schedules and bank holidays were then integrated into the project plan to accurately portray the project schedule. The next step that we were able to complete was to work with the delivery managers in charge of the team members to investigate how much of each resource’s time could be allocated to working on this particular project. The delivery manager is the supervisor to most team members and is responsible for the thorough, timely delivery of the project. The project plan was again updated with this information so that it would accurately represent task durations.

That was all of the information that we were able to gather during our time at

JPMorgan. At that point the project plan was handed back to the project manager to complete the analysis. The final step in creating an accurate and thorough project plan to use in the critical path analysis is to assign resources to each project task. After that the project manager can use Microsoft Project’s Detailed Gantt Chart view to see the critical path. The last step is to analyze the critical path to determine ways to accelerate the project. The following options should be considered and analyzed:

1. Allocating extra resources to critical tasks

2. Revising task dependencies to allow for more scheduling flexibility

3. Breaking the new functionality developed by this project into individual features that could be delivered to production separately and at different times

This document is not intended for public distribution. 36 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 4.2. Project Management Process Improvements

This section aims to document and evaluate current project management processes in place in the Securities Lending Technology division of JPMorgan. The three areas of project management practices that were evaluated in this project are: 1) the process for gathering information for Investment Council estimates and project plan creation, 2) the use of Microsoft Project plans to help manage the project and 3) the tracking of action items identified in team meetings. In addition to a discussion surrounding each of these three areas, there is also a section about general observations regarding the culture of project teams and how projects are executed. These observations are important in understanding how to improve the way projects are managed to result in faster project delivery without compromising quality.

Overall, project management within JPMorgan Securities Lending Technology does not have a defined set of best practices. The process used to manage projects differs between project managers and even between projects by the same project manager. This results in confusion, poor practices being used, and time wasted by team members learning a new system for each project. For the most part deadlines are very fluid and team members are not held accountable when deadlines are missed. Another observation is that there is a poor sense of team within the group. Many people seem to have tunnel vision around their specific role and seldom offer input or suggestions regarding aspects of the project that they are not directly responsible for. This poor communication across team roles contributes to the lack of a clear, big picture throughout the team of how the project will be executed and the objectives satisfied.

This document is not intended for public distribution. 37 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 4.2.1. Investment Council Estimates and Project Plan Creation

4.2.1.1. Current Process

Two important functions performed by JPMorgan Project Managers are to provide project cost and duration estimates to the Investment Council for project funding and to create a project plan. As stated in the methodology section of this paper, three main types of information need to be collected from each project team member in order to create a project plan. The first is a listing of all tasks and activities required to complete the project. The second is estimates regarding the duration of each task.

Duration specifically means the amount of actual working time necessary to complete the task. It does not mean the elapsed time between the start of the task and the completion of the task. The third is a list of predecessors for each task. The first two types of information are also necessary for providing estimates to the Investment Council.

The current process for gathering this information is to first send out the STMate tool template to each team member. The team members fill in the STMate tool worksheet for their job type (e.g. business analyst) with tasks and estimated durations for each task. They also compile a list of assumptions they used when making the estimates.

Each person sends his/her STMate worksheet and list of assumptions to the project manager. The project manager then takes the total estimated man hours and associated cost from each worksheet and creates a separate overall project summary in Excel. The

STMate tool does have the capacity to compile a summary of all of the estimates if all worksheets are in the same workbook, however, this function does not get used. Next the team meets to discuss the assumptions each resource used when making their estimates.

This document is not intended for public distribution. 38 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. The purpose is to compile an overall assumptions list to be used in the presentation to the

Investment Council.

At this point, all of the necessary information for Investment Council funding request presentations has been obtained. However, a project plan cannot be created without a thorough understanding of task predecessors. The project manager gathers this information in a piecemeal fashion in a few different ways. For tasks that are assigned to a single resource, the project manager meets with that person to discuss the order in which the tasks need to be completed. For tasks that require a group effort, the project manager both meets with the delivery manager to discuss possible task dependencies and also attends brainstorming meetings where the team develops a strategy for completing a certain group of tasks. In all instances, the project manager spends large amounts of time gathering task predecessors. This is a very inefficient process. Due to the piece meal fashion in which task predecessors are identified and relayed to the project manager to enter into the project plan, several weeks can pass before a thorough, accurate project plan can be analyzed and used to drive the project.

4.2.1.2. Problems

While the current system does accomplish the task at hand, there are many inefficiencies and potential sources for error associated with it. The majority of the problems lie in the STMate tool. As a brief overview, as stated in the Background

Section of this paper, the tool is an Excel workbook template that has worksheets within it for each type of team member. The types of team members are Project Manager,

Business Analyst, Quality Assurance, Technical Manager, and Hardward Systems. Each

This document is not intended for public distribution. 39 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. team member fills in the worksheet designed for their role with all anticipated tasks and effort estimates for each. The tool then calculates the total man-days each resource requires to complete their share of the project and the associated cost. This information is displayed in a summary sheet. Some example snapshots of the tool can be seen in

Appendix C. The main problem is that the tool is unnecessarily complex with no clear

set of use instructions. There is a very comprehensive list of all possible project tasks that could potentially be completed in each resource spreadsheet. Many team members feel that the list is too cumbersome to read through because projects in this division vary so greatly that the majority of the tasks on the list do not pertain to a given project. The goal is to remind people of any tasks they may not have remembered without the list.

People should just leave the estimated time for non-applicable tasks at zero. However, there is not a set of instructions that explains this explicitly, which leads confusion. The

figure below illustrates this point. It is a snapshot of a technical manager’s E1 estimate.

The majority of the tasks do not apply to the DML to SLX Integration Project and

therefore, have a duration of zero days entered into the columns in blue.

This document is not intended for public distribution. 40 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 9: STMate Snapshot Showing Unnecessary Task List

Another way in which the tool is unnecessarily complex is that it is designed around the functionality that allows a person to enter estimates based on the complexity of a task. This function is applicable if a task involves repeating an activity several times, where the level of difficulty is not the same for each iteration. A very basic example of how this functionality works is that a chef is making an elaborate dinner and wants to make a project schedule. As part of the dinner he will make six pies: 1 pumpkin, 3 apple, and 2 strawberry rhubarb. He estimates that the pumpkin pie is easy to make and will take 1.5 hours, the strawberry rhubarb pies are medium difficulty and will take 2 hours each, and the apple pies are of a high difficulty and will take 4 hours each to make. The tool has three sets of columns to calculate an overall estimate for making pies. In the

This document is not intended for public distribution. 41 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. activity units by complexity columns he would enter the number of pies for each complexity level. In the effort static data columns he would enter the amount of time it takes to make a pie for each complexity level. The effort calculation columns are the product of the two other sets of columns. The table below shows a simplistic version of how the chef would enter this data into the STMate tool. The total estimated time to make pies is 17.5 hours.

Activity Units By Effort Calculation Effort Static Data Complexity (Man Hours) Task High Medium Low High Medium Low Total High Medium Low Make 3 2 1 12 4 1.5 17.5 4 2 1.5 Pies Table 4: STMate Entry Based on Complexity of Repeated Tasks

The problem with this functionality is that in Securities Lending Technology

projects most tasks are unique and not repeated and therefore, do not require the use of

this functionality. However, the template for the tool is designed around this capability,

leaving no place to enter simple estimates for a one time task. Many times the project

manager is forced to meet with team members to discuss how they entered estimates into

the tool to make sure the way the project manager is interpreting the estimates is the way

the team member intended. The complexity of this aspect of the tool is not justifiable

when considering the rare instances in which it is useful, and the confusion and extra time

it introduces to the estimation process.

There are some other smaller issues with the tool and the way it is used. Many

cells have built-in formulas that should not be changed, but these cells are not locked. A

user can easily erase or modify these formulas without realizing it. Additionally, the

STMate tool and the way it is currently used, does nothing to ensure that the tasks one

This document is not intended for public distribution. 42 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. team member believes are necessary to complete the project align with the tasks another team member envisions. As mentioned previously, due to the wide range of peoples’ expertise, it is a challenge with many of these projects to have all project resources collectively come to an agreement on a single strategy for how the project should be completed. The process for gathering the data needed for estimates and project plan creation should be designed to encourage all team members to have a common ideology and subsequent high-level tasks in mind when they provide their list of tasks, estimated durations, and predecessors. Lastly, the STMate tool is not designed to capture task dependencies and predecessors. Likewise, there is no formal process for gathering this information and passing it to the project manager, who can then incorporate it into the project plan. The project managers receive this information informally little by little through attending project meetings and holding discussions with team members about project deliverables as they become due. However, a thorough and timely understanding of task dependencies is vital to the creation of a useful project plan.

4.2.2. Use of Microsoft Project

Using all of the data discussed in the previous section, the project manager creates a project plan using Microsoft Project. When new tasks arise through team meetings and progress on the project, team members notify the project manager who updates the project plan. The main use of the project plan for JPMorgan Securities Lending Project

Managers is to determine a completion date for the overall project and to keep track of all deliverables and key tasks associated with the project. However, the tool can be leveraged much more to effectively manage the project and schedule tasks to ensure a speedy delivery. In the current system, rather than using the project plan to dictate when

This document is not intended for public distribution. 43 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. team members should be working on specific tasks to give priority to critical tasks and ensure proper allocation of resources, team members notify the project manager of when they will be able to complete tasks and then the project plan is updated based on this information. In this case it is used more as a tracking tool than a planning tool. With this method people are not held as strictly to due dates and the project can easily be delayed.

Also in the current system the project plan is mainly used by the project manager. The other team members do not follow the project plan and its schedule to dictate when they work on which tasks. Therefore, team members are not aware of which activities are critical tasks and as such do not know how to best prioritize the tasks assigned to them.

4.2.3. Action Item Tracking

Because Securities Lending projects often require a project team with widely varied roles and backgrounds, many brainstorming sessions, working meetings, and update meetings are held to discuss the tasks required to complete the project. New action items are often identified at these meetings. These action items are too granular to put into the project plan but are essential to the successful completion of key deliverables and the overall project. Since they are not captured in the project plan, another system is required to track action items. The current system is that after most meetings, the meeting coordinator sends out a recap of the meeting via email. Part of this is a list of action items and the person responsible for the completion of each. There is no consistent method in place for checking back on the status of these action items. Also there is no one place that houses a compiled list of action items. The project team members are responsible for searching through their email to find action items. Since there are so many meetings each with a separate meeting recap email and also so many

This document is not intended for public distribution. 44 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. other emails received daily, it can become very time consuming for a team member to keep track of all of the action items assigned to them. Likewise it is even more difficult for the project manager to track the progress on all of the action items associated with a project. This system is very inefficient and can cause team members to lose sight of important tasks that are vital to a timely and successful project completion.

4.3. Data Mapping Exercises

4.3.1. Parsing the DML Loan Master

The loan master parser successfully parsed the comma separated file and did so without causing strain on the workstation. Running time for this program was under 10 minutes. The program was also created with good object-oriented software design practices in mind. Table 5 shows important code metrics for this program. These metrics

were generated by the Eclipse Metrics plug-in.

Cyclomatic Lines Num of Number of Number of Method Complexity of Code Levels Parameters Statements

8 126 5 1 58 main(java.lang.String[]) setOutputFileName(java 2 19 2 2 6 .lang.String, int) setErrorFileName(java.l 2 18 2 1 6 ang.String) Table 5: Code Metrics for DMLLoanMasterParser.java

Because this program is contained all within one class, there are very few object

dependencies. An analysis was performed on the class and package level with the use of

the CAP (Code Analysis Plugin) plugin for Eclipse. The results for the package are

shown in Figure 10, Figure 11, Figure 12, & Figure 13.

This document is not intended for public distribution. 45 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 10: Class Dependency Diagram for DMLLoanMasterParser

Figure 11: Instability Metric for DMLLoanMasterParser (green dot)

This document is not intended for public distribution. 46 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 12: Efferent Couplings for DMLLoanMasterParser

Figure 13: Class and Package Statistics for DMLLoanMasterParser

This document is not intended for public distribution. 47 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 4.3.2. Aggregation and Summarization of Data Fields

Due to inefficiencies of Microsoft Access, the queries written in Appendix G took

a very long time to run. It was decided to run the queries once and then export the results to a Microsoft Excel workbook, as the data in the referenced tables did not change after the initial import.

4.4. Capacity Planning Analysis

The findings of the capacity planning analysis on the servers can be broken down into three categories: processor, memory, and storage. Each category looks at current bottlenecks and anticipated bottlenecks. The complete analysis can be found in the

Capacity Planning document at Appendix I.

4.4.1. Processor Analysis

Data was extracted from the servers showing how busy the processors were at a

given point in time during the day. Data points were recorded every five minutes. To paint a picture of what a typical day looked like, the median value was taken (out of the seven days) for each of the five minute intervals. The median was used instead of the mean because weekend downtime would have skewed the plot for certain time intervals.

Each of the three production servers was then plotted over a 24 hour horizontal axis.

This plot can be found in Figure 14.

This document is not intended for public distribution. 48 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Median production server CPU usage (10/15/06 - 10/21/06)

1

% APP06 CPU 0.8 Utilization

% DBP01 CPU Utilization

0.6 % APP10 CPU Utilization

6 per. Mov. Avg. (% APP06 CPU 0.4 Utilization)

6 per. Mov. Avg. (%

CPU usage (as a ratio usage of100) (as CPU DBP01 CPU Utilization)

0.2 6 per. Mov. Avg. (% APP10 CPU Utilization)

0 0 1 2:00 3:00 4:00 5:00 6: 7: 8: 9:00 10 11 1 13:00 14:00 15:00 1 1 1 19 20 21 22 23:00 0:00 :0 :0 0 0 0 2:00 6:00 7:00 8:00 0 0 0 0 0 :0 :0 :0 :0 :0 :0 0 0 0 0 0 0 Time of day (Eastern Time)

Figure 14: Production Server CPU Usage

As illustrated in Figure 14, CPU usage has two areas of peaks. The first peak

starts at 9:00 PM and runs until 3:00 AM. The second peak starts at 3:00 PM and ends

around 7:00 PM. These peaks can be attributed to SLX batch processes that are running during those periods. Surprisingly the server load is under 50% for most of the trading

day. As a result, processing time for messages is under the 2 minute service level

agreement.

4.4.2. Memory Analysis

Data was extracted from the servers showing how much physical and virtual

memory was used during the day. In the case of memory, a bigger time interval was

This document is not intended for public distribution. 49 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. observed because memory changes less rapidly. Memory allocations for servers APP06,

APP10, and DBP01 can be seen in Figure 15, Figure 16, & Figure 17 respectively.

Figure 15: Memory Usage on APP06 (GTI Hobbit, 2006)

Figure 16: Memory Usage on APP10 (GTI Hobbit, 2006)

Figure 17: Memory Usage on DBP01 (GTI Hobbit, 2006)

This document is not intended for public distribution. 50 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Figure 15, Figure 16, & Figure 17 show memory usage on each of the servers over a 48 day span. The plot in blue represents physical memory usage. The plot in red represents swap space via hard disk paging files. One interesting observation is that

APP06 in Figure 6 appears to run out of physical memory by the 4th day of uptime. The server then falls over to allocating and using swap space.

4.4.3. Storage Capacity

Data was extracted from the servers that showed how full each hard drive volume was on all production servers. The goal was to find volumes that were at risk of exceeding their quota after the implementation of the DML feedback loop. Two hard drive volumes were identified of being at risk (volumes that are more than 80% full).

These volumes are shown in Figure 18 & Figure 19.

Figure 18: /local/app/a_slx on APP06 frequently uses more than 90% of quota

This document is not intended for public distribution. 51 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 19: /app/a_slxftp on the JIP NAS is near capacity

4.4.4. SLMU Message Statistics

The next step was to observe intraday processing of SLMU messages as they hit

the SLX systems. Information was extracted from SLX queries which gave an hourly

breakdown of the number of SLMU messages that were processed during that hour. In

addition to the number of messages, the average time it took to process a single message

during that hour period was returned from the query. This information was plotted on top

of the CPU graph found in Figure 14. This allowed for correlations to be made between

how busy the processors were and how long it took to process an SLMU message. This

graph can be found in Figure 20. According to the graph, the processing time for an

SLMU message is more than two minutes between 3:30PM and 6:00PM daily. This means that SLX is not meeting the service level agreement (SLA) that guarantees processing time within two minutes.

This document is not intended for public distribution. 52 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Average SLMU messages & processing time (from 10/15/06 - 10/21/06)

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

SLMU 0.6 1,000 Messages/25000

SLMU message processing time

Processing SLA of 0.4 100 2 minutes Processing Time (seconds) Time Processing

6 per. Mov. Avg. (% APP06 CPU Utilization)

0.2 10 6 per. Mov. Avg.

# of messages and CPU usage (as a ratio of 100) (% DBP01 CPU Utilization)

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Figure 20: SLMU Messages in SLX

The infrastructure team has commented on this and remarked that there are no

memory limitations on TSJIPUAPP06 server. Piyush Ghelani, system administrator of

JPMorgan’s servers said, “the illusion of being out of memory comes from the fact that

users do not count file cache usage feature of the Unix kernel. Actual memory usage is

4.125 GB.” (Personal Communication, November 22, 2006) Starting after week 44 the

system administrators also limited the size of the file cache so that it would resize itself to prevent the swap space from being used. Even with this change in place, the processing times for SLMU messages remain unchanged as shown in Figure 21. This graph was

created using the same methods that were used in creating the October performance

assessment. The only difference is that for November statistics, the week of November

12th was used.

This document is not intended for public distribution. 53 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. October Performance vs November Performance

15000 100,000

12000 10,000 Projected DML Messages

Oct. SLMU Messages 9000 1,000

Nov. SLMU Messages # of messages # of 6000 100 Processing SLA of 2 minutes Processing Time (seconds) Time Processing

Oct. SLMU processing time 3000 10

Nov. SLMU processing time

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Figure 21: Comparing SLMU Performance Between Oct and Nov

4.4.5. DML Projections

The final step was to take intraday trading information from DML and to project when and how many messages would be sent to SLX during the day. The method of plotting the information was to look at the sum of the loan and component updates during the hour period and to treat each as a single message. This closely matches what SLMU does when sending messages to SLX. The plot was then shown on the same graph as the

SLMU messages and CPU usage. In doing this, one could see if the SLMU message peaks matched up with DML peaks. This combined graph can be seen below in Figure

22.

This document is not intended for public distribution. 54 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

SLMU Messages/25000 0.6 1,000 DML Messages/25000

SLMU message processing time 0.4 100 Processing SLA of Processing Time (seconds) Time Processing 2 minutes

6 per. Mov. Avg. (% APP06 CPU Utilization) 0.2 10 6 per. Mov. Avg. # of messages and CPU usage (as a ratio of 100) (% DBP01 CPU Utilization)

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Figure 22: SLMU and Anticipated DML Messages over a Trading Day

In looking at the graph, DML is anticipated to have a spike in messages between

11:00AM and 1:00PM. As a result, it is predicted that the processing time will be greater than two minutes during this time frame and thus, the two minute SLA will not be met between 11:00AM and 1:00PM.

This document is not intended for public distribution. 55 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 5. Recommendations

5.1. Project Management Process Improvements

Based on our evaluation of current project management practices and our analysis of the areas in need of improvement, we have developed a project management process that encompasses many of the responsibilities of JPMorgan project managers. In particular, the process involves the three areas that have been focused on throughout this paper: gathering project cost and duration estimates for the Investment Council, creating a project plan and using it to manage the project, and tracking action items. The flowchart below illustrates the process that we have developed. We recommend that

JPMorgan adopts this process. The main recommendations that summarize this process are:

1. Use the Estimation Workbook in place of STMate to gather information for estimates and project plan creation.

2. Take full advantage of Microsoft Project and the critical path method to schedule tasks and better manage projects.

3. Use the Action Item Tracking Sheet to track the status of action items and project risks.

4. Hold weekly status meetings to discuss progress and encourage communication across project roles.

The following sections explain this process in greater detail. We also recommend that the firm conduct a more in-depth study to identify project management best practices and develop a common process to be used by all project managers for all projects in

This document is not intended for public distribution. 56 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. JPMorgan Securities Lending. This effort could possibly be spearheaded by another team on WPI students.

Figure 23: Project Management Process

This document is not intended for public distribution. 57 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 5.1.1. The Estimation Workbook

We have developed a new tool called the Estimation Workbook that we recommend be used in place of STMate to gather estimates for Investment Council project funding requests. The Estimation Workbook is a simple Excel workbook that will capture all of the necessary information from project team members, while eliminating the unnecessary complexity and confusion associated with the STMate tool. To ensure success in using this tool, the project team should follow the process that we have developed for its use.

The project manager should first hold a meeting with the entire project team to make a list of the project milestones and deliverables and their dependencies. Each team member should then work offline to develop a list of all foreseen tasks that will be required to accomplish the project milestones and deliverables. The team should then come together again to compile a comprehensive list of all project tasks and task dependencies. The team should reference the list of tasks in STMate to make sure they account for all tasks that will need to be completed. For clarification purposes, a milestone is a significant accomplishment or intermediate project goal. A deliverable is any tangible outcome that is produced by the project. These can be documents, plans, computer systems, buildings etc. Lastly, a task is any piece of work that is undertaken that adds value to the creation of a project deliverable or fulfillment of a project objective.

Once this list is compiled and the team agrees on it, the project manager should set up the Estimation Workbook. A new Estimation Workbook will need to be created

This document is not intended for public distribution. 58 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. for each project and should be based on the example that we have created for this project.

In every Estimation Workbook there will be tabs named Instructions, Assumptions,

Summary Sheet by Phase, and Summary Sheet by Role. There will also be a tab for each key project team member labeled with their role in the project. Each team member tab and the summary sheet by role tab should have the comprehensive list of project tasks along the left edge. The milestones and deliverables should be in bold with the respective subtasks listed directly beneath each. After the project manager sets up the Estimation

Workbook with all worksheets, columns and formulas, he/she should post it on a shared drive and notify the project team that it is ready for their inputs. The figures below are snapshots of the sample Estimation Workbook that we have developed.

Figure 24: Estimation Workbook- Instructions

This document is not intended for public distribution. 59 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. This instruction sheet is a summary of the process that we are recommending. It gives instructions for the project manager on how to create the Estimation Workbook and also for team members on how to fill it in with their estimates. This instruction worksheet should be copied into every new Estimation Workbook.

Figure 25: Estimation Workbook- Team Member Worksheet

Each member of the project team that is required to provide estimates should have

their own tab in the Estimation Workbook. The tab should be labeled with the team

member’s role such as BA for Business Analyst or QA for Quality Assurance. The tabs

should all be identical and should look like the figure above. They will each have the

comprehensive list of all project tasks in the left hand column and a list of the project

This document is not intended for public distribution. 60 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. phases along the top edge of the worksheet. Once the project manager sends out notification that the workbook is posted on the shared drive, each team member should open the workbook and find the tab labeled with their role. They should then fill in their estimates in man-days for how long they think each task will take them in each phase of the project. These are estimates of pure effort not of elapsed duration. If a task does not apply to a particular team member, then that team member should leave that task blank or enter a zero. Each team member should also click on the assumption tab and review the list of assumptions started by the project manager. If a team member wants to modify or add to the list of assumptions, he/she should do so, making sure to note their name and the date next to any changes. Team members should not enter any information into the summary sheets. These sheets are linked to the team member estimate tabs and will aggregate automatically. Once a team member finishes entering their information into the Estimation Workbook, he/she should save the changes on the shared drive and notify the project manager.

This document is not intended for public distribution. 61 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 26: Estimation Workbook- Summary Sheet by Phase

The Summary Sheet by Phase provides the project manager with total time and

total cost broken down for each phase of the project. The time totals are shown in days,

hours, and months. Once all team members have filled in their worksheet, the project

manager should pull the file off the shared drive. Since this summary sheet is linked to the data entered in the team member estimate sheets, the project manager just needs to fill in the cost table with the cost per day of each team member. Once that is complete, the

Estimation Workbook will show the total time and cost for each phase of the project.

This document is not intended for public distribution. 62 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 27: Estimation Workbook- Summary Sheet by Role

The summary sheet by role is also linked to the team member estimation sheets

and contains the same comprehensive list of project tasks. When the project manager

sets up the Estimation Workbook, he/she should enter the task predecessors in the right

hand column of this tab. After all team members enter their estimates, this summary

sheet shows the total time and cost for each team member along the bottom edge of the

worksheet. The time totals are again given in days, hours, and months. Also, along the

right side of the sheet, next to the task predecessors are total man-days for each task across all team members. Between the two summary sheets, the project manager should have total duration and cost estimates broken down in a variety of ways that will be useful in presenting to the Investment Council.

This document is not intended for public distribution. 63 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. We realize that this process requires a substantial amount of time upfront to compile the comprehensive list of project tasks and dependencies; however, we feel the benefits of gathering estimates using the Estimation Workbook outweigh this extra upfront time. Some of the most important benefits are:

• Clearer understanding of project tasks by all team members because of the comprehensive task list on all estimate tabs

• Cost and duration totals broken down in several ways

• Eliminates the need for the project manager to manually merge the estimates from each team member into one estimate

• Simpler design and linked worksheets reduces the time all team members have to spend providing estimates

5.1.2. Microsoft Project and the Critical Path Method

The next step in our proposed project management process is to create a project plan in Microsoft Project using the comprehensive task list, task predecessors, and task effort estimates from the Estimation Workbook. The project plan should be set up in such a way that it has two layers. The high level layer should include the network of project milestones and deliverables and their dependencies. Below this, there should be a more granular layer with the network of tasks needed to complete each deliverable and reach each milestone. A diagram illustrating this point is shown below. The pink boxes show the network of project deliverables. The blue circles represent the network of tasks required to accomplish each deliverable.

This document is not intended for public distribution. 64 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 28: Diagram of Two Layer Project

The project manager and the delivery manager should also work together to

compile a list of all of the resources that will be working on the project. Then they

should look up each person’s vacation schedule and determine what percentage of each

person’s time will be allocated to that particular project. Once all of this information is

entered into Microsoft Project, it will automatically create a project schedule. The

project manager should use the Detail Gantt Chart view to see the critical path for the project. All critical tasks are highlighted in red and should be given priority. To ensure

that resources are not over-allocated and that the schedule depicts realistic dates, the

project manager should re-schedule concurrent tasks assigned to each resource to give

priority to critical tasks and subsequently fill in non-critical tasks in the open areas of the

schedule.

This document is not intended for public distribution. 65 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. We recommend that project managers take a Microsoft Project training class to ensure they are leveraging as many functions of the program as possible. The main goal is to have the most accurate project plan as possible by accounting for such things as vacation time and resource allocation to other projects. With a solid project plan as the foundation, the project manager can ensure on-time delivery by giving priority to critical tasks. It may even be possible to expedite the completion of a project by analyzing options such as allocating extra resources to critical tasks or breaking the project into key deliverables that could be delivered separately and at different times.

Once the project manager develops a sound project plan, the project team should use it to prioritize and manage their work loads and due dates. To help with this, we recommend that the project manager print and post relevant sections of the project plan on a highly visible wall. Using the project plan in this way ensures that it is used as a planning and tracking tool, rather than simply a tracking tool as it was with the current system. The foreseen benefits of this process are as follows:

• Substantially less time required by a project manager to create the project plan because information is taken directly from the Estimation Workbook

• More accurate sense of project end date

• More likely on-time delivery because all team member recognize and give priority to critical tasks

• Fewer project delays due to increased accountability and awareness of due dates

This document is not intended for public distribution. 66 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. 5.1.3. Action Item Tracking

We recommend that the JPMorgan Securities Lending Technology division begin using the following process and tool to track the status of action items that arise through formal and informal team meetings. As team members identify new action items, they should send an email to the project manager with the following information:

• A description of the action item

• The milestone or deliverable from the project plan that the action item will help to fulfill (the team should refer to the project plan for list of tasks and deliverables)

• The open date and due date of the action item

• The team member responsible for the action item

This should also be extended to tracking risks associated with the project. When new risks are identified, the same information from above should be sent in an email to the project manager. Additionally, the email should contain a brief risk mitigation strategy.

The project manager should then use the Action Item Tracking Sheet that we have developed to input this information for each action item and risk. The tool is an excel workbook with three worksheets. The first is an instruction sheet for the project manager and can be seen below.

This document is not intended for public distribution. 67 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 29: Action Item Tracking Sheet Instructions

The other two sheets are where the actual action items and risks are input and tracked. The project manager should input all information from the team emails in the appropriate sheet. These two sheets can be seen in the figures below. The initial status of all action items and risk items should be open. As progress and status changes are made on action items, notification should be sent to the project manager through email.

This document is not intended for public distribution. 68 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Figure 30: Action Item Tracking Sheet for Action Items

Figure 31: Action Item Tracking Sheet for Risk Items

This document is not intended for public distribution. 69 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Using the Action Item Tracking Sheet in the manner stated above will result in many benefits for Securities Lending Technology project teams at JPMorgan. Action items that are essential to completing project deliverables will all be housed in one location. This will eliminate the need to search through numerous emails and the risk that the team loses sight of important action items. The new capability to track the status and outcome of action items and risks ensures a more thorough completion of action items and mitigation of risk. This will result in higher quality project deliverables.

Lastly, the due dates and updates at weekly status meetings will increase accountability of team members and help to keep the project on schedule.

5.1.4. Weekly Status Meetings

The other key part of our process is that the delivery manager holds weekly status meetings. At these meetings the team should have a discussion, led by the delivery manager, to bring the team up to date on the status of project milestones, to determine a strategy for how to reach the next project milestones, and to identify next steps.

Additionally, we recommend that the team discuss any open action items with pressing due dates or that are pending closure. During this time, each team member should give a brief report regarding the status of the items they own. If a team member believes an action item or risk status should be changed to closed, the team should come to a consensus that sufficient work has been done to close the item. After these meetings the project manager should update the Action Item Tracking Sheet, which should be housed in PKS for all team members to review frequently. The project manager should also update the project plan, if applicable, after the weekly status meetings. He/she should then print and post a new version of the project schedule on the wall.

This document is not intended for public distribution. 70 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. These weekly status meetings will ensure that all team members are aware of the current status of the project, even regarding areas where they are not directly involved.

We believe this will boost communication and the flow of ideas between team members and also bring any problems or roadblocks to the forefront sooner. We also feel that the weekly status meetings will increase accountability and adherence to deadlines because team members will be required to report on their progress each week.

5.2. IT Equipment Upgrades for SLX

Based on the capacity planning research and analysis, we have developed a list of recommendations to aid JPMorgan with the successful implementation of the DML to

SLX feedback loop into production, such that server performance is not compromised.

Our recommendations are as follows:

• JPMorgan should use the capacity planning document, found in Appendix I, to estimate server loads and to upgrade hardware on the servers in order to meet future demand.

• A member of the infrastructure team should be contacted and brought on board for the rest of this project. This individual should be responsible for maintaining and updating the capacity planning document.

• The infrastructure team should perform further analysis if necessary on quality assurance, disaster recovery, and development environments to ensure that all SLX environments have adequate resources.

• Storage space on two volumes should be increased to meet future demands from the DML feedback loop. These storage volumes are: o /dev/slxp1-slxhom1lv o jipnas1:/vol/jipnas1_app0/tss_slx_prd1

This document is not intended for public distribution. 71 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. • Add more processors in order to meet the two minute service level agreement for intraday message processing time. Without these upgrades the service level agreement will not be met between 11:00AM and 1:00PM and 3:30PM and 6:00PM.

This document is not intended for public distribution. 72 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Works Cited

[Annual Report 2005] (2006). Annual Report 2005. Retrieved November 30, 2006 from http://files.shareholder.com/downloads/ONE/62656512x0x34666/D25C186E- 7D76-4BA3-AC27-445C45622401/ar05_complete.pdf

[Fortune] (2006). Fortune500 2006 Our Annual Ranking of America’s Largest Corporations. Retrieved October 23, 2006. from http://money.cnn.com/magazines/fortune/fortune500/full_list/

Ghelani, Piyush (2006). WSS Unix webdoc 1.1 for JPMChase. Retrieved November 30 2006 from http://dcfdoc.gis.chase.com

[GTI Hobbit] (2006). Hobbit Monitor 4.1.2: WSS Unix Health Status. Retrieved November 30, 2006 from http://wssunixmonitor.gis.chase.com/

[History] (2006). The History of Our Firm. Retrieved November 30, 2006 from http://www.jpmorganchase.com/cm/cs?pagename=Chase/Href&urlname=jpmc/ab out/history

[Home] (2006). JPMorgan Home Page. Retrieved November 20, 2006 from http://www.jpmorgan.com

Internet Center for Management and Business Administration, Inc. (2006). CPM- Critical Path Method. Retreived November 16, 2006 from http://www.netmba.com/operations/project/cpm/.

[Java Trademark] Java and the Java logo are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. or other countries.

JPMorgan Chase & Co. (2006). Securities Lending Overview. New York, NY: Amar Devasthali.

JPMorgan Chase & Co. (2006). Comprehensive Business Requirements Document: DML to SLX Integration. New York, NY: Brian Kenney.

Morris, K.M. & Siegel, A.M. (1993). The Wall Street Journal Guide to Understanding Money & Investing. New York: Lightbulb Press, Inc.

[Project Delivery Framwork] (2006). WSS Project Delivery Framework (PDF). Retrieved November 29, 2006, from http://tss-cb-cto-tech.jpmorganchase.com/dept/WSS/permits.asp

This document is not intended for public distribution. 73 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. [WSS] (2006). About Worldwide Securities Services. Retrieved October 23, 2006, from http://www.jpmorgan.com/cm/ContentServer?c=TS_Content&pagename=jpmorg an%2Fts%2FTS_Content%2FGeneral&cid=1114735359705.

This document is not intended for public distribution. 74 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Appendix A: Securities Lending Technology Systems Relationships Party FED DTC FTID Crest Pirum rd Loanet Market Market 3 Internet Interfaces EuroClear Custodians EQUILEND Clearing & Settlement Clearing Browser Hub (WMQI) Dallas Dallas Message SLX Infrastructure • Initiation Trade SLX SLX SEI Custody •B1 U.S. U.S. •B1 Mainframe & Transformation • Routing Message TITAN • Custody U.S. SLMU • Domestic Domestic Trade Initiation [SunGard] Mgmt • Non-custody WorldLend • Collateral Cash SLAM / CTE / SLAM COSMIC • Int’l Custody Asset Management Loan Maintenance Loan DML Alloc/Reporting • Collateral LC Blockade • International International [SunGard] •UK Domestic •UK GlobalOne Reconciliations, business functions Accruals, and other other and Accruals, UDT’s UDT’s MIS Views Views Reporting Global- Portfolio

Reporting

E-mail

Other Corp Other

Core Processing Core User Interfaces User WSS and and WSS

Diagram courtesy of JPMorgan (Devasthali, 2006)

This document is not intended for public distribution. 75 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Appendix B: High Level Task Document for Project Plan Creation

We would like to brainstorm if these are all of the major activities and begin to determine the predecessors of each. This will facilitate the E2 estimate process and subsequently a critical path analysis to examine ways to expedite the project. Also we would like to map out and agree upon the high level design for the completion of the project.

Keep in mind:

**Every high level task should add value to fulfilling either a required document for the PDF process or one of the direct objectives for the project. **

The objectives are as follows:

Objective 1 – SLX to have accurate international utilization at start of day

Objective 2 – SLX to have accurate intraday international utilization

Objective 3 – Enable traders to view a current picture of each open loan in SLX rather than having to toggle between SLX and DML.

Objective 4 – The ability to define at will which value should be used to impact utilization: either the required collateral or the DML Calculated Value.

Major Milestones by Phase

Requirements Phase

Tasks Documentation SLX/DML High Level Pre-design Work • DML Calculation of limits* • Verify that DML does not calculate limits differently for different loan types Detailed Requirements Document • DML Functions that affect utilization and date sensitivity around those functions*

Data Mapping Exercise • Spreadsheet mapping from DML loan master to the loan snapshot file layout • Analysis of borrower relationships Initiate Requirements Traceability Matrix in DML vs. SLX, valid values, valid codes, etc. • Proof of Concept Prototype in SLX to

This document is not intended for public distribution. 76 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. validate data mapping (To include prototyping the End User Transactions Summary and Detailed Screens) Initiate Test Strategy/Plan E2 Estimates Performance Capacity Planning • Examine CPU and RAM usage (currently executing this task) • Analyze Current Performance pertaining to SLMU feedback loop to identify inaccurate/inefficient processes and determine whether we can change existing processes due to poor performance that will ameliorate the use of the future SLX/DML Feedback Loop Architecture Permit to Build (If changes arise because of this task, there will be some significant Testing Effort required to test changes to the current process) • Work with GTI to determine next steps for solving anticipated problems • Track existing issues facing production and infrastructure teams on all SLX environments Business Case

Design Phase

Tasks Documentation High Level Design document (author Brian with input from developers) [This doc will build upon the requirements document to Update Test Strategy show the users a picture of our High Level Design Concept]

**Implementation of a common I/O (input/output) Module in DML [Dependent upon previous tasks listed in asterisks: Detailed Design/System Specifications Doc DML calculation of limits and DML functions that affect…] Messaging Task Finalize Release Schedule End User Transactions Architecture Step 3 Perimeter Infrastructure Risk Assessment Data Transactions Form Design in SLX Update Requirements Traceability Matrix Design in DML E3 Estimate

This document is not intended for public distribution. 77 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Design in SLX MF Detailed Design Walkthrough Design SOD File

Build/Unit Test

Tasks Documentation Source Code Build/Unit Test DML Update Requirements Traceability Matrix Source Code Build/Unit Test SLX Revise Requirements Source Code Build/Unit Test SLX MF Update End User Documentation Peer Code Review Initiate Application/System Documentation SIT Tests Technology Recovery Action Plan (TRAP) Deployment to QA Update SQA Checklist Update Tollgate Checklist

Testing

Documentation Update End User Documentation Finalize Requirements Traceability Matrix Revise Requirements Initiate Production Readiness Review Update SQA Checklist Update Tollgate Checklist

QA Testing Tasks SIT Volume Performance Test Test Setup (getting access to programs, etc.) Write Test Scripts Test Script Review Execute Error Handling Testing Request GTI Resources Execute Performance Testing Assisted by GTI Resources Execute Functional Requirements Testing Analyze Test Results Regression Testing

UAT Testing Test Setup Plan UAT Execute UAT for DML

This document is not intended for public distribution. 78 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Execute UAT for SLX Execute UAT for SLX MF Analyze Test Results

Training We need to brainstorm ideas for a training strategy.

Implementation

Tasks Documentation Finalize Operations Turnover Code Deployment DML Documentation Code Deployment SLX Update Tollgate Checklist Code Deployment SLX MF Finalize Ender User Documentation Finalize Application/System Implement Changes to Batch Schedule Documentation Finalize ECM Calendar Entry Finalize SLA Finalize Implementation Plan • Create Implementation Tasks • Create ECMS Record (Allow 10-15 Finalize Production Readiness Review days for Review as part of Risk for the project) Update SQA Checklist

Project Closure

Tasks Documentation Post Production Support (2 week warranty Project Closure Document period) Post Implementation Survey Customer Satisfaction Survey Finalize SQA Checklist Finalize Tollgate Checklist

This document is not intended for public distribution. 79 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Appendix C: Snapshots of the STMate Tool

This document is not intended for public distribution. 80 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

This document is not intended for public distribution. 81 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Appendix D: Screenshots of the same loan in DML and SLX

Omni view of a loan in DML

Component view of a loan in DML

This document is not intended for public distribution. 82 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

Trade level view of a loan in SLX Production

Position level view of a loan in SLX Production

This document is not intended for public distribution. 83 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Appendix E: DML Loan Master Parser Source Code

/** * DML Loan Master Parser * * Written by: Michael Kristan ([email protected]) * Securities Lending Technology * JPMorgan Worldwide Securities Services * * Copyright 2006 JPMorgan Chase & Co. All Rights Reserved. * Internal use only. Not for distribution or use outside * of the firm without permission from the author or the firm. * * * About this program: * This program parses a DML Loan Master file (in .csv format) * and splits it up into smaller parts to allow for easier * processing by the Microsoft Office suite of applications. * * Command line arguments (in order): * Maximum number of records that can appear in an output file * Input .csv file name * Output .csv file name * * Example usage: * java.exe DMLLoanMasterParser 10000 C:\input.csv C:\output.csv * * In the example above, the file input.csv is read and split up * into groups of 10,000 records. Each group is output into a csv file * containing the name C:\output * * Most java virtual machines will require the full package name * of the class when being executed. In that event, use * com.jpmorgan.wss.seclend.dmlslx.DMLLoanMasterParser after java.exe * * Notes: * The output file is modified by the software by appending * a number to the file name. Example, the first 10000 records * are stored in output1.csv, the second 10000 records are in * output2.csv. * * There is an interesting situation where some of the DML * loan records have extra commas in them which causes errors when * trying to import into Microsoft Access. Records that do not match * the expected number of columns (defined as 125 in constant * numberOfCSVColumns) are instead written into a file with an ERR * suffix (in this example it would be outputERR.csv). * * It has been found that using MS-DOS 8.3 file naming in the * arguments helps when trying to load files that have long file names * or spaces. Example, instead of passing H:\SLX Integration\Loan Master.csv * use H:\SLXINT~1\LOANMA~1.CSV * * This program has been tested on the Sun Java 5.0 SDK & JRE * platforms. The runtime environment for this version of java can * be found at http://java.sun.com This software makes use of a * non-standard java package called csvreader. The class files for * this package can be freely obtained at http://www.csvreader.com at * the time of this writing. * * To compile this file, you need the Java SDK. The command to * compile is javac.exe DMLLoanMasterParser.java. To run this program, * you only need DMLLoanMasterParser.class and do not need the SDK. * */

package com.jpmorgan.wss.seclend.dmlslx;

This document is not intended for public distribution. 84 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. // External classes that DMLLoanMasterParser depends on import com.csvreader.CsvReader; // Used to read CSV files import com.csvreader.CsvWriter; // Used to create CSV files

/** * DMLLoanMasterParser * * This program only uses one class, DMLLoanMasterParser. All * functions in this program are static and are accessed via the * void main function. * * @author Michael Kristan */ public class DMLLoanMasterParser {

/** * Main function * * This function does the parsing of the CSV file as described above * in the program description. There is no error checking to ensure * that the first parameter is an integer or that the paths in the other * parameters are valid. * * @param args A string array that contains the expected arguments as described above * @throws Exception in the event that the wrong number of command arguments are passed */ public static void main(String[] args) throws Exception {

// Validate that we have enough command line arguments if(args.length != 3) throw new Exception("Not enough command line arguments");

// Extract argument parameters into integers and character strings final int maxRecordsPerFile = Integer.parseInt(args[0]); String paramInputFileName = args[1].trim(); String paramOutputFileName = args[2].trim();

// Output general information to the console screen System.out.println("DML Loan Master Parser\nJPMorgan Securities Lending\n"); System.out.println("Number of records per file : " + maxRecordsPerFile); System.out.println("Input file name : " + paramInputFileName); System.out.println("Writing contents of input file to .CSV files, please be patient...");

// Declare the CSV Reader and the CSV Writer for errors CsvReader myReader = new CsvReader(paramInputFileName); CsvWriter errWriter = new CsvWriter(setErrorFileName(paramOutputFileName));

// Setup parameters for size of documents and output file name int fileNumberCounter = 0; // Used for output file names String outputFileName = setOutputFileName(paramOutputFileName, fileNumberCounter); int errNumberCounter = 0; // Used for looking at statistics

// Setup parameters for the definition of the CSV file final int numberOfCSVColumns = 125;

// Boolean based on the myReader.readRecord() output boolean recordsLeftToRead = true;

// Create a loop that will run until the end of the input CSV file is reached while(recordsLeftToRead == true) { // Declare new output file stream and print name to console screen CsvWriter myWriter = new CsvWriter(outputFileName);

This document is not intended for public distribution. 85 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. System.out.println("Output file name : " + outputFileName);

// Loop populate an output CSV file until the maxRecordsPerFile // upper bound is reached. Then break out and have the parent // loop create a new CSV file for(int index = 0; index < maxRecordsPerFile; index++) { // Read a record and implement early exit recordsLeftToRead = myReader.readRecord(); if(recordsLeftToRead == false) break;

// If we run into a situation where the the number of columns are not // equal to the defined constant (numberOfCSVColumns), output that row // to a different file so that the regular CSV file doesn't break when // importing into Microsoft Access if(myReader.getColumnCount() != numberOfCSVColumns) { for(int colCounter = 0; colCounter < myReader.getColumnCount(); colCounter++) { errWriter.write(myReader.get(colCounter)); //System.out.print(myReader.get(colCounter)); } System.out.println("\tOutput Record number : " + index + ", COL_NUM_ERROR: Columns = " + myReader.getColumnCount() + " Skipping..."); errWriter.endRecord(); // Adds a CR and LF to the text file

// Because we're not sending our output in this if statement to myWriter, // there is no need to update the counter that measure the numbers of // records being written to that file. Therefore as a hack, we need to decrement // the index counter by one because it will get incremented automatically by the // for loop. index--;

// Increment the error number counter. errNumberCounter++; }

else { // Iterate through the number of columns and output line for(int colCounter = 0; colCounter < myReader.getColumnCount(); colCounter++) { myWriter.write(myReader.get(colCounter)); //System.out.print(myReader.get(colCounter)); } System.out.println("\tOutput Record number : " + index); myWriter.endRecord(); // Adds a CR and LF to the text file } }

// Close the output file stream myWriter.flush(); myWriter.close();

// Increment the file number counter and set the new output file name fileNumberCounter++; outputFileName = setOutputFileName(paramOutputFileName, fileNumberCounter); }

// Close the file reader and error writer myReader.close(); errWriter.flush(); errWriter.close();

System.out.println("\nFile creation complete!\n\tNumber of errors : " + errNumberCounter); }

This document is not intended for public distribution. 86 Permission must be obtained from JPMorgan Chase & Co. to disclose this document.

/** * setOutputFileName * * This method parses the output filename and addes a numeric counter to the end of the * file name but before the .csv file suffix. * * @param fileName The name of the file that needs to be parsed * @param fileNumberCounter The integer that needs to be concatentated * @return A file name with the fileNumberCounter appended to the file prior to the .csv suffix */ public static String setOutputFileName(String fileName, int fileNumberCounter) { // Parse the string so that we only have the portion before the .csv suffix // then add the filenumber and then add the .csv suffix if(fileName.indexOf(".") > 0) fileName = fileName.substring(0, fileName.indexOf(".")); //System.out.println(fileName); return fileName + fileNumberCounter + ".csv"; }

/** * setErrorFileName * * This method parses the output filename and addes a numeric counter to the end of the * file name but before the .csv file suffix. * * @param fileName The name of the file that needs to be parsed * @return A file name with the characters 'ERR' (without quotes) appended to the file * prior to the .csv suffix */ public static String setErrorFileName(String fileName) { // Parse the string fileName so that we only have the portion before the .csv // suffix. Then add the letters ERR and then add the .csv suffix if(fileName.indexOf(".") > 0) fileName = fileName.substring(0, fileName.indexOf(".")); return fileName + "ERR.csv"; } }

This document is not intended for public distribution. 87 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Appendix F: DML Loan Master Parser JUnit Test Case

package com.jpmorgan.wss.seclend.dmlslx; import junit.framework.TestCase;

/** * Test for DMLLoanMasterParser helper methods * * @author Michael Kristan */ public class TestDMLLoanMasterParser extends TestCase {

/** * Test method for {@link com.jpmorgan.wss.seclend.dmlslx.DMLLoanMasterParser#setOutputFileName(java.lang.String, int)}. */ public void testSetOutputFileName() { String inputStringA = "H:\\output.csv"; String outputStringA1 = "H:\\output1.csv"; String outputStringA2 = "H:\\output2.csv"; String outputStringA_99 = "H:\\output-99.csv";

assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 1), outputStringA1); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 2), outputStringA2); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, -99), outputStringA_99);

inputStringA = "H:\\output";

assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 1), outputStringA1); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 2), outputStringA2); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, -99), outputStringA_99); }

/** * Test method for {@link com.jpmorgan.wss.seclend.dmlslx.DMLLoanMasterParser#setErrorFileName(java.lang.String)}. */ public void testSetErrorFileName() { String inputStringA = "H:\\output.csv"; String outputStringA1 = "H:\\outputERR.csv";

assertEquals(DMLLoanMasterParser.setErrorFileName(inputStringA), outputStringA1);

inputStringA = "H:\\output";

assertEquals(DMLLoanMasterParser.setErrorFileName(inputStringA), outputStringA1); }

}

This document is not intended for public distribution. 88 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Appendix G: Microsoft Access SQL Queries

JOIN DML/SLX BROKERS

SELECT DISTINCTROW [_internal_Remove Quotes from SLX Borrowers].[MR Short Name], [_internal_Remove Quotes from SLX Borrowers].[Relationship Short Name], [_internal_Remove Quotes from SLX Borrowers].DBRELID, [_internal_Remove Quotes from SLX Borrowers].[BG Short Name], [_internal_DMLBrokers Query].[ACCT-LINE], [_internal_DMLBrokers Query].[ACCT-PREFIX], [_internal_DMLBrokers Query].[ACCT- NO], [_internal_DMLBrokers Query].[SETTL-CODE], [_internal_DMLBrokers Query].[COLAT-CODE], [_internal_DMLBrokers Query].[ACCT-TYPE], [_internal_DMLBrokers Query].[_ACCT-NO] AS DMLID FROM [_internal_DMLBrokers Query] INNER JOIN [_internal_Remove Quotes from SLX Borrowers] ON [_internal_DMLBrokers Query].[_ACCT-NO]=[_internal_Remove Quotes from SLX Borrowers].DMLID; DML BROKERS WITHOUT MATCHING SLX RECORDS

SELECT [_internal_DMLBrokers Query].[ACCT-LINE], [_internal_DMLBrokers Query].[ACCT-PREFIX], [_internal_DMLBrokers Query].[ACCT-NO], [_internal_DMLBrokers Query].[_ACCT-NO], [_internal_DMLBrokers Query].[SHORT-NAME], [_internal_DMLBrokers Query].[SETTL-CODE], [_internal_DMLBrokers Query].[COLAT-CODE], [_internal_DMLBrokers Query].[ACCT-TYPE] FROM [_internal_DMLBrokers Query] LEFT JOIN [_internal_Remove Quotes from SLX Borrowers] ON [_internal_DMLBrokers Query].[ACCT-NO] = [_internal_Remove Quotes from SLX Borrowers].DMLID WHERE ((([_internal_Remove Quotes from SLX Borrowers].DMLID) Is Null));

SLX BROKERS WITHOUT MATCHING DML RECORDS

SELECT [_internal_Remove Quotes from SLX Borrowers].DMLID, [_internal_Remove Quotes from SLX Borrowers].[MR Short Name], [_internal_Remove Quotes from SLX Borrowers].[Relationship Short Name], [_internal_Remove Quotes from SLX Borrowers].DBRELID, [_internal_Remove Quotes from SLX Borrowers].[BG Short Name] FROM [_internal_Remove Quotes from SLX Borrowers] LEFT JOIN [_internal_DMLBrokers Query] ON [_internal_Remove Quotes from SLX Borrowers].DMLID = [_internal_DMLBrokers Query].[ACCT-NO] WHERE ((([_internal_DMLBrokers Query].[ACCT-NO]) Is Null));

UNIQUE LOAN-TYPE VALUES

SELECT DISTINCT [Loan Master].[LOAN-TYPE] FROM [Loan Master];

UNIQUE SEC-LOC VALUES

SELECT DISTINCT [Loan Master].[SEC-LOC] FROM [Loan Master];

UNIQUE SEC-TYPE VALUES

SELECT DISTINCT [Loan Master].[SEC-TYPE] FROM [Loan Master];

This document is not intended for public distribution. 89 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Appendix H: Invitation to Final Presentation

You are cordially invited to attend a presentation hosted by Megan Slonski and Michael Kristan of Worcester Polytechnic Institute where they will discuss their contributions to the SLX-DML Integration initiative within WSS Securities Lending Technology.

Date: December 14th, 2006 Time: 3:30PM – 4:30PM Location: 28th Floor Conference Center, 1 Chase Manhattan Plaza, New York, NY

Dial-In: 1.866.870.8212 Participant Passcode: 66370645

Details

As part of our college graduation requirements we must complete a project that ties together our four years of study. This project is referred to as a Major Qualifying Project (MQP) at the university. For our MQP, we spent time at JPMorgan Chase assisting with the SLX-DML Integration project, which was initiated by the Securities Lending Technology team within JPMorgan Worldwide Securities Services. Our contributions include a critical path analysis, recommendations for improvements to project management processes, data field mapping, and server capacity planning. This project was advised by Professors Arthur Gerstenfeld and Michael Ciaraldi of WPI. Individuals who can not physically attend are welcome to connect via the phone number listed above. Time will be reserved for questions and answers.

About Worcester Polytechnic Institute (WPI)

Founded in 1865 in Worcester, Mass., WPI was one of the nation’s first engineering and technology universities. WPI's 18 academic departments offer more than 50 undergraduate and graduate degree programs in science, engineering, technology, management, the social sciences, and the humanities and arts. WPI's world-class faculty work with students in a number of cutting-edge research areas, leading to breakthroughs and innovations in such fields as biotechnology, fuel cells, information security, materials processing, and nanotechnology. Students also have the opportunity to make a difference in communities and organizations around the world through the university's innovative Global Perspective Program. There are more than 20 WPI project centers throughout North America and Central America, Africa, Australia, Asia, and Europe.

High level project description

In 2004, a decision was made to adopt SLX as a strategic front end single trading platform, replacing the trading activity performed on the three lending platforms (SLMU, DML, Global One), leaving them in place only for loan maintenance activities. During 2005, SLX development primarily focused on enabling a viable front end for US domestic trading with the delivery of an SLMU to SLX feedback loop. The "feedback loop" design approach adopted for the US Front-End will be leveraged now for the creation of a DML to SLX loop as the first in a series of efforts that will be undertaken to enable SLX as a viable International trading front end.

This document is not intended for public distribution. 90 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Appendix I: Review of SLX Server Performance

The document in Appendix I provides JPMorgan Chase with a thorough analysis of all server systems related to the DML to SLX Feedback Loop. This document is currently in use by members of the project team as well as by the members of JPMorgan

Global Technology & Infrastructure (GTI) group.

This document is not intended for public distribution. 91 Permission must be obtained from JPMorgan Chase & Co. to disclose this document. Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Worldwide Securities Services Securities Lending DML to SLX Integration

Review of SLX Server Performance

Version 0.5

Project Code: 71707

Copyright © 2006 J. P. Morgan Chase Bank, N.A. All rights reserved. This document contains proprietary and confidential information of JPMorgan. Reproduction, disclosure, or use of any portion of this document without specific written authorization from JPMorgan is strictly prohibited. This restriction applies to the information on every page of the document. Contents may be disclosed only to authorized JPMorgan employees and consultants for the purpose of performing their job responsibilities.

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Table of Contents Introduction ...... 4 Purpose of Document...... 4 Definitions ...... 4 Glossary of Terms & Acronyms ...... 4 SLX Environments ...... 6 Hardware Specifications ...... 6 Production server specifications...... 6 Development server specifications...... 7 Disaster Recovery server specifications...... 7 Quality Assurance server specifications...... 7 Cross reference of SLX hard disk volumes to UNIX mounted file systems...... 8 Analysis...... 11 Current Database Usage...... 11 CPU load on production servers...... 11 Possible zombie processes ...... 12 Memory allocation on production servers...... 12 Disk space on production servers...... 14 SLX New Loan Volumes...... 15 SLMU Volumes...... 17 Number of SLMU messages...... 17 Processing times for messages...... 19 Table sizes...... 19 Anticipated volume increases ...... 20 DML Volumes ...... 20 Current DML intra-day load ...... 20 Table Sizes ...... 22 Anticipated volume increases ...... 22 Requirements around timing of DML batch processes...... 23 Recommendations and Next Steps...... 23 Recommendations...... 23 Next Steps ...... 24 Reference Documents ...... 25 Document Control...... 25 Document History ...... 25

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 2 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Document Storage...... 25 Appendices...... 26 PowerPoint Slides For Capacity Planning...... 26

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 3 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

INTRODUCTION

PURPOSE OF DOCUMENT The purpose of this document is to give the reader a comprehensive understanding of the resources required to run SLX in a production environment and to analyze the impact that the DML feedback loop will have on the systems. Information has been gathered from many sources and compiled together to form a complete picture of the resources that SLX depends on. The goal is to provide enough information in order to make infrastructure decisions moving forward.

DEFINITIONS Glossary of Terms & Acronyms

Term Description APD06 APD06 is short for the server name TSJIPUAPD06. APD06 is the development server for all SLX environments (including production, DR, & QA).

APP06 APP06 is short for the server name TSJIPUAPP06. APP06 is the main SLX processing server for SLXP.

APP10 APP10 is short for the server name TSJIPUAPP10. APP10 is the user SLX server for SLXP. JPMorgan traders interface through this server.

APR05 APR05 is short for the server name TSMCCUAPR05. APR05 is a disaster recovery SLX server that is not located in the same area as the SLX production servers.

APR06 APR06 is short for the server name TSMCCUAPR06. APR06 is a disaster recovery SLX server that is not located in the same area as the SLX production servers.

APQ05 APQ05 is short for the server name TSMCCUAPQ05. APQ05 is a quality assurance SLX server.

APQ06 APQ06 is short for the server name TSMCCUAPQ06. APQ06 is a quality assurance SLX server.

Autoborrow Autoborrow is an intelligent, fully automated way for borrowers and lenders to transact US and non-US equity and fixed income securities. Customizable schedules and rules automate the process between the borrower and lender, making the trade execution simple. Autoborrow is part of the Equilend trading system.

Cache Cache is a collection of data duplicating original values stored elsewhere or computed earlier, where the original data is expensive (usually in terms of access time) to fetch or compute relative to reading the cache. Once the data is stored in the cache, future use can be made by accessing the cached copy rather than refetching or recomputing the original data, so that the average access time is lower.

CPU The CPU (also known as the Central Processing Unit or the processor) is the main processor on the server. The current CPU load measures how busy the server is.

DBP01 DBP01 is short for the server name TSJIPUDBP01. DBP01 is the production server that hosts the SLX Oracle Database.

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 4 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Term Description DML DML (also known as SLS) is the Securities Lending platform currently used to support trading and loan maintenance activities for international assets.

DR DR stands for disaster recovery.

Equilend Formed in 2001 by its ten original investors, EquiLend delivers standardized, centralized and global securities lending solutions to the equity and fixed income markets, while providing operational efficiencies across organizations. The EquiLend platform is designed to process equity and fixed income securities lending transactions on a global basis.

GB GB stands for gigabyte. It is a measure of size on computers. A gigabyte is equal to 1,024 MB (megabytes). Gigabytes are frequently used when measuring sizes of hard disks and sometimes used to measure memory.

GHz GHz stands for gigahertz. It is a measure of frequency and is used with computer when discussing speed. The higher the frequency, the faster the processing speed of a computer. 1 GHz is equal to 10-9 seconds.

MB MB stands for megabyte. It is a measure of size on computers. A megabyte is equal to 1,024 KB (kilobytes). Megabytes are frequently used to measure size of memory.

NAS Network-attached storage (NAS) is hard disk storage that is set up with its own network address rather than being attached to the department computer that is serving applications to a network's workstation users. By removing storage access and its management from the department server, both application programming and files can be served faster because they are not competing for the same processor resources. The network-attached storage device is attached to a local area network (typically, an Ethernet network) and assigned an IP address. File requests are mapped by the main server to the NAS file server. Network-attached storage consists of hard disk storage, including multi-disk RAID systems, and software for configuring and mapping file locations to the network-attached device. Network-attached storage can be a step toward and included as part of a more sophisticated storage system known as a storage area network (SAN).

SAN In computing, a storage area network (SAN) is a network designed to attach computer storage devices such as disk array controllers and tape libraries to servers. As of 2005, SANs are common in enterprise storage.

SLA SLA stands for Service Level Agreement. An SLA is a contract or agreement that guarantees a certain level of performance or turnaround time usually between two applications or systems.

SLMU SLMU (also known as Securities Lending MenU) was the Securities Lending platform previously used by US domestic traders. SLMU is only used for loan maintenance now and has been integrated into SLX by a feedback loop.

SLX SLX (also known as Securities Lending eXpress) is a front-end trading platform that is strategically envisioned to replace all trading activity currently performed on any of the lending platforms with those platforms then being retained for loan maintenance type activities only.

SLXP SLXP stands for the production environment of SLX.

Swap/Page file A swap file (or swap space or, in Windows NT, a pagefile) is a space on a hard disk used as the virtual memory extension of a computer's real memory (RAM). Having a swap file

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 5 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Term Description allows your computer's operating system to pretend that you have more RAM than you actually do. The least recently used files in RAM can be "swapped out" to your hard disk until they are needed later so that new files can be "swapped in" to RAM. In larger operating systems (such as IBM's OS/390), the units that are moved are called pages and the swapping is called paging. One advantage of a swap file is that it can be organized as a single contiguous space so that fewer I/O operations are required to read or write a complete file. In general, Windows and Unix-based operating systems provide a default swap file of a certain size that the user or a system administrator can usually change.

TB TB stands for terabyte. It is a measure of size on computers. A terabyte is equal to 1,024 GB (gigabytes). Terabytes are frequently used when measuring sizes of large hard disk arrays that store log files, databases, or other data-intensive files.

QA QA stands for quality assurance. Development efforts on SLX are first tested on the QA servers and environments before being phased into production.

SLX Environments SLX Type Environment Names Related Server(s) Production SLXP TSJIPUAPP06, TSJIPUAPP10, TSJIPUDBP01 Development SLXG, SLXD, SLXI, SLXT TSJIPUAPD06 Disaster Recovery (DR) SLXP TSMCCUAPR05, TSMCCUAPR06 Quality Assurance (QA) SLXQ, SLXR TSMCCUAPQ05, TSMCCUAPQ06

HARDWARE SPECIFICATIONS Production server specifications APP06 APP10 DBP01 Total Number of Processors 4 2 8 Processors assigned to SLX 4 2 3.5 Processor Speed 1.7 GHz 1.7 GHz 1.7 GHz Processor Type 64 bit 64 bit 64 bit L2 Cache Size 1.5 MB 1.5 MB 1.5 MB Physical Memory 7.75 GB 7.75 GB 62 GB Known Paging Files /dev/hd6 : 4 GB /dev/hd6 : 4 GB /dev/hd6 : 4 GB /dev/paging00 : 6GB SLX Disk Drives 9 11 6 Network Interfaces en8 : 10.75.107.54 en8 : 10.75.107.62 en25 : 10.75.107.22 % Allocation to SLX 100% 100% 14.29%

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 6 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

APP06 APP10 DBP01 Monthly Operating Cost $1,218.00 $1,218.00 $2,493.52

Development server specifications APD06 Total Number of Processors 2 Processor Speed 563.3 MHz Processor Type 64 bit L2 Cache Size 1.5 MB Physical Memory 7.75 GB Known Paging Files /dev/hd6 : 6GB SLX Disk Drives 10 Network Interfaces en2 : 10.75.85.30 % Allocation to SLX 100% Monthly Operating Cost $1,447.77

Disaster Recovery server specifications APR05 APR06 Total Number of Processors 2 1 Processor Speed 1.7 GHz 1.7 GHz Processor Type 64 bit 64 bit L2 Cache Size 1.5 MB 1.5 MB Physical Memory 7.5 GB 7.5 GB Known Paging Files /dev/hd6 : 4GB /dev/hd6 : 4GB SLX Disk Drives 11 9 Network Interfaces en11 : 10.31.77.67 en11 : 10.31.77.75 en12 : 10.31.87.98 % Allocation to SLX 100% 100% Monthly Operating Cost $1,218.00 $1,218.00

Quality Assurance server specifications APQ05 APQ06

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 7 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

APQ05 APQ06 Total Number of Processors 2 3 Processor Speed 1.7 GHz 1.7 GHz Processor Type 64 bit 64 bit L2 Cache Size 1.5 MB 1.5 MB Physical Memory 7.5 GB 7.5 GB Known Paging Files /dev/hd6 : 4 GB /dev/hd6 : 4 GB SLX Disk Drives 11 10 Network Interfaces en3 : 10.31.77.65 en3 : 10.31.77.73 en4 : 10.31.87.73 en4 : 10.31.87.93 % Allocation to SLX 100% 100% Monthly Operating Cost $1,218.00 $1,218.00

Cross reference of SLX hard disk volumes to UNIX mounted file systems Device path Mounted File System Quota APP06 /dev/slxp1-sftwrelv /software 640 MB /dev/slxp1-local1lv /local 192 MB /dev/slxp1-slxhomlv /local/app/a_slx 8.06 GB /dev/slxp1-mqmhomlv /local/app/mqm 64 MB /dev/slxp1-mqdatalv /local/data/mqm 64 MB /dev/slxp1-mqmloglv /local/data/mqm/log 320 MB /dev/slxp1-mqmerrlv /local/data/mqm/errors 64 MB /dev/slxp1-gtamsglv /local/app/gtamsg 64 MB /dev/slxp1-oraclelv /local/SLX/oracle 2.97 GB APP10 /dev/slxp2-local1lv /local 192 MB /dev/slxp2-slxhomlv /local/app/a_slx 8.06 GB /dev/slxp2-mqmhomlv /local/app/mqm 64 MB /dev/slxp2-mqdatalv /local/data/mqm 64 MB /dev/slxp2-mqmloglv /local/data/mqm/log 320 MB /dev/slxp2-mqmerrlv /local/data/mqm/errors 64 MB /dev/slxp2-gramsglv /local/app/gtamsg 64 MB /dev/slxp2-oraclelv /local/SLX/oracle 2.97 GB /dev/slxp2-homedirlv /local/home 320 MB

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 8 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Device path Mounted File System Quota /dev/slxp2-ibmhttlv /local/IBMHTTPD.8 512 MB /dev/slxp2-slxloglv /local/app/a_slx/logs 64 MB DBP01 /dev/dbp01-slxora1 /local/SLX/oracle 8 GB /dev/dbp01-slxolog /local/SLX/logs/oracle 2.92 GB /dev/dbp01-slxoaud /local/SLX/oraaudit01 1 GB /dev/dbp01-slxwora /local/SLX/work/oradba 1 GB /dev/dbp01-SLXobck /local/SLX/orabackup01 54.5 GB /dev/dbp01-SLXoftp /local/SLX/oraftp 32 GB

APR05 /dev/slxp1-sftwrelv /software 2.5 GB /dev/slxp1-local1lv /local 1.5 GB /dev/slxp1-slxhom1lv /local/app/a_slx 8 GB /dev/slxp1-mqmhomlv /local/app/mqm 512 MB /dev/slxp1-mqdatalv /local/data/mqm 512 MB /dev/slxp1-mqmerrlv /local/data/mqm/errors 512 MB /dev/slxp1-gtamsglv /local/app/gtamsg 512 MB /dev/slxp1-oraclelv /local/SLX/oracle 16 GB /dev/slxp1-mqmloglv /local/data/mqm/log 2.5 GB /dev/slxp1-ibmhttlv /local/IBMHTTPD.8 512 MB /dev/slxp1-slxloglv /local/app/a_slx/logs 512 MB APR06 /dev/slxp2-local1lv /local 1.5 GB /dev/slxp2-slxhomlv /local/app/a_slx 8 GB /dev/slxp2-mqmhomlv /local/app/mqm 512 MB /dev/slxp2-mqdatalv /local/data/mqm 512 MB /dev/slxp2-gtamsglv /local/app/gtamsg 512 MB /dev/slxp2-mqmerrlv /local/data/mqm/errors 512 MB /dev/slxp2-oraclelv /local/SLX/oracle 16 GB /dev/slxp2-mqmloglv /local/data/mqm/log 2.5 GB /dev/slxp2-ibmhttlv /local/IBMHTTPD.8 512 MB APQ05 /dev/slxq1-locallv1 /local 128 MB /dev/slxq1-slxhomlv /local/app/a_slx 12.4 GB /dev/slxq1-slxftplv /local/app/a_slxftp 4 GB /dev/slxq1-usrhomlv /local/home 320 MB /dev/slxq1-oraclelv /local/SLX/oracle 2 GB

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 9 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Device path Mounted File System Quota /dev/slxq1-mqdatalv /local/data/mqm 64 MB /dev/slxq1-mqmerrlv /local/data/mqm/errors 64 MB /dev/slxq1-mqmhomlv /local/app/mqm 64 MB /dev/slxq1-gtamsglv /local/app/gtamsg 64 MB /dev/slxq1-sftwrelv /software 640 MB /dev/slxq1-ibmhttlv /local/IBMHTTPD.8 512 MB APQ06 /dev/slxq2-slxhomlv /local 160 MB /dev/slxq2-usrhomlv /local/app/a_slx 12.4 GB /dev/slxq2-usrhomlv /local/home 320 MB /dev/slxq2-oraclelv /local/SLX/oracle 2 GB /dev/slxq2-mqdatalv /local/data/mqm 64 MB /dev/slxq2-mqmerrlv /local/data/mqm/errors 64 MB /dev/slxq2-mqmhomlv /local/app/mqm 64 MB /dev/slxq2-gtamsglv /local/app/gtamsg 64 MB /dev/slxq2-sftwrelv /software 640 MB /dev/slxq2-ibmhttlv /local/IBMHTTPD.8 512 MB APD06 /dev/slxd1-slxora /local/slx/oracle 2 GB /dev/slxd1-gtamsg /local/app/gtamsg 32 MB /dev/slxd1-mqdatalv /local/data/mqm 256 MB /dev/slxd1-mqlogslv /local/data/mqm/logs 256 MB /dev/slxd1-mqhomelv /local/app/mqm 256 MB /dev/slxd1-slxhomlv /local/app/a_slx 27.7 GB /dev/slxd1-slxoralv /local/SLX/oracle 3 GB /dev/slxd1-slxftplv /local/app/a_slxftp 4 GB /dev/slxd1-appesmlv /local/app/esm 256 MB Shared Network Attached Storage (NAS) jipnas1:/vol/jipnas1_app0/tss_slx_prd1 /app/a_slxftp 4 GB jipnas1:/vol/jipnas1_app0/tss_slx_prd2 /app/a_slx/slxp/SLXP/export 4 GB jipnas1:/vol/jipnas1_app0/tss_slx_prd3 /app/a_slx/slxp/SLXP/out 4 GB mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX /app/a_slxftp 4 GB mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX2 /app/a_slx/slxq/SLXQ/export 2 GB mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX3 /app/a_slx/slxq/SLXQ/out 2 GB mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX4 /app/a_slx/slxr/SLXR/export 2 GB mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX5 /app/a_slx/slxr/SLXR/out 2 GB

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 10 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

ANALYSIS

CURRENT DATABASE USAGE In order to do a successful capacity study on SLX, attention needs to be paid to the current state of server resources. In the following sections, an analysis was made on CPU, memory, and disk space on SLX production servers. DR, QA, and development were not included in this study because performance isn’t a priority on those servers. They also share resources with other systems that affect performance on SLX. CPU load on production servers In Figure 1, the production server’s CPU usage was plotted on a 24 hour horizontal axis. The values for each point were calculated by observing all 7 days of CPU values for the specified time interval and taking the median value. The median was used instead of the mean because weekend downtime would have skewed the plot for certain time intervals. A six term moving average was plotted on top of the data points to emphasize trends.

Median production server CPU usage (10/15/06 - 10/21/06)

1

% APP06 CPU 0.8 Utilization

% DBP01 CPU Utilization

0.6 % APP10 CPU Utilization

6 per. Mov. Avg. (% APP06 CPU 0.4 Utilization)

6 per. Mov. Avg. (%

CPU usage (as a ratio usage of100) (as CPU DBP01 CPU Utilization)

0.2 6 per. Mov. Avg. (% APP10 CPU Utilization)

0 0 1 2:00 3:00 4:00 5:00 6: 7: 8: 9:00 10 11 1 13:00 14:00 15:00 1 1 1 19 20 21 22 23:00 0:00 :0 :0 0 0 0 2:00 6:00 7:00 8:00 0 0 0 0 0 :0 :0 :0 :0 :0 :0 0 0 0 0 0 0 Time of day (Eastern Time)

Figure 1 - Production server CPU usage There are two areas of peaks. The first peak period starts around 9:00 PM and ends at 3:00 AM. The second peak period starts 3:00 PM and ends around 7:00 PM. Surprisingly, the server load is under 50% for most of the US trading day.

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 11 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Possible zombie processes On APP06 there were 4 instances of the wait process that were allocating 16% of total CPU usage (16% per process split over 4 processors). These processes were initiated on February 11th, 2006. Further investigation is needed to determine why these processes are consuming so many resources while the other wait processes are not. Table 1 shows the top processes on the server APP06.

[ps] USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMAND root 12294 16.1 0 48 44 - A 11-Feb-06 261490:39 wait root 20490 16.1 0 48 44 - A 11-Feb-06 260903:48 wait root 16392 16 0 48 44 - A 11-Feb-06 260090:25 wait root 8196 16 0 48 44 - A 11-Feb-06 259121:55 wait a_slx 2150436 3.7 4 233760 233680 - A 12:49:15 170:55:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcengine a_slx 2224178 3.1 2 123380 123328 - A 12:49:15 146:00:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcengine a_slx 1511664 3 5 296064 295984 - A 12:49:16 137:38:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcengine a_slx 1990868 2.4 4 257536 257456 - A 12:49:15 110:45:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcengine a_slx 282740 2.3 3 163272 163192 - A 12:49:16 104:31:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcengine a_slx 2093268 2.1 2 115688 115608 - A 12:49:15 99:22:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcengine a_slx 532526 2 2 114376 114300 - A 12:49:16 94:39:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcengine a_slx 815148 2 2 140768 140688 - A 12:49:16 90:49:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcengine a_slx 1810524 1.5 2 135460 135380 - A 12:49:16 68:59:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcengine a_slx 528512 0.7 2 95060 94952 - A 12:49:15 31:59:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcengine a_slx 1228806 0 0 9128 9144 - A 12:49:14 1:33 /app/a_slx/slxp/SLXP_I/bin/tfsrvmgr SLXP_I_TSJIPUAPP06

Table 1 - Top processes on APP06

Memory allocation on production servers Figure 2, Figure 3, & Figure 4 show memory usage on each of the servers over a 48 day span. The plot in blue represents physical memory usage. The plot in red represents swap space via hard disk paging files. One interesting observation is that APP06 in Figure 6 appears to run out of physical memory by the 4th day of uptime. The server then falls over to allocating and using swap space.

Figure 2 - Memory usage on APP06

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 12 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Figure 3 - Memory usage on APP10

Figure 4 - Memory usage on DBP01

The infrastructure team has commented on this and remarked that there are no memory limitation or constrains on TSJIPUAPP06 server. The illusion of being out of memory comes from the fact that users do not count file cache usage feature of the Unix kernel. Actual memory usage is 4.125 GB. Starting after week 44, the system administrators also limited the size of the file cache so that it would resize itself to prevent the swap space from being used. A performance assessment was taken after week 44 to reassess the processing times. Data was collected during the week of November 12th in the same way that it was done in Figure 9. Figure 5 shows that processing times have not significantly changed after the adjustment was implemented.

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 13 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

October Performance vs November Performance

15000 100,000

12000 10,000 Projected DML Messages

Oct. SLMU Messages 9000 1,000

Nov. SLMU Messages # of messages # of 6000 100 Processing SLA of 2 minutes Processing Time Processing (seconds)

Oct. SLMU processing time 3000 10

Nov. SLMU processing time

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Figure 5 - Comparing October performance to November

Disk space on production servers Given the number of mounted file systems on each server, only graphs showing disk utilization greater than 80% for an SLX drive are shown. Despite the fact that these drives are mostly full, there is no evidence that short term growth will cause these volumes to become completely full.

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 14 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Figure 6 - /local/app/a_slx on APP06 frequently uses more than 90% of quota

Figure 7 - /app/a_slxftp on the JIP NAS is near capacity

SLX NEW LOAN VOLUMES Table 2 shows intraday statistics on how many new loans are booked during a trading day. This data was compiled from loan statistics captured between January 2006 and June 2006. There are two different types of loans. The first are manual loans which are hand initiated by JPMorgan securities lending traders via the SLXP client. Autoborrow loans are automatically initiated loans that are processed through Equilend. A majority of loans are autoborrow loans which are handled automatically by SLX servers and do not require user intervention. Based on the gathered data, the average manual loan is worth $8,305,877.38 and the value of an autoborrow loan $927,666.75. These numbers are strictly averages and are used for estimating financial loss due to SLX downtime.

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 15 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Loans Booked - Average Hourly Breakdown From: 1-Jan-06 To: 4-Jun-06 Manual Loans Autoborrow loans Total Time - ET # of Loans # of Loans # of Loans

Hourly Avg. 32 69 100 Daily Avg. 757 1,653 2,410 0:00 3 0 3 1:00 5 1 6 2:00 17 1 18 3:00 16 1 16 4:00 14 25 40 5:00 43 57 100 6:00 87 107 194 7:00 72 337 410 8:00 66 316 382 9:00 90 201 291 10:00 78 244 322 11:00 66 147 214 12:00 84 130 215 13:00 75 78 152 14:00 9 0 9 15:00 3 0 3 16:00 1 0 1 17:00 2 2 4 18:00 3 1 4 19:00 3 1 5 20:00 3 1 4 21:00 5 0 5 22:00 5 1 6 23:00 7 0 7 Table 2 - New loans booked in SLX

This data was then plotted against the recorded CPU usage. This plot can be found in Figure 8. An analysis of processor usage during the day shows that new loan bookings correlates very closely to CPU load. Servers APP06 and DBP01 closely follow new loan booking between the hours of 3:00AM and 2:00PM. Fortunately, because the servers have excess CPU capacity between 3:00AM and 3:00PM (nearly 60% of CPU time is idle time) there is plenty of room for growth without risking a performance hit. While it has not been proven that high processor activity on APP06 and DPB01 impact the SLX client response time that traders encounter (they communicate via the less stressed APP10), loan bookings may take slightly longer to process in the background.

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 16 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

New loans compared to CPU usage

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

% APP10 CPU 0.6 1,000 Utilization

New loan bookings

0.4 100 6 per. Mov. Avg. (% APP06 CPU Numberof new loans Utilization)

CPU usageCPU (as a ratio of100) 6 per. Mov. Avg. (% DBP01 CPU Utilization) 0.2 10 6 per. Mov. Avg. (% APP10 CPU Utilization)

0 1 0:00 1: 2:00 3:00 4: 5:00 6:00 7: 8:00 9:00 1 11:00 12: 13: 14:00 15:00 16 17:00 18: 1 20:00 21:00 22:00 23:00 0: 0 00 00 0: 9: 00 0 00 0 00 :00 00 00 0 Time of day (Eastern Time)

Figure 8 - New intraday loans in SLX

SLMU VOLUMES Number of SLMU messages The total number of SLMU messages per day is 485,098*. That number is broken down into four categories: • Message In Credit Snapshot (MICS) • File In Credit Snapshot (FICS) • Message In Loan Snapshot (MILS) • File In Loan Snapshot (FILS)

*according to SLXP at 7:43PM on 11/13/2006

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 17 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Credit Snapshot Loan Snapshot Instance Type: File In 69,534 232,059 Instance Type: Message In 47,339 136,166

Table 3 - Breakdown of SLMU Messages by type

From the data in, further derivations can be made for SLMU messages as it passes through the buffer tables. These are expressed in Table 4.

Formula Value SLXTradeSnapshotBuffer2 FICS x 2 139,068 SLXPositionSnapshotBuffer2 FILS – FICS 162,525 SLXPositionSnapshot FILS – FICS 162,525 SLXTradeSnapshot FICS 69,534

Table 4 - Calculations based on SLMU messages

In Figure 9 the CPU data from the production servers has been overlaid with SLMU messages and processing times over the course of the same sampled week. The main SLMU message peak occurs between 3:00 PM and 5:00 PM. There is a secondary peak of messages that occurs at 10:00 AM.

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 18 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Average SLMU messages & processing time (from 10/15/06 - 10/21/06)

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

SLMU 0.6 1,000 Messages/25000

SLMU message processing time

Processing SLA of 0.4 100 2 minutes Processing Time Time Processing (seconds)

6 per. Mov. Avg. (% APP06 CPU Utilization)

0.2 10 6 per. Mov. Avg.

# of messages and CPU usage (as a ratio of 100) (% DBP01 CPU Utilization)

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Figure 9 - SLMU Messages in SLXP Processing times for messages Processing of SLMU messages should take under 2 minutes (120 seconds) according to the service level agreement (SLA). According to Figure 9, the SLA is currently not being met between the hours of 3:00 PM and 2:00 AM. Processing time is directly related to server CPU load. During this period, both APP06 and DBP01 use more than 50% of the available processing capacity. Although the high processing time occurs outside of the US trading day, this is unacceptable from an international standpoint. Table sizes SLX uses a series of tables to store and process SLMU intraday messages. Table 6 shows each of the tables the SLMU feedback loop uses to process along with their sizes. Most of this data is retained for one day and the tables are purged nightly. The SLMU tables are part of the main SLX table space which has a set quota. Table Q shows how much space is being used by SLX tables.

TABLESPACE TOTAL_MB FREE_MB USED_MB PCT_FREE SLX 24528 8267 16258 33 UA 10731 1804 8926 16 Table 5 - Table space statistics on SLX and UA

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 19 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

OWNER SEGMENT NAME SIZE IN MB SLX_DBA SLX_POSITION_SNAPSHOT 184 SLX_DBA SLX_PSTN_SNPSHT_BFFR2 184 SLX_DBA SLX_SNAPSHOT_BUFFER1 352 SLX_DBA SLX_TRADE_SNAPSHOT 72 SLX_DBA SLX_TRADE_SNAPSHOT_BUFFER2 96 SLX_DBA SLX_UPDATE_LIMIT_USED 30 SLX_DBA SLX_UPDT_BRRWR_CRDT_LN_USD 192 Table 6 - SLMU Feedback loop table sizes in SLX

SLX also gets a start of day file from SLMU that contains all snapshot information. This file is 148MB in size. Anticipated volume increases Given that SLMU should only be used for loan maintenance activities, the volume increase should be very small.

DML VOLUMES Current DML intra-day load Hourly observations were taken based on DML activity. This plot was added to the previous chart and shown below in Figure 10.

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 20 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

SLMU Messages/25000 0.6 1,000 DML Messages/25000

SLMU message processing time 0.4 100 Processing SLA of Processing Time Processing Time (seconds) 2 minutes

6 per. Mov. Avg. (% APP06 CPU Utilization) 0.2 10 6 per. Mov. Avg. # ofand# messagesa ratio usage CPU of100) (as (% DBP01 CPU Utilization)

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Figure 10 - SLMU and anticipated DML messages over a trading day

The alarming part of the DML message projection is the 11AM spike which occurs during the US trading day. A spike will potentially cause the processors to hit their 100% utilization ceiling and then have an impact on regular SLX activity (as highlighted in Figure 8). Using a loan data from SLX, if a 1 minute downtime occurs each hour between 10 and 2 due to the DML feedback loop spike, the potential loss for JPMorgan is $20million in new loans. This prediction is very conservative but does highlight the need for further investigation. A breakdown in loss can be found in Table 7.

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 21 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Loans Booked - Average Hourly Breakdown From:1-Jan-06 To: 4-Jun-06 Manual Loans EquiLend Autoborrow Loans # of Total Value # of Total Loan Value # of Time - ET Loan USD Outage Potential # Potential Value Loan USD Requests Outage Potential # Potential Value Total potential Hrly Av. 32 $ 262,011,497.64 Time - of Lost of Lost Loans 69 $ 63,876,620.12 Expired or Time - of Lost of Lost Loans value of lost Minutes Loans USD Not Minutes Loans USD loans Daily Av. 757 $ 6,288,275,943.25 1,653 $ 1,533,038,882.82 Processed 23:00 - 00:00 3$ 9,776,454.73 0$ 55,334.76 00:00 - 01:00 5$ 21,394,272.78 1$ 1,714,372.40 01:00 - 02:00 17$ 113,351,879.75 1$ 5,441,366.77 02:00 - 03:00 16$ 129,951,300.08 1$ 3,253,248.69 03:00 - 04:00 14$ 83,025,547.10 25$ 37,447,879.91 04:00 - 05:00 43$ 884,015,442.57 57$ 354,717,281.86 05:00 - 06:00 87$ 1,016,299,575.04 107$ 143,778,231.86 06:00 - 07:00 72$ 539,109,513.86 337$ 295,179,792.66 07:00 - 08:00 66$ 536,976,007.01 316$ 229,954,281.45 08:00 - 09:00 90$ 838,829,903.57 201$ 117,932,077.76 09:00 - 10:00 78$ 560,998,062.46 244$ 93,985,724.65 10:00 - 11:00 66$ 388,989,641.01 147$ 53,361,969.91 11:00 - 12:00 84$ 386,406,884.98 1 1$ 4,580,315 130$ 107,176,712.33 1 2$ 1,643,972 $ 6,224,287 12:00 - 13:00 75$ 430,055,601.56 1 1$ 5,765,786 78$ 79,039,907.30 1 1$ 1,016,427 $ 6,782,213 13:00 - 14:00 9$ 171,728,279.02 1 0$ - 0$ 1,849,706.81 1 0$ - $ - 14:00 - 15:00 3$ 75,867,084.33 0$ 2,719,934.33 15:00 - 16:00 1$ 5,162,395.33 0$ 69,085.51 16:00 - 17:00 2$ 4,235,037.57 2$ 1,526,454.28 17:00 - 18:00 3$ 6,633,586.62 1$ 605,583.31 18:00 - 19:00 3$ 10,135,257.59 1$ 926,060.65 19:00 - 20:00 3$ 13,884,355.94 1$ 1,421,584.56 20:00 - 21:00 5$ 14,946,578.59 0$ 234,267.84 21:00 - 22:00 5$ 23,914,216.79 1$ 596,853.58 22:00 - 23:00 7$ 22,589,064.98 0$ 51,169.65

Total Loss : $ 13,006,500

Table 7 - Financial loss due to SLX downtime

Table Sizes Because DML will use a feedback loop system similar to that of SLMU, the expected sizes of the tables should be equal to sizes in Table 5 & Table 6. Increasing the SLX table space for the new DML feedback loop should only add about 1GB of data at most and so it will not push the SLX table space over its quota. The start of day loan file from DML is roughly 220MB in size which again will not have a significant impact on free storage. Anticipated volume increases Over a 6 month observed period, DML activity has grown at a linear rate. As shown in Figure 11, the number of loans grows by 1,990 every 90 days. The total number of loan components grows by 4,774 every 90 days. Following this trend, by July 31, 2007, DML will have 43,223 loans and 82,510 components. That is a 23% increase and a 61% increase from the previous year respectively. A table showing the observed 6 month period can be found in Table 8.

Loans Components 3/31/2006 33087 47215 6/30/2006 35242 51260 9/11/2006 36703 55973 Table 8 - DML loan and component statistics

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 22 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Growth of DML Activity

60000

y = 53.04x - 2E+06 R2 = 0.9885 50000

40000

y = 22.115x - 825078 Loans R2 = 0.9978 Components 30000 Linear (Loans) Linear (Components)

20000 Number of loans & components & of loans Number

10000

0 3/14/2006 4/28/2006 6/12/2006 7/27/2006 9/10/2006 10/25/2006 Date

Figure 11 - Number of DML loans and components

In Figure 11, the formula that is next to each line is a best fit linear regression generated by Microsoft Excel. The value for X is equal to the number of days after 1/0/1900 (Excel’s interpretation of dates as integers) Requirements around timing of DML batch processes DML batch processes will generate more messages for SLX to process. Examples of batch processes include nightly recalculations and the early morning Japanese price batch process. Starting times of these batches should be observed. If they happen when the SLX server load is high, the SLA of 2 minutes will not be met.

RECOMMENDATIONS AND NEXT STEPS

RECOMMENDATIONS The recommendations can be summarized into the following points:

• Review the service level agreements. The proposed service level agreement (guarantee) of a two minute maximum processing time of a transaction is not being met between the hours of 1AM and 3AM and between 3:30PM and 6:00PM. While the 1AM – 3AM time is not applicable because it is in the middle of batch processing, the SLA should be modified for 3:30PM – 6PM.

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 23 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

• Batch processes should be scheduled if possible to not coincide with SLMU peaks.

• There is no immediate need to increase file storage capacity to accommodate the DML feedback loop

NEXT STEPS The next steps going forward can be summarized into the following points:

• Establish contact with a member of the infrastructure team and maintain regular contact for the rest of the project’s duration.

• Identify times of all SLX and DML batch processes

• Determine if there is a need to increase file storage capacity on the volumes /dev/slxp1-slxhom1lv & jipnas1:/vol/jipnas1_app0/tss_slx_prd1 to allow for future growth

• Determine which volume the SLX tables are stored on (we think it’s /dev/dbp01-slxora1)

• Come up with a way to translate DML messages into storage space

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 24 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

REFERENCE DOCUMENTS

Reference Document Location AIX Server description webdoc http://dcfdoc.gis.chase.com

GTI Hobbit, real-time server monitoring utility http://wssunixmonitor.gis.chase.com

GTI Memory and CPU plotting macro CPU&RAM SLXv5.xls

SLMU / SLX feedback Loop documentation (multiple docs) ƒ Worldwide Securities Services PKS / ƒ SLEP - TSS_WSS-INV-SLX/DML Integra - TSS_WSS-INV-SLX/DML Integration / ƒ Research / Information

SLX Production Incident Impact Calculator S:\SECLEND\Production\Production Support\SLX\SLX Production Incident Impact Calculator.xls

DOCUMENT CONTROL

DOCUMENT HISTORY

Version Name Date Description 0.1 Michael Kristan 11/14/2006 Initial Draft 0.2 Michael Kristan 11/20/2006 Revised based on comments from Brian Kenney and from Michael Salamina. 0.3 Michael Kristan 11/22/2006 Revised after Scope/Requirements Review on 11/21. Expanded section on SAN storage and updated recommendation list. 0.4 Michael Kristan 11/30/2006 Revised to include performance assessment from November (post-swapfile fix), cost analysis from Danielle Rumore. 0.5 Michael Kristan 12/3/2006 Added SLMU feedback loop table sizes and financial impact from downtime.

DOCUMENT STORAGE System Name: Worldwide Securities Services PKS Project ID: 71707 Project Name: SLEP - TSS_WSS-INV-SLX/DML Integra - TSS_WSS-INV-SLX/DML Integration

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 25 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

APPENDICES

POWERPOINT SLIDES FOR CAPACITY PLANNING

Capacity Planning An analysis of performance on SLX servers

Michael J Kristan Securities Lending Technology JPMorgan Worldwide Securities Services

Observed data (SLMU)

• An analysis was made on the effect of SLMU messages on SLX servers during the week of October 15th. • A report was run which took an hourly snapshot of the total number of SLMU messages processed during that hour • SLMU hourly data was then aggregated over 7 days (including weekends)

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 26 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Observed data (CPU)

• CPU usage data was gathered off of three servers for that week period (10/15-10/21) – TSJIPUDBP01 –TSJIPUAPP06 –TSJIPUAPP10 (not shown in these graphs) • In order to remove any severe outliers, the median value of the 7 day samples was used for each time interval – Median was used instead of average because the servers went offline for part of the weekend (resulting in an observed CPU usage of 0%) and it would have skewed the average

Observed data (DML)

• DML data was projected onto the graph in order to help determine when processing spikes may occur

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 27 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

SLMU Messages/25000 0.6 1,000 DML Messages/25000

SLMU message processing time 0.4 100 Processing SLA of Processing Time (seconds) Time Processing 2 minutes

6 per. Mov. Avg. (% APP06 CPU Utilization) 0.2 10 6 per. Mov. Avg. # of messages and CPU usage (as a ratio (as andusage 100) of # of a CPU messages (% DBP01 CPU Utilization)

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Response time is directly related to processor usage, not messages

Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

SLMU Messages/25000 0.6 1,000 DML Messages/25000

SLMU message processing time 0.4 100 Processing SLA of Processing Time (seconds) Time Processing 2 minutes

6 per. Mov. Avg. (% APP06 CPU Utilization) 0.2 10 6 per. Mov. Avg. # of messages and CPU usage (as a ratio (as andusage 100) of # of a CPU messages (% DBP01 CPU Utilization)

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 28 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Number of messages however has an influence on processor usage Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

SLMU Messages/25000 0.6 1,000 DML Messages/25000

SLMU message processing time 0.4 100 Processing SLA of Processing Time (seconds) Time Processing 2 minutes

6 per. Mov. Avg. (% APP06 CPU Utilization) 0.2 10 6 per. Mov. Avg. # of messages and CPU usage (as a ratio (as andusage 100) of # of a CPU messages (% DBP01 CPU Utilization)

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Response time is quick when both CPUs are under 50% Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

SLMU Messages/25000 0.6 1,000 DML Messages/25000

SLMU message processing time 0.4 100 Processing SLA of Processing Time (seconds) Time Processing 2 minutes

6 per. Mov. Avg. (% APP06 CPU Utilization) 0.2 10 6 per. Mov. Avg. # of messages and CPU usage (as a ratio (as andusage 100) of # of a CPU messages (% DBP01 CPU Utilization)

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 29 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

SLA is not being met right now

Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

SLMU Messages/25000 0.6 1,000 DML Messages/25000

SLMU message processing time 0.4 100 Processing SLA of Processing Time (seconds) Time Processing 2 minutes

6 per. Mov. Avg. (% APP06 CPU Utilization) 0.2 10 6 per. Mov. Avg. # of messages and CPU usage (as a ratio (as andusage 100) of # of a CPU messages (% DBP01 CPU Utilization)

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Risk: DML messages are high from 11AM – 1PM

Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

SLMU Messages/25000 0.6 1,000 DML Messages/25000

SLMU message processing time 0.4 100 Processing SLA of Processing Time (seconds) Time Processing 2 minutes

6 per. Mov. Avg. (% APP06 CPU Utilization) 0.2 10 6 per. Mov. Avg. # of messages and CPU usage (as a ratio (as andusage 100) of # of a CPU messages (% DBP01 CPU Utilization)

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 30 of 31 For Internal Use Only - Do Not Duplicate

Worldwide Securities Services – DML to SLX Integration - Project Code: <71707>

Review of SLX Server Performance

Good news: SLMU peak occurs when DML messages are low Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)

1 100,000

% APP06 CPU 0.8 10,000 Utilization

% DBP01 CPU Utilization

SLMU Messages/25000 0.6 1,000 DML Messages/25000

SLMU message processing time 0.4 100 Processing SLA of Processing Time (seconds) Time Processing 2 minutes

6 per. Mov. Avg. (% APP06 CPU Utilization) 0.2 10 6 per. Mov. Avg. # of messages and CPU usage (as a ratio (as andusage 100) of # of a CPU messages (% DBP01 CPU Utilization)

0 1 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 Time of day (Eastern Time)

Next steps

• Take a closer look at memory usage and disk usage on servers • Take a larger sample (2-3 weeks instead of 1) • Identify times of all SLX batch processes • Come up with a formula to model CPU usage based on messages

Date Created: 14-Nov-06 © 2006 J. P. Morgan Chase Bank, N.A. Owner: Securities Lending Date Revised: 19-Dec-06 All Rights Reserved. Version: 0.5 Last Saved By: Michael Kristan Confidential And Proprietary Page: 31 of 31 For Internal Use Only - Do Not Duplicate