MTA Office of the Inspector General GOVERNANCE ASSESSMENT of MTA CAPITAL PROGRAM MEGA PROJECTS OVERVIEW in the Early 1980'S, T

MTA/OIG Report #2010-04 November 2010

GOVERNANCE ASSESSMENT OF MTA CAPITAL PROGRAM MEGA PROJECTS

Barry L. Kluger MTA Inspector General State of New York

OVERVIEW

In the early 1980’s, the Metropolitan Transportation Authority (MTA) developed a capital program to rebuild the subway and commuter rail network that had suffered years of neglect and decay. To ensure that capital program dollars were spent effectively, New York State law charged the MTA Board of Directors (Board) with responsibility for overseeing this program. Specifically, the law created a Capital Program Oversight Committee (CPOC or the Committee) that was to consist of Board members.

To help CPOC members understand the often technical subject matter of capital projects, the law also provided that CPOC would be assisted by an Independent Engineering Consultant (IEC).1 The IEC advises CPOC on issues affecting capital projects through reports that it provides when the Committee meets on a monthly basis. An MTA Headquarters (MTAHQ) department, the Office of Construction Oversight (OCO), is in charge of directly managing the IEC, which is presently retained under a $7 million annual contract. Together, these groups and their responsibilities form the governance structure for overseeing the capital program.

Since 1982, the purpose of the MTA’s capital program has been largely to restore the MTA’s railcars, buses, and other critical infrastructure, and also to maintain those assets in a state of good repair. More recently, however, the capital program has also included several massive system expansion projects known as “mega-projects.” The mega-projects currently under construction consist of: Second Avenue Subway (SAS); East Side Access (ESA); Fulton Street Transit Center (FSTC); and 7 Line West Extension (7W). Because of the size and complexity of these projects, the MTA created a subsidiary agency called MTA Capital Construction (MTACC) to manage them. The IEC regularly reports to CPOC on the status of MTACC- managed mega-projects, as it does for major “core” capital program projects managed by the other MTA agencies.

Despite the foregoing, MTACC-managed mega-projects have experienced major budget overruns and schedule delays. Since their inception, three of these mega-projects – SAS, ESA, and FSTC – have experienced cost increases totaling over $1.93 billion and their eventual completions have been delayed by 2½ to 5 years.

1 Specifically, New York Public Authorities Law Title 11, Section 1263(4)(b) states that the CPOC shall use an “independent transit engineering firm.” However, this firm is commonly (and contractually) referred to as the “Independent Engineering Consultant.”

MTA Office of the Inspector General 1

MTA/OIG Report #2010-04 November 2010

Findings and Recommendations

The failure to complete mega-projects on time and within budget has raised serious concerns about how MTACC has managed these projects and indicates a need for more effective oversight from MTA. To help address these concerns, the Office of the MTA Inspector General (OIG) undertook a comprehensive review, including interviews of MTA Board members and key personnel in the Office of Construction Oversight, the Independent Engineering Consultant IEC and MTACC, as well as examination of reports, records, and contracts. We found that although the governance structure now in place is essentially sound, its execution can be improved. While the chairman and committee members have increased their scrutiny over mega-projects through the oversight committee, it is also clear that the challenge is enormous and MTA must continually seek to more effectively monitor and oversee these-projects. The OIG shared our preliminary findings and recommendations with the MTA Chairman/CEO Jay Walder and members of CPOC.

In July 2010, at the invitation of Chairman Walder, the Inspector General publicly presented a number of the issues contained in this report directly to the members of CPOC. 2 At that meeting he stressed the importance of moving toward a risk-based approach to oversight and of ensuring that MTACC and the engineering consultants provide the Committee members with the high- level analyses necessary for the members to perform that function.

Thereafter, at the September CPOC meeting, Linda Kleinbaum, Deputy Executive Director/Administration, shared with Committee members a new proposal for use of the IEC. She explained that over time, the role of the IEC had changed. Where now the IEC was used more and more to monitor capital programs, originally the consultants assisted CPOC in evaluating how projects were managed and regarding the wisdom of various investment decisions. Her proposal was to return to using consultants in a broader capacity to assess the implementation of the capital program. The proposal was developed by MTAHQ staff to be responsive to the issues and concerns raised by CPOC members and the Chair, as well as to those raised by the Inspector General in July.

The OIG’s findings and recommendations cover four general areas:

1. The respective roles and responsibilities of the IEC and MTACC in the oversight process have not always been clear. As a consequence, the IEC and MTACC have come into conflict. Such conflict has sometimes required MTAHQ intervention, thus hampering efficiency in the oversight process and slowing the acceptance of critical findings and the implementation of appropriate recommendations.

The OIG recommended that as the agency responsible for building the mega-projects, MTACC should have the primary responsibility for reporting on their status. For its part, the

2 This presentation was broadcast and may be viewed on the MTA website at Board Materials » Live Webcasts » Archived Meetings » 7/26 CPOC Meeting.

MTA Office of the Inspector General 2

MTA/OIG Report #2010-04 November 2010

IEC should comment on MTACC’s reports and alert CPOC to any issues, whether raised in the reports or not, that could cause the mega-projects to experience significant cost increases, serious schedule delays, and/or reductions in project scope.

MTA agreed and will now require MTACC to report on the status of the mega-projects. The IEC will evaluate the MTACC report and provide supplemental information as necessary. If issues are raised that warrant additional monitoring, the IEC will conduct targeted, in-depth analysis.

2. The IEC and MTACC have not always presented project reports to CPOC in a clear, concise, and meaningful fashion. Reports lack elements such as executive summaries and impact assessments that would improve their usefulness. Reporting is often highly detailed, technical, and may dwell on matters that should not rise to Board-level scrutiny. The usefulness of reports is also undermined by the combined effect of MTACC not fully addressing IEC findings and recommendations and the IEC not always tracking open items so that major issues can continue to receive needed attention.

The OIG recommended that reports to CPOC be improved by summaries and impact assessments, as well as by tracking and follow up of key recommendations. Here again the OIG noted that the IEC should both examine and look beyond issues raised in MTACC’s project reporting, identifying key issues not included in those reports. In all cases, the IEC should verify actions planned or taken by MTACC to address issues raised by MTACC and/or the IEC.

In response to the Inspector General’s concerns, OCO asked MTACC to develop a process and format for reporting on the mega-projects to CPOC. OCO requested that the process provide the IEC with an opportunity to review and comment upon MTACC’s reports and that the reports contain charts showing cost and schedule trends. In addition, MTAHQ directed that each MTACC report now discuss some key issues identified in the risk assessments and, importantly, also propose mitigation strategies, befitting MTACC’s accountability for managing these critical issues. Recommendations will now be tracked to ensure satisfactory resolution.

3. CPOC has not set expectations and priorities for the IEC. The Committee does not formulate a work plan that is informed by an assessment of what issues carry the greatest risk to the MTA. Presently, the large majority of the IEC’s time and resources is devoted to project monitoring and reporting across all areas, rather than risk based monitoring and analyses that would add value by identifying the areas of greatest risk within the mega-projects as well as cross-project issues that MTACC should address.

The OIG recommended that the IEC prepare an annual risk-based work plan that is reviewed with and approved by the Committee. The IEC's focus should shift from project reporting to higher-level analyses that will help inform CPOC about ways to improve project oversight. OIG also suggested MTAHQ consider reallocating its oversight resources to place more emphasis on the mega-projects (as opposed to the core capital program).

MTA Office of the Inspector General 3

MTA/OIG Report #2010-04 November 2010

Based on similar concerns raised by the Chairman, OCO has directed the IEC to conduct risk assessments, and monitor the projects based on risk. OCO is to present CPOC with a new proposed work plan in October, and seek the committee’s approval of that plan. As noted above, the MTA has proposed to CPOC a return to using consulting resources in a strategic capacity. Moreover, the MTA Chairman has directed OCO to revisit the IEC’s resource allocation.

4. OCO and CPOC do not evaluate or set formal criteria for IEC performance. At present, there exists no formal performance assessment or criteria to determine whether and how much the IEC is adding value to the oversight and implementation of the capital program.

The OIG recommended that OCO and CPOC formally evaluate the IEC’s performance. As part of this evaluation they should determine whether the IEC has satisfactorily implemented its work plan, assess the IEC’s contribution to improving oversight of the capital program, and generally verify that the IEC has discharged its contractual obligations. OCO should meet annually with the Chairman and the Chairs of the operations committees of NYC Transit (or Transit), Long Island Rail Road (LIRR), Metro North Railroad (MNR), and Bridges and Tunnels (B&T), who are also members of CPOC, to obtain feedback on the IEC’s performance. For its part, the IEC should provide semi-annual status reports on its progress in implementing its work plan, its accomplishments to date, and summaries of the major capital program issues with the status of corrective actions.

MTA HQ and OCO accepted these recommendations and plan to implement an evaluation process for the IEC.

The OIG supports the changes that the MTA has already implemented, and note the active role taken by the Chairman in his dual role as chair of the Board and of CPOC. We suggest that the MTA further consider the findings and recommendations of this report to help bring greater clarity, less conflict, and more effectiveness to the oversight process.

MTA Office of the Inspector General 4

MTA/OIG Report #2010-04 November 2010

BACKGROUND

Faced with the prospect of implementing several concurrent multi-billion dollar projects, the MTA created Capital Construction to be its agency exclusively responsible for the construction of mega-projects. MTACC is in the process of completing one such project and presently manages four others with budgets totaling $15.32 billion. Three of these four projects – SAS, ESA, and FSTC – have increased a total of $1.93 billion in cost and their eventual completions have been delayed by 2 ½ to 5 years. Such cost overruns and delays have raised public concerns. Presently, the Office of the MTA Inspector General is engaged in an across-the-board review examining the causes of mega-projects being over budget and behind schedule, of which this governance assessment is the first part. Upcoming work will explore other potential causes.

To illustrate the reasoning behind this approach, we use as an example a well publicized and disturbing series of delays in finalizing the still elusive East Side Access project completion date.

MTACC maintains a schedule for the project delivery, and two oversight entities (the MTA’s IEC and the Federal Transit Administration [FTA] Project Management Oversight Contractor [PMOC]) provide independent assessments of that schedule. When the relevant funding agreement with the federal government was signed in December 2006, after years of planning, the funding agreement stated, and the MTACC, the IEC and the FTA all concurred, that East Side Access would be completed by December 31, 2013.

About a year later, in November 2007, the same three parties agreed that the completion date needed to be moved out to the end of April 2014. However, in April 2009, the completion date assessment changed again, but the parties were no longer in agreement. To begin, the MTACC had added approximately 10 more months to its estimated completion date extending it to February 28, 2015. The FTA PMOC expressed its view that the completion date was likely to be somewhat longer (August 31, 2015). For its part, the IEC did not commit to a date, indicating in its monthly report only that the ESA schedule was “under pressure” of delays beyond the MTACC’s assessment.

In June 2009, MTACC revised its expected completion date to the end of September 2016. The FTA PMOC reported at that time that the ESA completion date was “under review.” The IEC, in its monthly report for December 2009, concurred with the MTACC projected date.

Thereafter, in a letter dated June 18, 2010, to the Chairman of the United States Senate Committee on Banking, Housing, and Urban Affairs, the FTA Administrator, Peter Rogoff, noted that while the MTA maintained that it could complete ESA “sooner and at lower costs than the FTA currently estimates,” the FTA determined, based on its own risk analyses, to amend the funding agreement commitment date for ESA to April 2018.

While there are many factors that contribute to the constant shifting of the project completion date and the divergence of the forecasts, the need for and importance of strong oversight and governance to address these issues is critical.

MTA Office of the Inspector General 5

MTA/OIG Report #2010-04 November 2010

Concepts of Governance and Oversight

For the purpose of this review, the OIG defines governance and oversight in terms of structure and execution. Governance consists of the overall structure put in place by the law of this state, MTA Board charters, and MTA executive management to facilitate effective oversight of MTA capital projects. Oversight comprises processes and activities that, when effectively executed by entities operating under the governance structure, support the MTA Board’s monitoring of the progress of major capital projects.

Governance Structure

New York State law assigns the MTA Board a key role in the oversight of core capital and mega- projects. In particular, it establishes a Capital Program Oversight Committee, a subdivision of the MTA Board whose role is to develop an understanding of all aspects of the capital program and which is responsible for monitoring, reviewing and challenging management’s performance. By creating CPOC and establishing a role for an Independent Engineering Consultant, the law partially codifies capital oversight governance. Governance is also established via Board committee charters and informal practices that develop based upon the preferences of committee members.

The MTA’s governance structure to monitor mega-project performance involves the organizational units that play a part in the implementation of the capital program, including key members of the Board and executive management. Highlights of the governance structure include:

• The Capital Program Oversight Committee is responsible for monitoring all aspects of the capital program. The Chairman/CEO of the MTA serves as the Chair of CPOC and the Committee’s members include the chairpersons of various operations committees (i.e. those for NYC Transit, LIRR/MNR, and B&T).

• MTA Capital Construction, which builds each mega-project for its particular “owner,” one or another of the MTA operating agencies, reports on the individual mega-project to the appropriate agency operations committee.

• The Independent Engineering Consultant has been engaged by CPOC to assist with its monitoring and oversight responsibilities. The IEC assesses project status based upon information it gathers from MTACC and the operating agencies through communication with staff members and attendance at project meetings, as well as by review of project design and construction budgets, schedules, and other documents. As to the mega- projects, the IEC reports its assessments to CPOC on a quarterly basis.3 One particularly

3 The IEC progress reports for CPOC committee meetings on a monthly or bimonthly basis for the various aspects of the MTA capital program that do not involve the mega-projects.

MTA Office of the Inspector General 6

MTA/OIG Report #2010-04 November 2010

important role of the IEC is to make recommendations on how to keep these projects within budget and on schedule.

• The MTAHQ Office of Construction Oversight directly manages the IEC.

Aside from the foregoing, oversight is also provided by the Federal Transit Administration on the ESA, SAS, and FSTC mega-projects.

Given the relationships defined within it, the MTA’s capital program governance structure appears to be sound. However, OIG finds that there is room for improvement in how the MTA Board and executive management perform oversight under this structure. The Institute of Internal Auditors’4 guiding principles of governance assessment, as generally accepted among many organizations, provide an example of linking structure to execution by evaluating performance in key areas such as:

• Ensuring effective performance, management and accountability. • Communicating risk and control information throughout the organization. • Coordinating activities and communication of information.

Therefore, this first stage of OIG’s review has focused on activities that are performed under the governance structure and facilitate capital program oversight through: definition of roles and responsibilities; reporting to CPOC; establishment of priorities and expectations; and evaluating engineering consultant performance.

4 The Institute of Internal Auditors (IIA) is an international professional association that provides education and certification, conducts research, and develops technical guidance for auditors in all types of organizations. Source: http://www.theiia.org.

MTA Office of the Inspector General 7

MTA/OIG Report #2010-04 November 2010

ROLES AND RESPONSIBILITIES

A lack of clearly defined roles and responsibilities under the MTA’s oversight structure has given rise to unproductive conflict, impeded the flow of information, and potentially slowed the implementation of critical IEC findings and recommendations.

Conflict Hampers IEC-MTACC Working Relationship

All parties agree that chronic, unresolved conflicts, especially between the IEC and MTACC, sometimes requiring OCO or other MTAHQ intervention, diminish the effectiveness of capital project oversight and implementation. Although discussions with the IEC, OCO, and MTACC have included frequent references to such conflicts, the OIG has not sought to assign blame or determine who, if anyone, is in the “right.” In fact, the presence of conflict itself is to be expected when multiple perspectives are brought to bear on an issue, as well as in cases where one party acts in an oversight role with respect to another. We note, however, that the essential conflict, which may be broadly characterized as disagreement over the role of the IEC, reduces the efficiency of mega-project oversight and reporting activities, and could erase any benefit MTACC may gain from the IEC’s observations and recommendations.

Based upon interviews with OCO, IEC, and MTACC staff, OIG finds that these conflicts are generally over the degree of IEC involvement in project management processes, the sharing of information, and the handling of IEC recommendations.

Clarify IEC Role in Project Management

In its role as a technical consultant to the Committee, the IEC may perform certain reporting, engineering, and other project management functions similar to those performed by MTACC. Since the mega-projects require greater scrutiny, not less, these overlapping functions can actually be beneficial to all of the parties. However, the roles must be assigned carefully to maximize effectiveness and efficiency and avoid wasteful duplication of effort.

The IEC consultants should not duplicate MTACC project management activities. MTACC asserts that the IEC is acting as a “shadow project manager” and OIG finds this to be true in the case of project status reporting. The IEC has engaged in project status reporting that is ordinarily the responsibility of those actually managing the project. Much of this project reporting has entailed collecting budget and schedule information from project managers and passing it on to CPOC. On the other hand, OIG does not find that the IEC’s mere presence at meetings and other project activities constitutes “shadow project management.” The IEC needs access to all project information in order to identify key concerns that should be reported to the MTA Board.

In particular, the IEC needs a more clearly defined role regarding risk assessment. The IEC’s contract states that for large complex projects it is responsible for facilitating risk assessment workshops prior to design completion. The IEC is further expected to document the outcomes of such efforts. However, with regard to the mega-projects, the IEC’s involvement in risk

MTA Office of the Inspector General 8

MTA/OIG Report #2010-04 November 2010

assessments has not consistently encompassed the role specified in the IEC’s contract, despite the fact that the MTA is paying the IEC to perform this function.

For example, while MTACC’s Management and Reporting Plan of September 21, 2009 stated that MTACC would be a partner with the IEC on formal risk assessments and the IEC has facilitated some mega-project risk assessment workshops, the IEC has had limited participation in others. Further, the SAS and ESA projects have used consultants to facilitate risk assessments at which the IEC engineers were present only as observers. Initially, the IEC was prevented even from providing input to the 7W risk assessment. We note, though, that the MTA has made progress in clarifying roles and responsibilities with respect to risk assessments.

At the OIG’s suggestion, OCO/IEC and all MTA agencies now agree to a Risk Assessment Guideline designed to mitigate this area of conflict. In this Guideline, MTAHQ establishes the IEC’s key role in all risk assessments and requires that MTA agencies provide OCO and the IEC with access to all information necessary to properly perform analyses that supplement or contribute to such assessments. At the same time, the Guideline makes clear that MTACC and the other agencies must also be active in developing assessments that are meaningful. OIG believes that this kind of mutually agreed-upon Guideline could serve as the model for an overall framework to clarify roles and responsibilities and thereby gain greater value from the IEC’s services.

Provide IEC with Timely Access to Information

IEC managers state that the IEC has experienced difficulty in obtaining timely information from MTACC about the mega-projects. For example, the IEC undertook an analysis of Second Avenue Subway’s proposed coordination of major tunneling and station contracts (Contracts 1 and 4B). In its report dated April 20, 2010, the IEC noted:

Although [the IEC has] provided a preliminary review of this revised strategy; this analysis is constrained by the fact that we have not received updated and detailed cost estimates and schedules fully disclosing the impact of the changes contemplated.

Without this information, the IEC was unable to assess the assumptions underlying MTACC’s approach. Consequently, the IEC could not provide recommendations on what additional alternatives MTACC might have considered or alerts about potential risks inherent in MTACC’s proposed plan. One such risk, a slower-than-expected underground tunneling rate, was subsequently acknowledged by MTACC.

In another instance, as the 7W completed its design of work to be performed at two project sites, the IEC managers stated they requested that the project provide updated construction cost estimates for IEC review and analysis. After several requests, the IEC received the estimates but only 5 days before MTACC advertised the contract for bid. The IEC found this timeframe too short for it to provide any meaningful analysis.

MTA Office of the Inspector General 9

MTA/OIG Report #2010-04 November 2010

It may be that the IEC’s perception of its duty to report on all project issues, no matter how small, contributes to MTACC’s resistance to providing information to the IEC. To the extent this is the case, the OIG believes that its recommendations to improve project reporting, provided in the subsequent section, should help address this concern.

Establish Approach for Addressing IEC Findings and Recommendations

The IEC’s findings and recommendations represent opportunities to bring outside perspectives to bear on problems arising in the mega-projects. This is not to say that MTACC must accept IEC findings or that MTACC should automatically implement all IEC recommendations. Indeed, our analyses of various instances where MTACC has not adopted IEC findings and recommendations reveal legitimate differences of opinion. Unfortunately, the parties do not always work out their differences.

In our view, the parties need an agreed-upon and structured approach by which MTACC addresses the IEC’s findings, considers its recommendations, and implements them as warranted. Without this approach, MTACC could delay or even miss opportunities to improve mega-project delivery, potentially resulting in additional cost increases and schedule delays.

We have identified two aspects of mega-project delivery that highlight the need for a structured approach: self-competition in procurement and the testing and commissioning process.

Self-Competition

When implementing capital projects, MTA agencies generally attempt to keep costs down through competitive procurements. By soliciting multiple bids from various contractors, MTA agencies seek to foster competition between contractors in order to drive down prices. However, in 2007, MTACC solicited bids on three contracts for tunneling and other underground work involving three different mega-projects within a short period of time. Because this work could be done only by a small number of highly specialized contractors, MTACC essentially competed against itself.

In its May 2007 report to CPOC on ESA, the IEC expressed concern that the simultaneous procurement of these contracts would, among other problems, “strain…the MTA’s negotiating resources and have a negative economic impact on all three contracts.” MTACC’s response to this concern was not conveyed to CPOC, and MTACC proceeded to issue the procurements within short periods of one another. Two of these procurements each garnered a proposal from only one bidder. Furthermore, both of those proposals were for amounts substantially greater than what MTACC had estimated.

Despite the IEC’s warnings, MTACC again appears to be on the verge of self-competition, this time in the procurement of contractors to install signal and communication systems toward the end of the mega-projects. For its part, the IEC has again warned MTACC that the agency could encounter similar self-competition problems and resulting cost increases, if it does not coordinate or stagger the procurements of these contractors. MTACC acknowledged this in its September

MTA Office of the Inspector General 10

MTA/OIG Report #2010-04 November 2010

2009 Management and Reporting Plan that it presented to the MTA Board. In this plan, MTACC stated that it would enhance competition and develop lessons learned.

MTACC has informed us that it has formed a working group specifically to address the issue of self-competition in upcoming systems contract procurements. The OIG recognizes that the issue is complex, and that all present options for addressing systems contract procurements involve some cost. However, we also note that the IEC first warned about potential competition between similar MTACC tunneling procurements in May 2007, and that while MTACC acknowledged the same problem for systems procurements in 2009, MTACC did not develop a systematic approach to mitigate self-competition in systems projects procurements until June 2010.

Testing and Commissioning

Testing and commissioning comprise the activities required at the end of a major project that lead to final acceptance by the operating agency. In the past, delays have developed because MTACC did not have a thorough testing and commissioning plan in place, as evidenced recently at South Ferry Station, which experienced costly delays when this deficiency was belatedly addressed.

Based on the experience with South Ferry, the IEC recommended that all the remaining mega- projects develop thorough testing and commissioning plans. As a result, each of the MTACC project management teams developed such a plan and shared it with the IEC. Nevertheless, the IEC has remained critical of the testing and commissioning plans as insufficiently developed or not allowing adequate time for the necessary activities.5

MTACC disagrees with the IEC’s complaint, saying that its testing and commissioning plans are at a level commensurate with the projects’ development. One project manager questioned why the IEC wants to see a fully developed commissioning plan when the activities in question are still six years away. He went on to note that the operating agency is satisfied with the amount of time allotted for testing and commissioning activities.

In our view, such disagreements underscore the need for a clear and effective process under which MTAHQ, OCO, MTACC, and the IEC evaluate alternative approaches to an issue, and either reach consensus on a course of action or agree to disagree and present both sides to CPOC. Under the present circumstances, there is no way for CPOC members to know with confidence that a way forward has been identified and is being pursued. Furthermore, constructive discussion between the parties increases efficiency. For example, if the MTA approves MTACC’s present approach to testing and commissioning, the IEC can focus on other concerns. On the other hand, the MTA may find that the IEC’s concerns about systems contracts and testing and commissioning are justified and warrant additional focus and consulting resources.

5 Indeed, in the case of ESA, MTACC’s belated development of a testing and commissioning plan, in response to the IEC’s recommendation, revealed to the IEC that MTACC needed to add $50 million to the project cost to cover testing and commissioning. ESA accepted this cost during the July 2009 project budget revision.

MTA Office of the Inspector General 11

MTA/OIG Report #2010-04 November 2010

More Clarity and Structure is Needed

As clearly evidenced above, CPOC needs to clarify the MTACC’s roles and responsibilities with respect to the IEC’s recommendations and provide clear guidance regarding their implementation. At present, there is no formal mechanism to foster a thoughtful discussion of differences that arise from significant IEC findings and recommendations, and for setting expectations going forward. No entity is clearly responsible for assessing whether recommendations are valid, and should be implemented, or whether existing approaches are adequate. Where MTACC and IEC differ on solutions to mega-project issues, there is no commonly accepted structure to either achieve consensus or to “agree to disagree,” with MTACC management ultimately accountable for the outcome.

MTA Office of the Inspector General 12

MTA/OIG Report #2010-04 November 2010

RECOMMENDATIONS

1. OCO should expand upon the initiative that developed the Risk Assessment Guideline by explicitly defining the specific roles and responsibilities of MTACC and the IEC in the mega-project oversight process and through MTAHQ, provide mechanisms to constructively address conflicts. 2. As the agency responsible for building the mega-projects within the allocated budgets and on schedule, MTACC should have the primary responsibility for reporting on their status. The IEC should evaluate and comment upon MTACC’s reporting and advise CPOC regarding any additional issues that could cause the mega-projects to experience significant cost increases, serious (i.e. “critical path”) schedule delays, and/or reductions in the scope of the project. 3. MTAHQ should ensure that MTACC is providing the IEC with the information and materials it needs in a complete and timely fashion, so that the IEC can effectively perform its oversight and consulting functions.

MTAHQ accepted these recommendations. As of October 2010, MTACC will have primary responsibility for reporting on the status of mega-projects to CPOC. The IEC will verify the information and present additional points or issues, as necessary.

MTA Office of the Inspector General 13

MTA/OIG Report #2010-04 November 2010

REPORTING TO CPOC

Both MTACC and the IEC produce quarterly reports on the mega-projects for CPOC. When the OIG initiated this review, MTACC reported progress and highlights to the previously-existing Capital Construction, Planning, and Real Estate Committee, but this reporting avoided any discussion of project problems. In contrast, the IEC’s reports to CPOC have focused on these problems, but often in excessive and technical detail.

The MTA has adjusted reporting requirements and formats since the start of 2010. For a variety of reasons, however, the reporting has not adequately enhanced CPOC’s ability to oversee the mega-projects so that they stay within budget and on schedule. To accomplish that goal, we believe that reports should include elements essential to policy makers, including:

• Executive Summaries CPOC members would benefit from reports prefaced by executive summaries. These would serve to prioritize and highlight issues entailing high risk, budget/schedule criticality and other key report content as determined by the needs of the Committee. • Assessment Tools The IEC reports generally do not contain the elements that make an oversight tool useful such as clear impact statements, deadlines for actions, alternatives to consider, and an assessment of whether or not MTACC resolves issues in a satisfactory manner. These tools should facilitate executive-level review. • Flag Priority Issues and Track Open Items There is currently no systematic process by which issues and recommendations are tracked or MTACC’s implementation of those recommendations is verified. Tracking and reporting on the status of recommendations will enhance the effectiveness and efficiency of Board review, and help ensure that timely and appropriate action is taken.

Status of IEC Findings and Recommendations Remains Unclear

MTACC does not currently respond to all IEC findings and recommendations, which undermines the IEC’s ability to track issues, and to monitor and report progress to CPOC. By not responding, MTACC also leaves Board members in the dark as to whether MTACC agrees or disagrees with the IEC and what, if any, action MTACC intends to take. One example of this is the IEC’s recommendation to initiate a lessons-learned exercise regarding the SAS project’s “Cost to Cure.”

MTACC experienced budget and schedule overruns resulting from conflicts with building owners over work in private buildings, which had to be performed at the MTA’s expense in order to construct the SAS 86th Street station. This unanticipated additional project expenditure is called “Cost to Cure.” In its October 2009 report, the IEC recommended that MTACC advance its contingency plan for mitigating potential delays to Contract 5A and initiate a

MTA Office of the Inspector General 14

MTA/OIG Report #2010-04 November 2010

Lessons-Learned study on the Cost to Cure process to prevent the recurrence of this problem in future contracts. MTACC did not reply to this recommendation.

Thereafter, in December 2009, the IEC repeated these recommendations that MTACC and the MTA Real Estate department initiate a Lessons-Learned exercise on the Cost to Cure process. In fact, MTACC had already initiated the Cost To Cure Lessons-Learned exercise, but nowhere communicated this fact in its response to the IEC’s recommendation.

To make the process more effective and efficient, MTACC should respond to all IEC findings and recommendations in its reporting to the Board.

Focus of Reporting Needs Improvement

At the January 2010 CPOC meeting, the Chairman expressed dissatisfaction that the IEC does not raise issues in a manner that informs and engages CPOC members, a practice that would enable these members to more meaningfully assess risk and take appropriate action. Too much in the reports is written about subjects or details that do not require the Board’s attention.

For example, the March 2010 IEC report to CPOC regarding the Fulton Street Transit Center explains how two aspects of a contract were slightly ahead of the “baseline schedule” but “one to two months behind the intermediate customer benefit milestones in 2011.” Although a paragraph is written about these delays, the IEC concludes that the “delayed access is currently not a critical issue to the contract completion.” Another paragraph in the same report on the FSTC is used to raise the IEC’s concerns that a consultant/Integration Manager has not yet been hired. While this may be an appropriate concern, a vacancy in a lower level management position is not usually a Board matter.

Rather than statements like “In order to mitigate risk, the schedule was reduced to a list of activities and durations that is included in the bid documents’ appendix,” as appeared in the March 2010 report, it is critically important that CPOC members be provided with meaningful risk analyses, presented in clear and straightforward terms. In our view, the time taken to compile and write material that does not further the mission of the Board is a lost opportunity to focus on critical issues, risk analyses, findings, and recommendations necessary to inform the judgment and action of the Committee to the betterment of the capital program.

Annual Report is Poorly Organized and Not Risk-Based

The IEC contract requires that at least annually it should report to CPOC and/or OCO on all (1) recommendations it made, including outcomes and (2) instances where its activities have contributed to improving capital program performance in the area of cost, project duration, quality, safety and management. While the IEC produces a document that constitutes its contractually required annual report, this document, in its current form, is not particularly useful to CPOC for reasons including the following:

MTA Office of the Inspector General 15

MTA/OIG Report #2010-04 November 2010

• The information is presented based on the date that it was reported rather than the importance of the subject matter.

• Although the document might be 40 pages or more in length, it does not prioritize issues and many of the critical findings and recommendations are buried in the report.

• Although the report is structured to show the Agency Comment and Action for each finding and recommendation presented, the Agency Comment section was blank for much of the report and where comments were listed they often did not address all components of the findings or recommendations at hand.

• The document did not highlight any instance where activities have contributed to improving capital program performance in the area of cost, project duration, quality, safety and management.

MTA Office of the Inspector General 16

MTA/OIG Report #2010-04 November 2010

RECOMMENDATIONS

4. Reports to CPOC regarding the capital program should (a) include executive summaries that highlight key issues to aid Board members’ review; (b) advise CPOC of those issues affecting the major project milestones; and (c) explain in simple and straightforward terms the potential impact that they will have on the project’s budget and schedule.

5. The IEC should evaluate and comment upon MTACC’s reports, noting whether the IEC concurs or holds a different view. Where the IEC holds a different view, it should explain its reason(s), citing industry best practices as applicable. Furthermore, the IEC should note any additional issues not included in MTACC reporting that could affect the major project milestones, explaining their potential impacts in simple and straightforward terms.

6. CPOC should require that both MTACC and the IEC track the status of issues that they report and schedule regular follow-up reporting on those issues. This tracking should encompass the actions taken by MTACC, including implementation of IEC recommendations, if any, and MTACC and the IEC should provide CPOC with regular progress reports.

7. The IEC should continue to identify and track issues not elevated to CPOC’s attention, as appropriate, but should limit its reporting of these to OCO rather than to CPOC.

MTAHQ accepted these recommendations and moved aggressively to redesign the report format and presentation that goes to CPOC for the mega-projects beginning in October 2010.

MTA Office of the Inspector General 17

MTA/OIG Report #2010-04 November 2010

CPOC PRIORITIES AND EXPECTATIONS

OCO and the IEC would benefit from additional CPOC guidance to help focus their efforts, energies, and resources on activities that will best support CPOC in keeping the capital projects within budget and on schedule. Two ways in which CPOC could more clearly evince its priorities and expectations are by requiring a risk-based plan from OCO/IEC and reallocating IEC resources.

OCO and the IEC Need a Better Work Plan

OCO officials prepare a schedule detailing what information and reports will be presented to CPOC at each monthly meeting. This schedule then sets the “work plan” for the IEC over the coming year. We find that CPOC would benefit if OCO also directed and optimized the efforts of the IEC in targeting areas where the MTA is most vulnerable to cost overruns or schedule delays. The work plan needs to indicate how OCO intends to use the IEC’s resources to assess capital program delivery, develop recommendations that address areas of concern for the Committee, or otherwise enhance CPOC members’ ability to oversee the mega-projects.

The MTA Should Shift Consultant Resources

According to contract, the IEC’s project monitoring responsibility includes the preparation of reports to CPOC that “address [each mega-project’s] overall status with respect to budget, schedule, quality, and safety achieved, adherence to original scope, and effectiveness of project management controls. Areas of deficiency should be highlighted with appropriate recommendations for remedial action.” The IEC contract further states that: “A major responsibility of the IEC is project monitoring which is expected to represent approximately 75% of billable time.”

Based upon its contract, the IEC is to devote the remaining 25% of its time to scoping, design, and programmatic reviews, as well as to risk assessments. These reviews are to be focused upon “critical capital program system areas and/or capital program management activities, which may impact all agencies.” The contract conveys the MTA’s expectation that:

The IEC will draw upon information gained through its review and oversight of projects to identify system problem areas where additional investment of IEC effort might be expected to add value to the MTA capital program.

The reality is, though, that in 2009 MTAHQ spent approximately 95% of the IEC contract funds on project monitoring across all categories of the capital program, rather than value-added programmatic reviews and risk assessments. This ratio raises concerns regarding the allocation of IEC resources. The MTA may be placing too much emphasis on general project monitoring, and not enough time and resources on other activities and analyses that will help inform CPOC about how to provide better oversight.

MTA Office of the Inspector General 18

MTA/OIG Report #2010-04 November 2010

Given the higher risks associated with the mega-projects relative to the risks of the core capital program, MTAHQ may want to direct OCO to reallocate efforts and resources to place more emphasis on the mega-projects. As the table below illustrates, only 21% of the IEC’s efforts and resources were spent on overseeing the mega-projects, despite the fact that these projects made up 31% of the MTA’s Capital Program expenditures. A more risk-based approach to resource allocation would place even greater than proportional resources (i.e., more than 31%) on the higher-risk mega-projects.

MTA 2005-2009 Oversight Expenditures Program vs. Oversight Capital Program Paid to IEC in 2009 Expenditures($ billions) Percent ($ millions) Percent

Core Projects 16.6 69% 5.31 79%

Mega-Projects 7.4 31% 1.45 21%

Total 24 6.76

MTA Office of the Inspector General 19

MTA/OIG Report #2010-04 November 2010

RECOMMENDATIONS

8. CPOC should require OCO/IEC to prepare an annual risk-based work plan that takes into consideration where the MTA is most vulnerable to cost overruns or schedule delays, and further require that this plan be reviewed with and approved by the Committee.

9. The IEC's contractual scope of work should be adjusted to require it to devote more time and resources to risk assessments, programmatic reviews, and other analyses that will help inform CPOC specifically about ways to improve mega-project delivery.

MTAHQ accepted these recommendations. The MTA Chairman directed OCO to revisit the IEC’s resource allocation and develop a risk-based approach to oversight. Consistent with these recommendations, other consultants will be retained to conduct more specialized programmatic reviews while the IEC will verify and monitor reported progress on the capital program.

MTA Office of the Inspector General 20

MTA/OIG Report #2010-04 November 2010

EVALUATING IEC PERFORMANCE

The MTA does not now have a process in place to evaluate the performance and effectiveness of the IEC. The MTA currently spends almost $7 million a year for IEC services, but it is unclear how much value the MTA receives for this expenditure. The value of IEC services is in how much its efforts contribute to CPOC’s ability to conduct effective oversight, as well as in how effectively and efficiently it timely alerts MTACC to issues that threaten budget, schedule, or scope.

Although specific expectations for the IEC are set in the contractual scope of work, and could provide the basis for a formal performance review process, OCO does not now utilize these provisions to best advantage. Rather, OCO’s present assessment of the quality of the IEC’s work primarily entails checking the quality of the materials produced by the IEC for monthly CPOC meetings. Currently, OCO does not facilitate regular formal assessments of the IEC’s performance or define overall criteria that would form the basis for such an assessment. Such criteria are available in the contractual scope of work and should be applied in a formal performance review process.

MTA Office of the Inspector General 21

MTA/OIG Report #2010-04 November 2010

RECOMMENDATIONS

10. The IEC should provide semi-annual status reports on its performance against its work plan and its accomplishments. As called for in the IEC’s contract, these status reports should highlight those instances where the IEC’s activities have contributed to capital program oversight.

11. OCO should meet annually with the Chairman and the Chairs of the operations committees for NYC Transit, LIRR, MNR and B&T to get feedback on the IEC’s performance.

12. OCO should conduct a formal annual assessment of the IEC’s performance to the satisfaction of CPOC. This assessment should determine whether the IEC has contributed to the MTA capital program oversight process and demonstrated the value added by its efforts. Criteria for assessing IEC performance should include whether the IEC has successfully implemented its work plan, fulfilled other contractual obligations, and contributed to capital program oversight.

MTAHQ accepted these recommendations and moved aggressively to revise expectations for the IEC and to develop a new work plan for the IEC and CPOC as well.

MTA Office of the Inspector General 22

MTA/OIG Report #2010-04 November 2010

CONCLUSION

MTACC mega-projects have experienced well-publicized budget overruns and disruptive schedule delays that have seriously undermined public confidence in the MTA’s management of these projects. In an effort to discover the causes of these overruns and delays, and to help address them, the Office of the MTA Inspector General has conducted an across-the-board review, of which this governance assessment is the first part. Upcoming work will explore other potential causes, including those more fundamental and systemic in nature.

During the course of our review, and though we found that the governance structure is essentially sound, we determined that the MTA Board’s Capital Program Oversight Committee must take an overtly risk-based approach to oversight of the mega-projects. We also determined that CPOC must ensure that both MTA Capital Construction, the agency responsible for building these projects, and the Independent Engineering Consultant to the Committee, provide CPOC with the high-level analyses necessary for its members to perform that function.

Given the importance of this issue, and at the invitation of the MTA Chairman, the MTA Inspector General publicly presented his significant preliminary findings directly to the members of CPOC even before issuing this report. We support the changes that MTA has recently implemented, both on its own and at our behest, and suggest that it further consider the findings and recommendations of this report to help bring greater clarity, less conflict, and more effectiveness to the oversight process.

MTA Office of the Inspector General 23