Oregon OSCIO, DHS &

A Report for Oregon OSCIO, DHS & OHA

ONE IE&ME Project Schedule and Budget Estimation Review, Assessment and Validation

7 February 2018 Engagement: 330045871

This Report, including any supporting materials, is owned by Gartner, Inc. and/or its affiliates and is for the sole use of the intended Gartner audience or other authorized recipients. This Report may contain information that is confidential, proprietary or otherwise legally protected, and it may not be further copied, distributed or publicly displayed without the express written permission of Gartner, Inc. or its affiliates. © 2018 Gartner, Inc. and/or its affiliates. All rights reserved. Engagement: 330045871 ONE IE&ME Project Schedule and Budget Estimation Review, Report for Oregon OSCIO, DHS & OHA Assessment and Validation – Final Report 30 January 2018 — Page i

Table of Contents

1.0 Executive Summary ...... 1 1.1 Background and Lessons Learned ...... 1 1.2 Schedule and Budget Estimates Assessment Conclusions ...... 3 1.3 Strengths and Success Factors ...... 4 1.4 Pillars of Success and Summary of Key Recommendations ...... 5 1.5 Recommendations ...... 6 1.6 Summary of Findings ...... 7 1.7 Risk Analysis Approach ...... 11 2.0 Approach and Methodology ...... 18 2.1 Background ...... 18 2.2 Assessment Goals ...... 18 2.3 Assessment Objectives...... 18 3.0 Analysis and Findings ...... 20 3.1 ONE IE&ME Project Scope...... 20 3.1.1 Description ...... 20 3.1.2 Project Scope Strengths ...... 25 3.1.3 Project Scope Challenges and Potential Risks ...... 25 3.2 Reasonableness of Estimation Methodology ...... 28 3.2.1 Estimation for Deloitte DDI ...... 28 3.2.2 Deloitte DDI Estimation Strengths ...... 28 3.2.3 Deloitte DDI Estimation Challenges and Potential Risks ...... 29 3.2.4 Estimation for State’s ONE IE&ME Legacy Readiness Activities ...... 29 3.2.5 Legacy Readiness Estimation Strengths ...... 32 3.2.6 Legacy Readiness Estimation Challenges and Potential Risks ...... 33 3.3 Reasonableness of Deloitte ONE IE&ME DDI Schedule ...... 34 3.3.1 ONE IE&ME Integrated Project Schedule ...... 34 3.3.2 Integrated Schedule Strengths ...... 35 3.3.3 Integrated Schedule Challenges and Potential Risks ...... 35 3.4 Reasonableness of State’s ONE IE&ME Project Support and Legacy Readiness Projects Schedule ...... 36 3.4.1 Four Phase Legacy Readiness Schedule ...... 36 3.4.2 Legacy Readiness Schedule Strengths ...... 37 3.4.3 Legacy Readiness Schedule Challenges and Potential Risks ...... 37 3.4.4 Eligibility Transformation Schedule ...... 38 3.4.5 Eligibility Transformation Schedule Strengths ...... 39 3.4.6 Eligibility Transformation Schedule Challenges and Potential Risks ...... 40

© 2018 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a trademark of Gartner, Inc. or its affiliates. For internal use of Oregon OSCIO, DHS & OHA only. Engagement: 330045871 ONE IE&ME Project Schedule and Budget Estimation Review, Report for Oregon OSCIO, DHS & OHA Assessment and Validation – Final Report 30 January 2018 — Page ii

3.5 Reasonableness of Deloitte DDI Costs ...... 40 3.5.1 Deloitte DDI Costs Strengths ...... 41 3.5.2 Deloitte DDI Costs Challenges and Potential Risks ...... 41 3.6 Reasonableness of State Costs...... 42 3.6.1 State Costs Strengths ...... 45 3.6.2 State Costs Challenges and Potential Risks ...... 45 3.7 Comparisons with Other States’ Similar Efforts for Consideration ...... 45 4.0 Prioritized Risks and Mitigation Strategies ...... 47 4.1 Risk Prioritization Approach ...... 47 4.2 Process-Related Risks...... 48 4.3 People-Related Risks ...... 55 4.4 Product-Related Risks ...... 59 5.0 Conclusions and Recommendations ...... 63 5.1 Schedule and Budget Estimates Assessment Conclusions ...... 63 5.2 Schedule and Budget Estimates Assessment Recommendations ...... 64 Appendices ...... 66 Appendix A — Listing of Prioritized Risks ...... 67 Appendix B — Legacy Readiness Scope and Effort ...... 77 Appendix C — Gartner’s Project Approach and Data Discovery ...... 91

1.0 Executive Summary

1.1 Background and Lessons Learned The State of Oregon was one of the early states to start planning for provisions of Affordable Care Act passed in 2010, as summarized below:  With the announcement of CMS rules for enhanced funding (90% instead of the usual 50%) for Medicaid related systems and 100% for state-based Health Insurance Exchange (HIX) systems in 2011, the State decided to invest in a state of art solution to initially deploy a Health Insurance Exchange (HIX) with Medicaid MAGI eligibility and enrollment capabilities, and eventually a fully Integrated Eligibility and Enrollment solution for all health and human services benefits for the State  In 2012, using the CMS approved Implementation Advanced Planning Document (IAPD) and through a competitive procurement process, Oregon chose to partner with Oracle Professional Services (OPS) to design, configure and deploy a state-based HIX and MAGI solution on top of the Oracle Siebel CRM Public Sector platform  Oregon encountered numerous challenges with OPS in successfully configuring, testing and deploying an effective solution by the targeted January 2014 deadline, leading Oregon to eventually terminate its relationship with OPS for further efforts to provide the envisioned new eligibility and enrollment solution for the full continuum of the State’s health and human services programs ONE Integrated Eligibility & Medicaid Eligibility (ONE IE&ME) Project History In 2014, Oregon selected Deloitte to help in conducting a feasibility study for alternative solutions in the market to address the best path forward for eligibility and enrollment beginning with MAGI capabilities. Through market research and site visits, the State chose to leverage the solution built by Deloitte for the Commonwealth of Kentucky for MAGI Medicaid (Kynect) which had already been successfully developed and deployed into production. This was evaluated as the best value way forward for Oregon. Oregon embarked upon a disciplined process for the successful deployment of Kynect for MAGI Eligibility and Enrollment, including the following:  Deloitte conducted a Fit/Gap analysis against Oregon’s key requirements related to its MAGI Medicaid program, and eventually negotiated a contract to deploy Kynect within 11 months with minimum changes and customization to the core platform, except for a connection to the Federally Facilitated Marketplace (FFM)  Oregon was able to successfully deploy Kynect into production by December of 2015, and spent the first few months in 2016 to stabilize the platform (known as ONE – OregONEligibility), and deploy the application portal, and subsequently contracted with Deloitte for Maintenance and Operations (M&O) Services and certain critical Enhancements for the ONE solution  As of October 2017, Oregon has enrolled 962,577 individuals in Medicaid and CHIP — a net increase of 53.68% since the first Marketplace Open Enrollment Period and related Medicaid program changes in October 2013

With CMS announcement in 2015 of the continuation of enhanced funding availability for Medicaid Claims and Eligibility and Enrollment (E&E) systems, as well as the extension of the OMB A-87 Cost Allocation Waiver through 2018, Oregon pursued its original vision of deploying a fully integrated E&E solution for a majority of its Medical and Social Services programs and related benefits, as summarized below:  In 2016, Oregon completed an analysis of available alternatives to achieve its consumer centric, no wrong door E&E strategy, and with encouragement and approval from CMS chose to continue to enhance the ONE system to become the integrated E&E solution for Oregon Health and Human Services referred to as the Integrated ONE System  Oregon achieved Federal approvals to contract with Deloitte to configure and deploy an integrated E&E solution by the December 2018 deadline for the A-87 Cost Allocation Waiver, to take advantage of maximum enhanced funding for healthcare and human services programs  The Design, Development and Implementation (DDI) budget of $126M was established which did not include full implementation services (initial release only), any M&O Services, adequate staffing levels, and did not contemplate full data conversion or changes to the legacy systems. It also planned for short durations of SIT, User Acceptance Testing (UAT), and Pilot, in a strategy similar to the initial MAGI ONE deployment approach  The State and Deloitte completed Fit/Gap analysis and Joint Application Development (JAD) sessions to better identify the full scope of work for the project  By September of 2016, after completing a number of core design sessions, it became clear to the Project Director that an additional $50M of funds will be needed for both Deloitte and State resources to be able to achieve completion by December 2018. Therefore, a budget request for $177M was prepared and presented to CMS and the Legislature for approval. The revised $177M budget still did not include adequate staffing levels, the Legacy Readiness nor M&O services, and did not extend the short testing and Implementation timelines In 2017, the State identified operational challenges that had to be addressed in order to enable the deployment of an integrated approach to business operations for health care and human services eligibility and enrollment. Therefore, the need to ensure alignment of State’s model of practice with the envisioned ONE IE&ME technology, through the transfer of the Deloitte Kentucky solution, became a critical priority. The State Directors of Department of Human Services (DHS) and Oregon Health Authority (OHA) with facilitation from the State CIO, agreed to combine all Eligibility operations under DHS leadership to implement a fully integrated Service Delivery Operations for Eligibility and Enrollment. To ensure success of the ONE IE&ME effort, and recognizing the complexity of the business and technology challenges that need to be addressed, the State made the decision to no longer pursue December 2018 as the ONE IE&ME completion deadline. A schedule was developed that allows the State to effectively move forward with addressing the project requirements and related complexities. In July of 2017, the Project Director requested a $63M increase to the budget, and extended the completion date to July of 2019 for a total State and Deloitte DDI budget of $240M. These additional six months were intended to allow for a more orderly deployment during the implementation phase, and included a small amount of funds for Legacy Readiness and Disaster Recovery.

By the end of Summer 2017, as both the Deloitte and State Legacy Readiness teams continued to make progress toward completion of the Functional Design Documents, it became increasingly clear that a comprehensive ONE IE&ME solution that can deliver a new model of practice and convert all the beneficiaries into the new system will require additional time to allow for proper testing, pilot and a 3-wave deployment. In December of 2017 the Project Director proposed the latest budget and schedule for IE&ME DDI that takes the overall (State and Deloitte) IE&ME DDI costs to $345M, and extends the schedule by another 6 months to July of 2020 to allow for longer Systems Integration testing, User Acceptance Testing, Pilot, Rollout, and the Office Simulation Design and Testing and related design addendums. Third Party Schedule and Budget Review The ONE IE&ME project is of critical importance for Oregon’s future vision of, and transformation towards consumer centric, no wrong door access to services and the eligibility process. Due to the challenges and lessons learned from the original OPS effort and the size, complexity and costs of the current ONE IE&ME effort, key interested parties including the State Legislature, State agency leadership, Federal funding partners, many special interest groups want to know that there is a highly level of confidence that this effort will be successful. Based on this reality, in 2017 the State Legislature called for an independent, third party assessment of the project to provide assurances that Oregon is on the right path for ONE IE&ME, the latest budget developed is sufficient, and that the schedule is reasonable for a project of this size and complexity. Starting in December of 2017, Gartner assigned its senior members of its national public-sector Health and Human Services Consulting Practice to conduct a detailed review of the scope and status of the ONE IE&ME project and the latest available budget, schedule and cost estimates.

1.2 Schedule and Budget Estimates Assessment Conclusions Based on the detailed review of the ONE IE&ME Project Business Case, Vision and Scope, the processes used by State and Deloitte teams to develop the schedule and budget, as well as the latest Integrated Project Schedule and the State’s overall budget published in December of 2017, it is Gartner’s opinion that:  The ONE IE&ME project incorporates the critical tasks and activities required to complete a robust Integrated Eligibility (IE) Solution for State of Oregon aligned with the State’s defined scope  The Solution selected by Oregon (Deloitte’s Benefind from Kentucky) is a proven solution that has been deployed into Production already in Kentucky (Full Integrated Eligibility) and Oregon (MAGI Medicaid), and its functional and technical components originate from the Systems Integrators’ work in IE for a number of other states in the last 12-15 years  The revised $164M Contract with Deloitte for Design, Development, and Implementation of Benefind is within the range of magnitude of investments observed by Gartner in other states that have invested in Integrated Eligibility solutions of similar scope in the last 7 years. In fact, it is on the low end of the scale based on Oregon’s decision to implement the solution as a “Transfer” with minimal customization  The State’s Legacy Readiness budget of $9.2M and an additional $1.4M in contingency funds is sufficient to cover the expenses related to the legacy systems remediation effort based on the completed planning and design work to date

 The overall ONE IE&ME DDI budget of $345M (State and SI Costs) is on the low end of the range of total project expenditure observed in other states for comparable projects  Given the level of planning completed to date, Gartner’s view of the current budget and schedule forecast is that it is adequately revised and expanded to be within a margin of ±15% of the actual costs to achieve the deployment of current scope of functionality. This conclusion is based on Gartner’s review of completed planning and design documents for ONE IE&ME and Legacy Readiness, as well as industry research in to the quality and accuracy of project estimates at the end of current Systems Development lifecycle stage  State of Oregon should expect to require additional effort and expenditure for optimizing processes and functionality (Optimization Phase) in the first five years post the state- wide go live date, separate from Maintenance and Operations and Enhancement related expenditures  The revised ONE IE&ME schedule with a target completion date of July 2020 is a reasonable schedule in comparison to other projects of similar scale and scope in other states, and will provide State of Oregon with sufficient time to conduct adequate testing  The robust rules and policy management capabilities planned as a part of the ONE IE&ME solution can be expected to minimize the type of Medicaid audit findings that have been identified in previous years for OHA

1.3 Strengths and Success Factors Since embarking on the current direction Oregon has progressed with considerable success and significant achievements including:  The rapid implementation of ONE (in 11 months) — the transferred and minimally modified KY solution for Modified Adjusted Gross Income (MAGI) Medicaid Eligibility integrated with the Federal Facilitated Marketplace  Establishment of strong program leadership and agreement on the scope and benefits aligned with the vision of the business for best practices for serving the citizens of Oregon to be achieved through the investment of time, people and dollars in the ONE IE&ME initiative  Based on a strategy of building upon the success of ONE and implementing minimally modified Benefind, for Modified Adjusted Gross Income (MAGI) and non-MAGI Medicaid, Supplemental Nutrition Assistance Program (SNAP), Temporary Assistance for Needy Families (TANF) and Child Care programs, Oregon has negotiated with the System Integrator (Deloitte) a favorable pricing compared with similar efforts in other States  Built a strong State (Business and Technical) and System Integrator team and Joint Governance Board to pursue this large and complex effort for the ONE IE&ME

1.4 Pillars of Success and Summary of Key Recommendations There are many aspects to the management and successful execution of such a large and complex effort like ONE IE&ME. From the Gartner review and findings, the State has in place many of the critical components essential for success. There are two key pillars for the continued success of the ONE IE&ME initiative and the State’s ability to address the challenges and risks identified through this third-party assessment of scope, schedule and budget. Figure 1 below depicts how the State can strengthen the ONE IE&ME effort through focused attention on the two key pillars in place to ensure continued momentum towards success and to identify, anticipate, and mitigate risks going forward — including what are identified as challenges/risks and related recommendations in this report. The two key pillars are:  Program Integration —Identification of clear roles and responsibilities for the Executive Governance Body (Joint Governance Board), Independent Project Director, and movement toward expanding the current Project Management Office to a Program Management Office (PMO) that will include all of the projects and initiatives that impact the ONE IE&ME initiative— focusing on integrated work plans; resource loading; management of dependencies and Interdependencies; proactive resource planning to mitigate resource contentions; risk and issues identification-logging-management  Contract and Vendor Management – The State has a contract in place with the System Integrator that reflects best practices in fixed price contracting. The State’s contract is based on a capped not to exceed amount driven by statements of work and deliverables expectation documents (DEDs) for Fixed Price Deliverables. Building on this contract structure, the State can enhance both contract and vendor management efforts to strengthen the ability to bring the ONE IE&ME effort in on-schedule and within budget Figure 1. Pillars for ONE IE&ME Success

1.5 Recommendations The following recommendations along with the suggested risk mitigation strategies presented in the body of this report will support the State of Oregon in moving forward with the successful delivery of the ONE IE&ME project: 1. Systems Integrator Partnership – State of Oregon should proceed with the implementation of ONE IE&ME project in collaboration with Deloitte. Gartner does not see any barriers or risks that would necessitate reconsideration of moving forward with the ONE IE&ME initiative 2. Program Management Office – The State of Oregon should consider elevating the current Project Management Office to a Program Management Office model with the related capabilities, mature processes, staffing and critical skill sets to exert adequate leadership and control over the project, and employ timely and effective risk management strategies over all of the projects and initiatives that impact the success of the One IE&ME effort, preventing any unplanned scope changes and avoid schedule delays 3. Strong Vendor Management and Performance Metrics – The Program Management Office, in collaboration with the Quality Assurance Vendor, must attempt to establish strong metrics to hold the Systems Integration vendor accountable to deliverable quality and timelines, and the State staff to conducting the reviews and sign-offs on a timely basis. This should be addressed through ensuring comprehensive Statements of Work and robust Deliverables Expectation Documents based on industry best practices for deliverable review and acceptance criteria 4. Reducing Data Quality Risk Impact – Since the deployment of ONE IE&ME is planned through a three wave, incremental geographical basis, it requires that any data quality and data conversion issues for all in-scope programs to be uncovered and resolved during a single pilot phase. ONE IE&ME project leadership should conduct a formal analysis of alternative approaches to deployment and data conversion as a contingency deployment strategy, including consideration of a multi-phase deployment strategy on a Program by Program (or Program Groups) basis, that could reduce the likelihood or extent of data quality issues that may arise 5. Eligibility Related Legacy Systems Evolution and Retirement – DHS leadership should continue to evaluate strategies to transform the family of Eligibility-related legacy mainframe and PowerBuilder based application systems that will not be replaced by Integrated ONE, with a view to enhance the underlying technology and integration architecture, in order to minimize complexity, and reduce technical debt and DHS/OHA IT infrastructure and application costs. As there are substantial data model differences between these legacy systems and Integrated ONE, even minor changes could require considerable time and cost to implement. The evolution of these legacy systems should enable a more functional, streamlined and less expensive model for functional enhancements to Integrated ONE going forward 6. M&O Strategy – State of Oregon should conduct a formal analysis of alternatives for the best value sourcing approach to Integrated ONE Application Maintenance and Operations (M&O) responsibilities. This analysis should consider a broad range of alternatives including a portion or a majority of these responsibilities being assigned to internal MS .NET Application Development resources vs. hiring an external vendor. Given the estimated $24M plus per year in Integrated ONE Application M&O costs, this will present the potential of considerable savings to DHS from a Total Cost of Ownership perspective, as well as additional employment opportunities for Oregonians

1.6 Summary of Findings ONE IE&ME Estimates Review Starting in December of 2017, Gartner assigned its senior members of its national public-sector Health and Human Services Consulting Practice to conduct a detailed review of the status of the ONE IE&ME project and the latest available budget and cost estimates. Below is a high-level summary of the key findings:  Estimation Process (Deloitte) – Deloitte’s approach to developing a cost estimate and schedule for Oregon appears to follow its common estimation and bidding practice across the country. It starts with its fixed costs for the project duration and then adds in an estimate for the number of widgets that have to be configured and/or built using its database of actual efforts associated with previous projects, and finally accounts for all special additional on the ground support for training, User Acceptance Testing (UAT) deployment, etc.  Estimation Process (State) – The State of Oregon does not have fully mature processes for capturing and measuring software development effort estimates. For Integrated ONE legacy Readiness effort, the Office of Information Services (OIS) leadership built a specialized and reasonably sophisticated model (with some input from Gartner Research) to allow it to take a developer’s experience based estimate for Development and Testing effort for a system and turn that into a fully extrapolated and risk adjusted estimate for the effort at hand. While the processes being used and updated are sufficient for this project, these early and one-time efforts will not increase the State's ability to produce more mature and higher quality estimates in the future  Schedule Estimate (IE&ME) – The latest re-baselined schedule ending ONE IE&ME Design Development and Implementation (DDI) activities in July of 2020, is consistent with industry observations of other states’ durations, and includes a suitable timeline for an E&E project of Oregon’s size and scope for integrated service delivery. However, there are a number of unknown areas whose impact will not be known until design “True Up” is completed in March 2018. Potential residual risk factors impacting the published schedule could include those related to the completion of Legacy Readiness effort by October of 2018, as well as the impact of Data Quality issues that arise around data conversion from the legacy systems during the deployment phase  Cost Estimate (State and Deloitte) – At a current total cost estimate of $345M, of which $166M are related to Deloitte’s ONE IE&ME DDI expenditure, the project has a favorable cost disposition compared to other Deloitte projects

Scope, Schedule and Cost – Strengths and Challenges and Potential Risks The findings for each of the areas analyzed, including Scope, Estimation Methodologies, Schedule and Cost, are summarized below.

Scope:  State:  Strengths: ─ Strong Program leadership and agreement on the scope and benefits ─ Scope aligned with the State’s vision for ONE IE&ME ─ Business Operations standards are maturing, focusing on the alignment with the envisioned model of practice and functional capabilities of the ONE IE&ME solution ─ Agreed scope is well communicated and widely understood ─ Highly transparent plans – The latest integrated schedule covers the entire agreed scope actively managed by the PMO, including the Legacy Readiness activities  Challenges and Potential Risks: ─ Remaining IE&ME Design Work – There is additional design work for ONE IE&ME yet to be completed that may result in a need to change the level of core activities required for the Deloitte’s DDI efforts ─ Remaining Legacy System Design Work – There is also additional design work on Legacy systems that may impact the resources necessary for Legacy Readiness, and possibly may have an impact on Deloitte’s DDI schedule and costs if it delays remediation of key legacy systems that are a part of Legacy Readiness project, beyond October 1, 2018 ─ Defining What Is Deloitte’s “Design, Development and Implementation (DDI) Completion” and understanding what remains unaddressed after Deloitte’s Completion of DDI: ─ Given the not-to-exceed, fixed cost, deliverables-based nature of the contract, and the magnitude of the costs involved, the most critical risk related to the Deloitte DDI Costs is to keep the project within the agreed scope

Schedule:  Deloitte:  Strengths: ─ Comprehensive and detailed ─ Updated and maintained on a regular basis by the PMO  Challenges and Potential Risks ─ Design work incomplete ─ Legacy data quality unclear and conversion strategy incomplete

─ Scope changes may impact the schedule ─ Earned Value not well communicated ─ Skilled Staff contention ─ A strategy is needed to ensure all ONE (MAGI Medicaid) System enhancements are included ─ Risk Management disjointed  Legacy Readiness:  Strengths: ─ System Integrator is providing support of the more complex cases ─ Most functional design is complete ─ High level of knowledge and experience involved ─ Experienced OIS staff have high confidence of meeting the target date ─ Technical staff added to the Legacy Readiness team  Challenges and Potential Risks: ─ Remaining design work ─ Staff resource contention for maintenance tasks ─ Major changes for Payment Issuance (JV) ─ Volume and scale of changes may risk unforeseen obstacles ─ Limited historical documentation; Additional documentation has been created as part of the Legacy Readiness work  Eligibility Transformation (ET):  Strengths: ─ Executive leadership ─ Business and System strategy alignment ─ ET Business Operations standards alignment ─ The focus of ET is to enable Oregon to use the Kentucky Solution while keeping customizations and changes to a minimum ─ Good business and technology representation ─ ET has staff assigned with appropriate knowledge and experience  Challenges and Potential Risks: ─ ET and ONE IE&ME have separate governance and reporting structures ─ ET workstreams are yet undefined ─ ET and ONE IE&ME interdependencies are not yet fully defined ─ ET may not have sufficient time to define and influence ONE IE&ME requirements with the current schedule

─ Some desired model of practice changes may have to be postponed until future versions of ONE IE&ME.

Costs  Deloitte:  Strengths: ─ Prices compare well with other States ─ Current price accounts well for the level of effort and number of resources logically anticipated  Challenges and Potential Risks: ─ Inflexibility of contract makes scope (including dependencies) management critical ─ Deloitte may have a tendency to inappropriately fill leadership gaps or make key project decisions that need State leadership and decision making responsibility  State:  Strengths: ─ The budget provides adequate funding ─ The PMO budget should be sufficient to increase staff as necessary ─ The bottom-up resource estimate is below the allocated budget for Legacy Readiness  Challenges and Potential Risks: ─ Limited State Subject Matter Experts (SMEs) available ─ Limited PMO process and capability maturity ─ Contention for key business and technical staff

1.7 Risk Analysis Approach For each of the areas analyzed (Scope, Estimation Methodology, Schedule and Cost) a number of Risks were identified, prioritized and recommendations to help mitigate each risk made. Each risk was assigned a Priority Score out of 100. This score was calculated by evaluating the likelihood (probability) and impact of the risk with scores out of 10. The Priority Score is the product of the likelihood score and impact score.

Based on the Priority Score each risk was assigned to a Priority Tier:

 High – Risks with a Priority Score of 45 or more; 12 Risks identified  Medium – Risks with a Priority Score greater than 20 but less than 45; 12 Risks identified  Low – Risks with a Priority Score of 20 or less; 4 Risks identified

Considering the size and complexity of the ONE IE&ME Project, the number and nature of the High Priority Risks, from Gartner’s experience, demonstrates the strength of the go forward path that the State has developed for the ONE IE&ME initiative.

Summary of Identified Risk by Priority Tier

Risk Priority Priority # Risk Score Tier 14 Resource Contention 64 H 18 Resource Redirection 64 H 2 Remaining Legacy System Design Work 63 H 27 Limited State Subject Matter Experts 63 H 7 Project Governance Alignment 56 H 12 Data Quality 56 H 16 Merging of ONE and IE&ME 56 H 22 ET and ONE IE&ME Coordination and Governance 56 H 1 Remaining IE&ME Design Work 49 H 6 Integrated Risk Management 49 H 28 Number of Expert Legacy Resources 49 H 19 Legacy Payment Issuance System 48 H 11 Reasonableness of IE&ME Schedule 42 M 13 Earned Value Reporting for Project Control 42 M 15 Resource Shortage for Testing 42 M 21 Limited Historical Legacy Documentation 42 M 9 Scope Management 40 M 26 DDI Contract Scope Management 40 M 20 Unanticipated Legacy Remediation Challenges 35 M 24 Model of Practice Requirements 35 M 25 Model of Practice Misalignment 35 M 8 Over Dependence on Vendor Skills and Resources 30 M

Risk Priority Priority # Risk Score Tier 23 ET Project Requirements and Interdependencies 30 M 5 Adequacy of Project Management Office (PMO) Capabilities 28 M 3 Lack of Clear Definition of "DDI Completion" 20 L 4 Understanding what remains unaddressed after Deloitte’s 20 L Completion of DDI 17 Legacy Functional Design Documents (FDD) Completion 20 L 10 Internal Software Development & Systems Integration Effort 16 L Estimation Process

The following table presents the top list of potential mitigation strategies that ONE IE&ME project leadership can employ to reduce the impact of risks identified as a part of this study. Please note that some of the mitigation strategies listed may already be a part of the risk management approach being employed by the ONE IE&ME project teams.

Suggested Risk Mitigation Strategies by Priority

Risk # Risk Mitigation Strategies 14 Resource Contention Formally remove all non-IE&ME related demands on the Legacy Readiness development team until October 2018.

18 Resource Redirection Formally remove all non-IE&ME related demands on the Legacy Readiness development team until October 2018 (see Risk #14). 2 Remaining Legacy Further develop contingency plans by identifying any work that is System Design Work not absolutely necessary to meet the October 1 deadline and additional sources of contract developers to further augment the Legacy Readiness team if this becomes necessary.

27 Limited State Subject Continue with establishing a formal process to identify weekly Matter Experts activities and resource requirements at a detailed level for the next 8-12 weeks, and work to avoid any duplicate booking of key resources by adjusting schedule and timing for meetings.

7 Project Governance Consider mechanisms to integrate the governance and oversight of Alignment key tasks and activities defined by the ET project into an overall IE/ME Program Management Office model.

Such a Program Management Office would keep focus on an enterprise approach and ensure effective sequencing of the planning, requirements and procurement efforts in accomplishing the eligibility transformation envisaged, help contain risks and maximize efficiency in the allocation and use of resources (see also Risk #22).

Risk # Risk Mitigation Strategies 12 Data Quality Place special emphasis on Data Profiling for legacy systems whose data is essential to be migrated to IE&ME System, and identify the systems whose data represent the most significant risk to successful deployment, and develop mitigation plans to address the potential risks in 2018.

16 Merging of ONE and Develop detailed traceability matrices and plans for testing and IE&ME verification of incorporation of ONE Enhancements to the original release. Develop key entry and exit criteria for each stage in the testing life cycle and pilot.

22 ET and ONE IE&ME Ensure business sponsors and executives are regularly engaged in Coordination and discussions related to status and interdependencies between the Governance two projects, and provide the appropriate resources to ensure success, and decide on best path forward based on discussion of tradeoffs (see also Risk #7).

1 Remaining IE&ME Review and update impact of final design changes on schedule and Design Work budget in April 2018 and identify strategies to fit new level of effort within existing schedule and budget.

6 Integrated Risk Conduct a thorough assessment of the Risk Management Management processes. Ensure that this results in strengthened and fully integrated processes with a high level of transparency.

Develop an integrated risk management process that properly identifies high complexity and high uncertainty tasks that have a potential to negatively impact the latest DDI schedule, and ensure executive business leadership in decisions related to managing those risks.

28 Number of Expert Continue to define and refine the resource requirements for legacy Legacy Resources system remediation activities with additional level of granularity and resource assignments, so that the potential needs of multiple legacy systems for the same key experts during the same time frame becomes clear, and can be avoided.

19 Legacy Payment Vigorously monitor and control the scope of required functionality, Issuance System and ensure appropriate level of resources are assigned to ensure timely completion and testing of the new functionality.

11 Reasonableness of Be prepared to judiciously evaluate any change that will have an IE&ME Schedule impact on the latest schedule published in December 2017, and eliminate sufficient number of non-critical / optional functionality or tasks to maintain fidelity to the latest schedule.

Risk # Risk Mitigation Strategies 13 Earned Value Incorporate earned value reporting (i.e., planned and earned value Reporting for Project matrices) that shows the value received for the budget expended to Control date into the regular reporting for the project.

15 Resource Shortage for Develop detailed test plans for required resources during the UAT Testing to ensure adherence to schedule.

21 Limited Historical Conduct daily standup meetings between March and October of Legacy Documentation 2018 to monitor progress and discuss documentation related issues that each of the legacy development teams may have run into, and assign issue owners and facilitator to expedite resolution of the same.

9 Scope Management Ensure that anything that threatens the Scope that is validated, agreed, and approved (see items 3 and 4 above) is escalated to the executive level of the Governance structure (see also Risk #26).

26 DDI Contract Scope Ensure that scope is tightly managed by the Business Sponsors Management and Executives, and any changes are thoroughly vetted to understand the impact on cost and schedule (see also Risk #9).

20 Unanticipated Legacy Conduct daily standup meetings between March and October of Remediation 2018 to monitor progress and identify unanticipated issues that Challenges each of the legacy development teams may have run into, and assign issue owners and facilitator to expedite resolution of the same.

24 Model of Practice ET should establish key business outcome metrics (e.g., Requirements processing times by program, case worker productivity, data quality, etc.) related to the new integrated delivery model, and continue on the path of minimizing any changes to Benefind's functionality and capabilities available out of the box.

25 Model of Practice ET should establish key business outcome metrics (e.g., Misalignment processing times by program, case worker productivity, data quality, etc.) related to the new integrated delivery model, and continue on the path of minimizing any changes to Benefind's functionality and capabilities available out of the box.

Risk # Risk Mitigation Strategies 8 Over Dependence on Strengthen Vendor Management with metrics that go well beyond Vendor Skills and those necessary for Contract compliance. Have the Quality Resources Assurance (QA) vendor monitor the scope, focus, and extent of the System Integrator's role.

Ensure that all important decisions are transparently and fully dealt with by the agreed governance processes.

Ensure that any Vendor recommendations and guidance are balanced and/or validated against third party advice and recommendations.

23 ET Project Complete the detailed planning for ET workstreams by March 2018, Requirements and identify all interdependencies and specific capabilities that need to Interdependencies be delivered by IE&ME once in production, and assess and manage the potential impact of those requirements on the planned IE&ME schedule and timeline.

5 Adequacy of Project Complete the procurement of additional PMO resources as soon as Management Office possible. (PMO) Capabilities To get the most form these additional available resources complete a skills gap assessment across the PMO considering the elevation of the current PMO to a Program Management Office model. Use the additional available resources and staff development to address these gaps urgently.

3 Lack of Clear Definition Update the formal Scope description and submit for approval by the of "DDI Completion" Joint Governance Board.

4 Understanding what Include within the formal Scope description and approval process remains unaddressed identification of what will remain unaddressed after Deloitte's after Deloitte’s Completion of DDI and an analysis of the impact of this remaining Completion of DDI unaddressed.

17 Legacy FDD Based on the March 2018 deadline to complete all legacy system Completion FDDs, assess the Legacy System team's ability to meet the October 2018 deadline for entry into Final SIT.

10 Internal Software Initiate a project within OIS to formalize development and Development & publishing of estimates at major milestones within large projects Systems Integration (i.e., greater than $0.5M), capture each estimate in an AD Effort Estimation Repository, and conduct actual vs. estimate post mortem analysis Process at the completion of the project against each milestone estimate. Establish formal time tracking for all IT Project activities and resources to have a factual baseline for the effort extended.

The table below provides a summary overview of the commonly observed factors behind large project failures as observed by Gartner Consulting’s Forensic Assessment work of troubled projects, as well as Gartner Research analysis of key failure influencers on a large population of projects from a multi-year study. The top four (4) influencers are highlighted here along with Gartner’s observations and potential action by ONE IE&ME leadership to minimize the impact of these common culprits. Risk Influencers

Failure Influencer As Shown By Examples Oregon Related Observations Capabilities Skills /  Sole qualification of the  Enhance the Project Experience at program director/manager is a Management Office Program reporting relationship to the capabilities to exert tighter Management program's executive sponsor. control over project activities, deliverables and  Program team members, project managers, are new to risks. This should include program management. consideration of moving to a Program Management Office model. Change "Ownership"  Change needed on the  Oregon has strong support business side has no agreed from business operations "owner" for action and for leadership. achieving a readiness "state" to  Define key business accept what the program will success metrics (e.g., Time deliver. to Process an Application,  The organization has a poor % of Applications requiring track record of realizing manual intervention, program benefits or getting the reductions in poor data best value from prior program quality by program, etc.) deliveries. and ensure the relevant metrics are tested during project testing, pilot, and deployment lifecycle stages. Governance Structure /  No formal governing structure  Consider conducting an "IT Decisions is defined, roles are fuzzy, and Project Success Aptitude" Rights / Sponsor governance just "grows" right survey of ONE IE&ME along with the program business leadership and content. sponsors, and subsequently develop action plans to  Decision authority is not specifically defined and increase the probability of successful project execution identified, or this is poorly done, with many unspoken and delivery of anticipated assumptions. benefits.  The executive sponsor has no experience with programs or at being a program sponsor. Reality Events /  Leadership is unwilling to hear  Ensure that the executive Evolution / "bad news" and takes no leadership is continually Escalation / effective action. engaged in discussions Decision Making related to changes in the  The organization leadership, and the program leadership status of the project, and don't realize — or accept — take ownership for making

Failure Influencer As Shown By Examples Oregon Related Observations that the program goals, the required decisions with direction, and content evolve respect to scope, resources, naturally over time; and that funding and timelines. they require continuing discussion and change to make adjustments and still fulfill strategic aims.

2.0 Approach and Methodology

2.1 Background Due to the challenges faced by the Cover Oregon initiative from 2012 to 2014, the State has put in place extensive oversight (seven oversight bodies) of the current effort to move forward with the ONE Integrated Eligibility & Medicaid Eligibility (IE&ME) Project which includes a Design Development and Implementation (DDI) services award to Deloitte as the System Integrator (SI). The State of Oregon Department of Human Services (DHS) and Oregon Health Authority (OHA) have received a request from Legislative Oversight to conduct a third party review of the ONE IE&ME project’s current project budget and timeline estimates: “Work with the Oregon Office of the State Chief Information Officer (OSCIO) to direct the quality assurance vendor to assist in the independent re-validation (preferably by an outside vendor) of all project estimates to assure that those estimates provided to date by the ONE IE&ME Project State staff and the system integration vendor are valid. Provide evidence that the re-validated estimates are based upon a well-defined estimating methodology and associated estimating assumptions. Include in this re- validation a clear definition of the “level of effort” required for each activity estimated by the system integration vendor.” Gartner was selected to conduct an objective third party review of the ONE IE&ME project budget and timeline including State and System Integrator schedule and cost estimates.

2.2 Assessment Goals The goals for the disciplined Schedule and Budget Estimation Review, Assessment and Validation include:  Through the Third-Party Review, provide the State of Oregon a high level of confidence in the Schedule and Budget Estimates for the ONE IE/ME Solution  Provide insights and recommendations for addressing any adjustments to the Schedule and Budget and/or mitigating any risks that are identified through this effort that will impact Schedule and Budget

2.3 Assessment Objectives To fulfill the project’s Goals the following Objectives must be achieved:  Conduct Project Initiation and Discovery – Ensure a coordinated and effective approach to managing the Schedule and Budget Estimation Review, Assessment and Validation effort during the Project Initiation phase. Conduct document review and key State and System Integrator stakeholder interviews to ensure that the Gartner team understands scope, schedule and budget development process and methodology employed as well as any constraints that have or may impact the estimations that have been made  Conduct Detailed Analysis of Schedule and Budget Estimates – Based on a review of planning and design documents and stakeholder interviews, analyze the information provided, assess assumptions and estimation methodology employed and conduct a working session with the State to review draft analysis and findings

 Develop Draft and Final ONE IE/ME Project Schedule and Budget Review, Assessment and Validation Report – Based on the Discovery effort and the analysis and findings, develop a draft final report and conduct working sessions with the State to review the draft report. Based on State input finalize the ONE IE/ME Schedule and Budget Review, Assessment and Validation Report  Executive Briefing – Develop and present a facilitated discussion with State leadership and interested oversight entity representatives involved with the ONE IE/ME Project on the analysis, findings and recommendations

For additional details on Gartner’s approach, key State of Oregon participants, and project documentation reviewed for the purposes of this assessment, refer to Appendix C.

3.0 Analysis and Findings

3.1 ONE IE&ME Project Scope

3.1.1 Description The ONE IE&ME Project Scope is outlined below in terms of Business Objectives, Programs Included, Systems Functionality and Data Conversion.

Business Objectives The ONE IE&ME project supports and is bounded by the following business objectives for integrated eligibility:  Allowing applicants to apply for benefits through online self-service portal, reducing the need to provide duplicate information when applying for benefits from more than one program, and eliminating unnecessary and undesirable travel without reducing the ability for applicants to come to an office for face-to-face services  Automating manual processes in order to:  Reduce the amount of time that elapses between completing an application, making an eligibility determination, enrollment, and distribution of benefits  Reduce the amount of time that staff must spend creating, reviewing, and acting on each application  Reduce the rates of errors in making eligibility determinations  Allowing for seamless sharing of information and transfer of cases among program staff supporting compliance with program specific federal funding requirements to support programmatic operations in order to:  Lower costs  Allow a single application for all programs  Allow multiple avenues to apply

Programs Included Programs within the scope of ONE IE&ME include:  Modified Adjusted Gross Income (MAGI) Medicaid currently in the OregONEligibility (ONE) system  Additional Programs for ONE IE&ME —  Non-MAGI Medicaid — This includes Oregon Supplemental Income Program (OSIP) Medical, also known as the Aged, Blind, Disabled Medical or Social Security Insurance (SSI) based Medical programs, Medicare Savings Programs, and Refugee Medical. These currently are processed in ORAccess, CM, and Medicaid Management Information System (MMIS). ─ Hospital Presumptive, Extended Medical, and Breast and Cervical Cancer Program are NOT in scope  Supplemental Nutrition Assistance Program (SNAP). This includes all portions associated with the SNAP program.

 Temporary Assistance for Needy Families (TANF), cash programs (This does NOT include case management) ─ Employment Related Day Care (ERDC). ─ Temporary Assistance to Domestic Violence Survivors (TA-DVS) ─ Other cash programs, such as special needs Details of the Programs in and out of scope are included in the tables below.

In Scope DHS Self Sufficiency Programs In Scope DHS Self Sufficiency Programs Type of Assistance Program (TOA) Description CTCE Categorical Eligibility EXCE Expanded Categorical Eligibility BBCE Broad Based Categorical Eligibility SNAP SNAP Supplemental ESNP Expedited SNAP Nutrition DSNP Disaster SNAP Assistance Program Able Bodied Adults without Dependents (SNAP) ABAWD Requirements Oregon Food Stamp Employment Transition OFSET Program Summer Meals Electronic Benefit Transfer for SEBTC Children TANF TANF EPPT Employment Payments TDVS TANF Domestic Violence TJPI Jobs Participation Incentive Temporary PSSI State Family Pre‐SSI Assistance for Jobs Opportunities and Basic Skills Needy Families Note ‐ Only determination of JOBS mandatory (TANF) JOBS individuals and assessment status, not ongoing case management CCTN Child Care Supportive Services for TANF receipts REFG Refugee Cash/TANF and Refugee Medical Child Care Program (CCPG) ERDC Employment‐Related Day Care

In Scope DHS Aging and People with Disabilities (APD) Programs

In Scope DHS Aging and People with Disabilities (APD) Programs Type of Program Program Description Assistance (TOA) QMB‐BAS Qualified Medicare Beneficiaries – Basic Qualified Medicare Beneficiaries ‐ Special Limited QMB‐SMB Medicare Beneficiaries Oregon Supplemental Income Program Medical ‐ OSIPM‐SSI Supplemental Security Income Oregon Supplemental Income Program Medical ‐ OSIPM‐1619B 1619B Non‐Modified OSIPM‐Survivor Oregon Supplemental Income Program Medical ‐ Adjusted Gross Widows Survivor Widows Income DAC Disabled Adult Children (Non‐MAGI) Pickle Pickle Amendment Clients Medicaid Oregon Supplemental Income Program Medical ‐ OSIPM‐AB Aid to the Blind Oregon Supplemental Income Program Medical ‐ OSIPM‐OAA Old Age Assistance Oregon Supplemental Income Program Medical ‐ OSIPM‐AD Aid to the Disabled OSIPM‐Acute Oregon Supplemental Income Program Medical ‐ Care Acute Care Oregon Supplemental Income Program Medical ‐ OSIPM‐EPD Employed Persons with Disabilities not receiving without Services services Oregon Supplemental Income Program Medical ‐ OSIPM‐EPD with Employed Persons with Disabilities receiving Services services QMB‐DW Qualified Medicare Beneficiaries ‐ Disabled Worker Special Needs Special Needs

In Scope Oregon Health Authority (OHA) Programs

In Scope Oregon Health Authority (OHA) Programs Type of Program Program Description Assistance (TOA) Modified Adjusted Gross Income (MAGI) MAGI Medicaid MAGI Medicaid Medicaid ADLT OHP Plus Adult CHIP OHP Plus CHIP CHL1 OHP Plus Child under age 1, under 185% CHL4 OHP Plus Child, age 1 through 18, under 133% EMAD CAWEM Adult EMC1 CAWEM Child under age 1, under 185% EMC4 CAWEM Child age 1 through 18 EMPC CAWEM Parent or Other Caretaker Relative CAWEM Plus Pregnant Parent or Other Caretaker EMPP Relative EMPR CAWEM Plus Pregnant Woman EMPW CAWEM Pregnant Woman FFCC OHP Plus Former Foster Care Youth HIA1 Hospitalized Adult inmate HIA2 Hospitalized Pregnant Woman inmate PACA OHP Plus Parent or Other Caretaker Relative OHP Plus Pregnant Parent or Other Caretaker PCPR Relative PREG OHP Plus Pregnant Woman TP45 OHP Plus Assumed Eligible Newborn

Out of Scope Programs, Sub-Programs and Functions

Out of Scope Programs and Sub‐Programs and Functions

Type of Program Assistance Description (TOA) CCPG CCJB Child Care for JOBS Participants PSTF Post TANF TANF PRTF Pre‐TANF N/A TANF related Case Management Functions MAGI‐Medicaid BCCP Breast and Cervical Cancer Non‐MAGI Medicaid N/A All Service Eligibility Functions

ONE IE&ME System Functions The ONE IE&ME System is designed to include functions in the eight areas described in Figure 2 bleow. Figure 2. ONE IE&ME System Functions

Data Conversion A number of existing systems have been identified as key sources of data that will be converted and added to the ONE IE&ME system to provide a starting set of data. These sources are listed in the table below.

Figure 3. Data Conversion Source Systems

3.1.2 Project Scope Strengths Analysis of the ONE IE&ME Scope identified the following strengths:  Strong Program leadership and agreement on the scope and benefits to be achieved through the investment of time, people and dollars in the ONE IE&ME initiative  ONE IE&ME Scope is aligned with the vision of the business for best practices for serving the citizens of Oregon through the programs that are in scope for ONE IE&ME  Business Operations standards are maturing, focusing on the alignment with the envisioned model of practice and functional capabilities defined for the ONE IE&ME solution  Scope is well communicated and widely understood  Highly transparent – The latest integrated schedule covers the entire agreed scope actively managed by the PMO, including the Legacy Readiness activities

3.1.3 Project Scope Challenges and Potential Risks  Remaining ONE IE&ME Design Work – There is additional design work for ONE IE&ME yet to be completed (targeted March 2018) that may result in a need to change the level of core activities required for the Deloitte’s DDI efforts  Remaining Legacy System Design Work – There is also some additional design work on Legacy systems that may impact the resources necessary for Legacy Readiness, and possibly impact Deloitte’s DDI schedule and costs if the additional design work leads to delays in the readiness of key legacy systems that are a part of Legacy Readiness project, beyond October 1, 2018

 Defining What Is Deloitte’s “Design, Development and Implementation (DDI) Completion” — What will constitute completion of the current ONE IE&ME Implementation Project and end of DDI?  Successfully standing up ONE IE&ME to meet the envisioned Model of Practice as specified by the agreed Requirements Traceability Matrices (RTM)  Well defined Deliverable Expectation Documents (DEDs)  Acceptance and sign-off on agreed upon deliverables as defined in DEDs  Completion of comprehensive system, operational, end-to-end integration and user acceptance testing  Acceptable level of residual defects  Achievement of Data Conversion goals and Data Quality targets  Documented “waivers” on specified capabilities that have been deferred to be introduced by future enhancements post DDI completion  Understanding what remains unaddressed after Deloitte’s Completion of DDI: Optimizing ONE IE&ME for full achievement of the Eligibility Transformation Vision including the following:  Addressing residual defects  Addressing deferred functionality  Any necessary Data Quality Remediation resulting from the encountered Data Quality constraints  Further changes to support optimizing business processes and Lean and Business Process Re-engineering (BPR) objectives related to Eligibility Transformation project  Optimizing processes to ensure streamlined operations and beneficiary responsiveness  Scope Management – All Budget and Schedule commitments are highly dependent on effective scope management, control, and continuous alignment to project success factors. Given the not-to-exceed, fixed cost, deliverables-based nature of the contract, and the magnitude of the costs involved, the most critical risk related to the Deloitte DDI costs and schedule is to keep the project within the agreed scope. The related potential scope management risks and challenges that may impact the project cost and schedule include:  Adequacy of Project Management Office (PMO) Capabilities – State of Oregon may not have sufficiently defined the PMO as a Program Management Office with the related capabilities, mature processes, staffing and critical skill sets to exert adequate leadership and control and employing timely and effective risk management strategies to prevent unplanned scope changes and avert potential schedule delays and undesirable budget variances – ─ Project Governance Alignment – Project Governance and Leadership is complex for this initiative and not fully aligned in all areas impacting the success of the ONE IE&ME and envisioned business transformation. There is no overall “Program” with a scope that fully includes and integrates all projects and activities impacting ONE IE&ME efforts including the State’s Eligibility Transformation Project

─ Risk Management is not Fully Integrated – Risk analysis, mitigation, management and communications are not well integrated or prioritized across the ONE IE&ME projects and initiatives– therefore some may not get adequately managed  Dependence on Vendor Skills and Resources – Deloitte may step in to drive or influence certain critical project decisions that should be strictly reserved for the State and the project leadership

3.2 Reasonableness of Estimation Methodology

3.2.1 Estimation for Deloitte DDI Oregon has contracted with Deloitte for a contract with a not to exceed price that is driven by a statement of work and defined deliverables with a fixed price and schedule with payments based on deliverable-based events, and agreed financial penalties on both sides. The foundation of the agreement is primarily driven by the transfer of the Kentucky E&E solution’s “out of box” functionality and process flows, with an emphasis on minimum changes needed to accommodate state’s unique requirements.

Cost and Risk are Based on Scope and Dependencies Agreed prices are derived from two party negotiations, and a cost model driven by scope, schedule and estimates for each Statement of Work and related Deliverable, overlaid by proprietary commercial risk calculations and Deloitte’s established business model for delivering E&E solutions. Figure 4. Deloitte DDI Price Development Process

3.2.2 Deloitte DDI Estimation Strengths Analysis of the Deloitte process to develop estimates for the DDI work indicates the following strengths:  Deloitte is fielding a strong team, with targeted experience who are demonstrating a good understanding of the scope they must deliver  Deloitte is contractually obligated to deliver within a “not to exceed” amount, linked to statements of work and fixed deliverables  Deloitte’s ability to deliver to agreed, not-to-exceed price is strengthened by:  High level of financial motivation

 Deloitte’s contractual commitment is to satisfy the documented requirements, and with substantial experience with the configurations and customizations required, there is a very high level of confidence in the estimates  Any risk left has been offset by a healthy margin (the commercial and risk calculations)  A considerable proportion of the technical work is low-complexity configuration  Any complex development / customization work is done offshore – where increased margins apply  Deep bench-strength  Deep pockets (cash reserves)

3.2.3 Deloitte DDI Estimation Challenges and Potential Risks Analysis indicates that there are no substantial risks associated with the Deloitte estimation process. Even if Deloitte’s Estimating Methodology has unidentified flaws the State is protected from that by the nature of the contract and the financial health of Deloitte. The most critical risk related to the Deloitte DDI costs and schedule is to keep the project within the agreed scope, and aligned with clear statements of work and deliverables expectations documents.

3.2.4 Estimation for State’s ONE IE&ME Legacy Readiness Activities Oregon ONE IE&ME and Legacy Readiness teams (cross functional workgroup of 8 leads) has developed an Experience-Based estimation model that uses the seven factors identified in the table below to develop an estimate for the required effort for the full DDI lifecycle for each legacy system. Seven Factors Used for Estimation

The most experienced developer(s) provide an estimate of Dev/Test effort for each legacy system using six distinct sizes ranging from 16 to 960 hours, based on an understanding of the scope of the legacy enhancement / refactoring project and data conversion requirements.

That estimate is refined in the following steps: 1. The developer then provides a maximum size for the effort along with a confidence level 2. A Risk and Complexity score is then assigned to the planned legacy system scope Risk Factors

3. An assessment of available resources to complete the effort is completed Dev/Test Phase Effort Estimate (Assumes 65% Developer Availability)

UAT Days Based on Risks of Change

4. An assessment of Data Management resources are made 5. Finally, Gartner’s 2017 government project resource distribution model is used to extrapolate the full DDI lifecycle effort Gartner 2017 – Government IT Projects’ Effort Distribution Resource Distribution

*SDD – Solution Development and Delivery

The tables below illustrate samples of the resulting estimates which are provided in more detail in Appendix A.

Sample System Using Selected Factors, Criteria, and Model

Results of Above Sample with Criteria Applied to Gartner Resource Distribution %

The initial set of estimates for all in-scope systems

The Development and Test estimated efforts as of January 2018 appear to show an estimate that is lower than the original estimate of 21,050 hours above (now 13,000-14,000 hours).

IE Legacy Software Development Status (as of January 2018)

3.2.5 Legacy Readiness Estimation Strengths Analysis of the process to develop estimates for the Legacy Readiness work indicates the following strengths:  The State used development staff with substantial experience of making changes to the systems involved and the underlying technologies to develop the initial estimates  The estimation model takes into consideration size, risks and complexity of the system to define the labor requirements and duration of the effort

 Oregon has used industry standard Software Development Lifecycle effort distribution models to extrapolate its experience based development and testing effort estimates  Initial high-level Legacy Readiness estimates of June 2017 have been updated and integrated in the ONE IE&ME Integrated Project Schedule  The updates to the initial timeline and effort estimate from June of 2017 have been transferred and maintained within the legacy project schedule and recently the Integrated Project Schedule  These estimates continue to be updated based on highly-experienced individuals interpretation of additional information as it becomes available

3.2.6 Legacy Readiness Estimation Challenges and Potential Risks Analysis of the process to develop estimates for the Legacy Readiness work indicates the following challenges and potential risks:  The Legacy Readiness team is not employing a consistent and repeatable estimation and time tracking process  A standardized estimation model and process is not available across OIS, and any specialized model developed for specific projects will not have the opportunity to be leveraged in future projects without institutionalization of a consistent process and tools to capture the outcomes from the previous estimation effort  The Legacy Readiness team is not systematically recording and tracking estimates to be able to measure (and improve) its estimation accuracy over time and be available to provide input for further estimation efforts. This impacts the quality of the continuous process of re-estimating (of all the work already in play) and initial and subsequent estimates for work that has not yet been defined

3.3 Reasonableness of Deloitte ONE IE&ME DDI Schedule

3.3.1 ONE IE&ME Integrated Project Schedule The ONE IE&ME Integrated Schedule (illustrated in figure 5) has recently been expanded to contain Deloitte Contract Amendment 4 and other state activities outside of Deloitte’s responsibilities, and is now the key tool of the PMO to manage the overall effort.

Figure 5. ONE IE&ME Integrated Project Schedule

The Five (5) execution phases (Detailed Design, Development, Testing, Security, Infrastructure and Implementation) are the responsibility of the Deloitte team. System Design and Development work is contained in three (3) of the phases:  The Deloitte Detailed Design and Development phases  The Legacy Integration phase which is the responsibility of the State (OIS and Business)  A key dependency between the Legacy Integration Efforts and Final System Integration Testing (SIT) at the beginning of October 2018 is depicted in red The Implementation phase contains the following activities:  To ensure alignment between the new ONE IE&ME system and the new Business Operations model including any policy and process adjustments resulting from the Eligibility Transformation project: “Business Operations Implementation Services”  To make all stakeholders aware of all important aspects of the implementation and ensure all system users are adequately trained in a short time prior to them commencing use of the system either as part of the Pilot or Roll Out: “People Readiness Implementation Services”

3.3.2 Integrated Schedule Strengths  The schedule is comprehensive and detailed up to the point of implementation (7,781 tasks) covering all important aspects of the system lifecycle to prepare and implement the new ONE IE&ME System  Both plans and actuals are being actively updated and maintained on a regular basis by the PMO

3.3.3 Integrated Schedule Challenges and Potential Risks  As noted in the Scope discussion, there is additional design work yet to be completed (targeted March 2018) both on the ONE IE&ME System side and Legacy Readiness side that may result in material change such as scope change, extension of schedule, and/or increased costs  At this point there is incomplete knowledge of the Legacy Systems Data Quality and the full strategy for Data Conversion has yet to be established  Any substantial scope change in the project could impact the schedule  Although percentage of completeness is being tracked, Earned Value is not being visibly tracked and used to manage the Project and pace of value delivery and realization  A complex effort like the ONE IE&ME initiative, with considerable staff resource needs, will require a focus on the following consideration:  PMO Capabilities — State of Oregon may not have sufficiently defined the PMO as a Program Management Office with the related capabilities, mature processes, staffing and critical skill sets to exert adequate leadership and control and employing timely and effective risk management strategies to prevent unplanned scope changes and avert potential schedule delays and undesirable budget variances  Effective Resource Allocation – Management of resource contention and gaps by active management and detailed control of the schedule focusing on: ─ Legacy Design, Development and Testing activities and related impact on schedule (e.g., Legacy Readiness are led by internal resources who can be interrupted to address other OHA/DHS projects) ─ Accessing Business SMEs resources as required for the ONE IE&ME effort including UAT and deployment planning and training  Successful merging of ONE MAGI Medicaid Enhancements into Integrated ONE (current IE&ME effort) is a critical element of the Integrated Project Schedule. This will require further review to ensure that a comprehensive approach and process is employed to verify that all ONE MAGI system functionality and related Oregon enhancements have been successfully ported into the envisioned ONE IE&ME System  Schedule risks are not currently managed and integrated alongside other risks, and risks are not seen as being managed or prioritized in a holistic way – therefore some risks may not be adequately managed

3.4 Reasonableness of State’s ONE IE&ME Project Support and Legacy Readiness Projects Schedule

3.4.1 Four Phase Legacy Readiness Schedule In June of 2017, the Oregon OIS Legacy Readiness development team developed a schedule (shown in the figure below) to complete the Design, Development and Systems Integration Testing of all required legacy system changes related to ONE IE&ME in 12 months over 4 phases with assistance from Deloitte to help produce the Functional Design Documents. Figure 6. Four Phase Legacy Readiness Schedule

As the FDDs were being worked on in 2017, the ONE IE&ME team was going through a re- baselining of the overall schedule to align with the latest learnings and decisions from the FDD JAD sessions including:  The ONE IE&ME team solicited input from the Legacy Readiness Development team to properly incorporate the planned changes into the Systems Integration Test cycle  Based on the available information in December of 2017 and the planned completion of Design True Up documents in March of 2018, an agreement was developed to deliver all required Legacy Systems’ updates into Final System Integration Testing by October 1, 2018. This key date was then used to develop the re-base-lined schedule with a target completion date of July 2020 –  This provides 6 months of development and Systems Integration Testing for all in scope legacy systems’ changes. These changes will also be subject to the full Systems Integration Testing and User Acceptance Testing after October 2018 and during the subsequent 12 months.

Figure 7. Exert from the Detailed Schedule

3.4.2 Legacy Readiness Schedule Strengths Analysis of the Legacy Readiness Schedule resulted in the identification of the following strengths:  Deloitte resources with in-depth knowledge of the new ONE IE&ME solution design led the development of Functional Design Documents (FDD) for eighteen (18) of the more complex core legacy E&E and Benefit Management systems to ensure complexities were adequately addressed  A number of critical FDDs, including Interface Design Documents, Requirements Traceability Matrix, and Data Conversion plans have been completed as of December 2017  The business areas are covered by a number of deeply knowledgeable business systems analysts (BSAs) involved with Legacy Readiness  OIS technical resources are knowledgeable and highly experienced programmers and systems analysts are capable of successfully implementing the required changes. In addition, the Legacy Readiness team has added seven (7) contract developers to assist with the IE&ME project  The Development leads are confident that they will be able to meet the October 2018 target date to enter Final Systems Integration Testing (SIT) with Deloitte  The Legacy Readiness team has identified sources for suitable skilled and experience contract staff that can augment the technical development aspects of the Legacy Readiness efforts and, so far, have added seven (7) contract developers to assist with the IE&ME project.

3.4.3 Legacy Readiness Schedule Challenges and Potential Risks Analysis of the Legacy Readiness Schedule resulted in the identification of the following challenges and potential risks:  Some aspects of the Legacy Readiness design remain undefined and unspecified for an additional dozen or more of the less complex legacy systems that are within the scope of the project as of end of 2017. The Legacy Readiness Design for these additional systems is scheduled for completion in March 2018  Due to the volume and scale of simultaneous changes to all the ONE IE&ME related legacy systems, the probability of running into unforeseen obstacles that may exceed the planned timeline may impact the ONE IE&ME overall schedule and the System Integrator’s ability to fulfill their role

 As a part of the Legacy Readiness, there is limited documentation for a number of the in- scope legacy systems (which is typical for aging legacy systems) that may negatively impact the schedule for some of the legacy readiness effort and will need to be managed  The legacy Readiness team needs to add major functionality to one of the legacy COBOL Payment Issuance Systems (JV) to enable payment processing for a number of programs to continue. Therefore, the end to end process of benefit delivery for a number of Programs are now contingent on the enhancements needed to JV that have to be delivered in a relatively short period time, and not later October of 2018 for Final Systems Integration testing by Deloitte  Resource contentions for members of the Legacy Readiness team that are responsible for the ongoing operations and changes to all applications running on the Mainframe platform, when they could be “called away” to take on unanticipated day-to-day priorities and urgent requirements regarding recovery from unexpected service disruption or other mandated needs

3.4.4 Eligibility Transformation Schedule The Eligibility Transformation (ET) project was launched in third quarter of 2017 and has completed some planning sessions to develop the detailed tasks and activities required for each of the seven (7) ET Workstreams:

1. Oregon Health Plan (OHP) Integration – January 1, 2017 – January 31, 2019 – Transfer the management, budget and related systems of the OHP Eligibility Office from OHA to DHS. Develop and implement future state streamlined process to move OHP to be compliant with Federal guidelines, and then towards the “same day next day” services in preparation for the rollout of the new ONE IE&ME system 2. VEC (Virtual Eligibility Center) and Storefronts – October 28, 2017 – September 19, 2019 – Create one virtual call center with multiple rural locations, as well as multiple storefront offices State wide to serve all DHS clients regardless of how they choose to access financial benefit eligibility services 3. Central Services – October 28, 2017 – September 19, 2019 – Identify all central services impacting APD (Adults with Physical Disabilities) and Self-Sufficiency Programs (SSP) and develop recommendations for change and alignment 4. Infrastructure – October 28, 2017 – January 31, 2019 – Identify infrastructure impacted by eligibility delivery alignment and make a recommendation for changes to fit new eligibility delivery model 5. Change Management — October 28, 2017 – September 19, 2019 – Develop an engagement and communication plan for all workstreams of the project 6. SSP Eligibility Delivery and Processing Centers – January 1, 2018 – January 31, 2019 – Align SSP processes in preparation for Eligibility Delivery models. All current work across the State is as much in alignment before IE as possible. Cross train SSP processing center staff on OHP Medicaid eligibility determination to build capacity for this work. (later on APD work in the VEC portion of the project cost allocation (RMSS)) 7. APD Eligibility Delivery – January 1, 2018 – January 31, 2019 – Align APD processes in preparation for the new Eligibility Delivery model. Bring all current work across the State in alignment as much as is feasible before the roll out of the new IE system. Clarify the roles of Eligibility workers and Case Managers and develop clear hand off protocols

Figure 8. Eligibility Transformation Preliminary Timeline.

3.4.5 Eligibility Transformation Schedule Strengths Analysis of the Eligibility Transformation Schedule indicates the following strengths:  ET is championed and led by top leadership at DHS  ET efforts for ONE IE&ME is aligned with the vision of the business for best practices for serving the citizens of Oregon through the programs that are in scope for ONE IE&ME  ET Business Operations standards are focusing on the alignment with the envisioned functional capabilities of the envisioned ONE IE&ME solution  The focus of ET is to enable Oregon to use the Kentucky Benefind system’s person centered and integrated model of practice by updating Oregon’s policies, rules, practices and training to keep all customizations and changes to a minimum  ET has key business and technology representatives from across DHS, OHA, OIS and the ONE IE&ME project  ET has already brought onboard two (2) Project Managers and six (6) resources with lean subject matter expertise to help with process and practice modeling, definition and documentation effort

3.4.6 Eligibility Transformation Schedule Challenges and Potential Risks Analysis of the Eligibility Transformation Schedule indicates the following challenges and potential risks:  ET and ONE IE&ME have separate governance and reporting structures and therefore will require proactive and intentional planning and coordination by the respective leadership and staff of each Project to manage interdependencies and keep both efforts synchronized  The detailed tasks for each of the ET workstreams have not yet been fully defined  ET and ONE IE&ME interdependencies are not yet fully defined  With the current ONE IE&ME’s pre-established and re-base-lined schedule, ET may not have sufficient time to define and incorporate its requirements for an optimized Integrated Model of Practice into the ONE IE&ME finalized design documents  ET may need to postpone desired changes to the ONE IE&ME model of practice that is a part of the Kentucky Benefind solution till after completion of the initial Production Release, and after the capture of key process metrics in the centralized contact center and the different size store front operations 3.5 Reasonableness of Deloitte DDI Costs During the history of the ONE IE&ME project since 2015 there have been a number of different budget estimates as new information has become available and the full extent of the project has become better understood and fully defined. The figure below presents a summary of how these budget estimates have developed. Figure 9. ONE IE&ME History of Budget Estimates Timeline

Forecasted expenditures with Deloitte through completion of ONE IE&ME rollout:  ONE DDI* (MAGI Medicaid) $38M  Integrated ONE DDI (IE&ME DDI, plus Enhancements $164M and support for Eligibility Transformation Project)  Legacy Readiness Design Support $2M

 Total DDI Related Expenditures $204M (First 5 Years – Through 7/2020 Rollout Completion) * Note: ONE DDI Deloitte costs have been included in the analysis to provide a holistic view of total cost and scope of an integrated E&E System that is comparable to the costs presented later in this document for similar projects in other States.

ONE IE&ME Project M&O – $25M-$30M per annum based on a rough estimate (prior to a planned competitive procurement there are no vendor sourced quotations)

ONE IE&ME Project M&O – $59M Total from 2016 through June 2020

3.5.1 Deloitte DDI Costs Strengths Analysis of the Deloitte DDI Costs indicates the following strengths:  Prices are comparable with equivalent prices from Deloitte seen in other states – (see section on comparison with other States)  Current price accounts well for the level of effort and number of resources (main component of Deloitte costs) involved in supporting the work contracted for related to finalizing portions of Functional Design Documents. March 2018 Design True Up, and evolution and maturation of Eligibility Transformation project may identify new requirements that are not currently defined in Deloitte’s scope of work and related deliverables)

3.5.2 Deloitte DDI Costs Challenges and Potential Risks Analysis of the Deloitte DDI Costs indicates the following challenges and potential risks:  Given the “not to exceed,” fixed cost deliverable-based (as defined by DEDs) nature of the contract, and the magnitude of the costs involved, the most critical risk related to the Deloitte DDI Costs is to keep the project within the agreed scope  The scope boundary can be impacted by anything that causes Deloitte’s level of effort to increase, such as: ─ State adding new functional areas ─ State making major changes to current agreed to requirements ─ Delays caused by the State in not providing agreed upon tasks and deliverables such as deliverable reviews, readying system interfaces, fulfilling the State’s responsibilities for key project milestones, and/or providing agreed staffing resources as defined in the project’s schedule

 In order to mitigate their own risks, Deloitte may have a tendency to inappropriately fill leadership gaps or make key project decisions that need to be State leadership and decision making responsibility 3.6 Reasonableness of State Costs The State of Oregon has developed and maintained a project budget for ONE IE&ME that has provided for coverage of State resources such as PMO, Subject Matter Experts, and Legacy Readiness Development resources. The latest budget for the expected duration of the ONE IE&ME project is listed in the slide that follows, and here is a brief overview of expected State resources:

 State SME and Operations Resources $54M  Legacy Readiness Development Resources $9M  PMO $5M  Disaster Recovery $10M  Hosting $13M  State Resources Budget Contingency $2.6M

Total $93.6M

Based on analysis of other Integrated Eligibility projects similar to Oregon’s, and a bottom’s up analysis of required resources for Legacy Readiness, PMO, and State’s Subject Matter Experts and Operations resources, the budget amounts allocated to internal State resources are seen as sufficient for the planned and known scope of tasks through the completion of the ONE IE&ME DDI lifecycle in July of 2020.

ONE IE&ME Budget Details as of January 2018

ONE IE&ME State Budget Details

Legacy Readiness Effort Estimates (Bottom-Up Budget Validation)

Notes on the bottom-up Budget Validation: 1. Based on the latest estimates by OIS as of Jan 8, 2018 2. Assumes Average loaded hourly cost of $85 per hour 3. Percentage of FDD completion have not been confirmed, and some FDDs may not be completed by FDD True Up target date in March 4. Estimates include IT and business SME resources and the full development and deployment lifecycle 5. Revised Estimate is a Gartner provided update to the State estimate based on a number of factors that drive the confidence in estimates provided 6. See Appendix B for additional details on legacy systems Planned Budget of $9M and a contingency of $1.5M will be able to cover the high end ($6.1M) of the bottoms-up estimate.

3.6.1 State Costs Strengths Analysis of the State Costs indicates the following strengths:  The budget does plan for adequate funding for State human resources for successful deployment of ONE IE&ME Project  The PMO budget should be sufficient to allow to quickly staff up the PMO office, given a timely and successful PMO resource procurement  The bottom-up resource estimate are below the allocated budget for Legacy Readiness

3.6.2 State Costs Challenges and Potential Risks Analysis of the State Costs indicates the following challenges and potential risks:  There are limited State subject matter experts who can participate in key information gathering and decision making sessions, and even though the budget is available to pay for these resources, there can be significant contention in a number of activities for the same resources  The State has not yet been able to develop the critical PMO capabilities as of yet due to a variety of constraints  The number of expert legacy systems’ resources who can support the Legacy Readiness required effort in a relatively short period of time is limited, and therefore resource contention becomes a key challenge to overcome

3.7 Comparisons with Other States’ Similar Efforts for Consideration It is the nature of the System Integrator pricing process that makes competitive procurements the most effective approach for obtaining best value. In the absence of a true competitive process, it is possible to demonstrate relative reasonability by comparisons with experience of other similar projects. Five other States engaged on similar Integrated Eligibility projects, post the Affordable Care Act, with Deloitte as the Systems Integrator, and are described at a high- level here — due to confidentiality, we are not able to identify any of the States involved:  For each of the projects the main characteristics of the scope of the project being undertaken (especially the extent to which it differs from the Oregon project) is indicated  The status of each project, insofar as it helps to further help explain the applicability of the cost information is provided  Efforts have been made to normalize the costs to strengthen a direct comparison, but the following caveats need to be considered:  Despite all examples being recent State Integrated Eligibility procurements / implementations and the normalization efforts; each of these examples are unique with meaningful variations in scope, approach and current status for Oregon  All of these examples used a competitive bid approach to obtain and contract for Systems Integrator services  Striving for best value, the competition process and the pressure of budget review means that initially negotiated prices always significantly fall below the actual expenditures incurred during the life of projects. The examples where the numbers

are based on negotiated bids PRIOR to work commencing (contract negotiation stage) have been flagged

Cost Comparisons with Other States (using Deloitte as the Systems Integrator)

Note: *Costs PRIOR to Initial Contract Signing (Note: Actual Costs Are Only Known in Retrospect)

Analysis of the above DDI costs for these comparable projects in other States indicates that Oregon’s costs are at the low end of the scale. This is largely due to Oregon’s decision to implement the solution as a “Transfer” with minimal customization.

4.0 Prioritized Risks and Mitigation Strategies

4.1 Risk Prioritization Approach For each of the areas analyzed (Scope, Estimation Methodology, Schedule and Cost) a number of Risks were identified, prioritized and recommendations to help mitigate each risk made. Each risk was assigned a Priority Score out of 100. This score was calculated by evaluating the likelihood (probability) and impact of the Risk with scores out of 10. The Priority Score is the product of the likelihood score and impact score. Based on the Priority Score each risk was assigned to a Priority Tier as follows:

 High – Risks with a Priority Score of 45 or more; 12 Risks identified  Medium – Risks with a Priority Score greater than 20 but less than 45; 12 Risks identified  Low – Risks with a Priority Score of 20 or less; 4 Risks identified

The risks are organized into 3 categories for further analysis and development of key recommendations:

 Process Risks:  Those mainly related to the many project and project management processes 16 Risks put into this category  People Risks:  Those mainly related to human resource issues; 6 Risks put into this category  Product Risks:  Those mainly related to the products resulting from project activities; 6 Risks put into this category. A complete prioritized list of all risks along with suggested mitigation strategies is provided in Appendix A to this report.

4.2 Process-Related Risks

Process-Related Risks, Scoring and Mitigation Strategies

Process Related Risks Risk Risk Name & Likelihood Impact Priority Priority Recommended # Description Score (L-M-H) Mitigation Strategy

14 Resource Contention – 8 8 64 H Formally remove all Resource contention for non-IE&ME related Legacy Design, demands on the Development and Testing Legacy Readiness activities and related development team impact on schedule (e.g. until October 2018. Legacy Readiness are led by internal resources who can be interrupted to address other OHA/DHS projects)

18 Resource Redirection - 8 8 64 H Formally remove all The Legacy Readiness non-IE&ME related team is responsible for demands on the the ongoing operations Legacy Readiness and changes to all development team applications running on until October 2018 the Mainframe platform, (see Risk #14). and therefore can be “called away” to take on unanticipated day to day priorities and urgent requirements regarding recovery from unexpected service disruption or other mandated needs

Process Related Risks 2 Remaining Legacy 7 9 63 H Further develop System Design Work – contingency plans by There is also additional identifying any work design work on Legacy that is not absolutely systems that may impact necessary to meet the resources necessary the October 1 for Legacy Readiness, deadline and and possibly may have additional sources of an impact on Deloitte’s contract developers DDI schedule and costs if to further augment it delays remediation of the Legacy key legacy systems that Readiness team if are a part of Legacy this becomes Readiness project, necessary. beyond October 1, 2018

27 Limited State Subject 7 9 63 H Continue with Matter Experts - There establishing a formal are limited State subject process to identify matter experts who can weekly activities and participate in key resource information gathering and requirements at a decision-making detailed level for the sessions, and even next 8-12 weeks, and though the budget is work to avoid any available to pay for these duplicate booking of resources, there can be key resources by significant contention in a adjusting schedule number of activities for and timing for the same resources meetings.

12 Data Quality - At this 8 7 56 H Place special point there is incomplete emphasis on Data knowledge of the Legacy Profiling for legacy Systems Data Quality systems whose data and the full strategy for is essential to be Data Conversion has yet migrated to IE&ME to be established System, and identify the systems whose data represent the most significant risk to successful deployment, and develop mitigation plans to address the potential risks in 2018.

Process Related Risks 16 Merging of ONE and 8 7 56 H Develop detailed IE&ME -Successful traceability matrices merging of ONE MAGI and plans for testing Medicaid Enhancements and verification of into Integrated ONE incorporation of ONE (IE&ME) is a critical Enhancements to the element of the Integrated original release. Project Schedule. This Develop key entry will require further clarity and exit criteria for regarding a each stage in the comprehensive approach testing life cycle and and process to verify that pilot. all ONE MAGI system functionality and related Oregon enhancements have been successfully ported into ONE IE&ME System

22 ET and ONE IE&ME 8 7 56 H Ensure business Coordination and sponsors and Governance - ET and executives are ONE IE&ME have regularly engaged in separate governance and discussions related to reporting structures and status and therefore will require interdependencies proactive and intentional between the two planning and projects, and provide coordination by the the appropriate respective leadership and resources to ensure staff of each Project to success, and decide manage on best path forward interdependencies and based on discussion keep both efforts of trade offs (see also synchronized Risk #7).

Process Related Risks 1 Remaining IE&ME 7 7 49 H Review and update Design Work – There is impact of final design additional design work for changes on schedule ONE IE&ME yet to be and budget in April completed (targeted 2018 and identify March 2018) that may strategies to fit new result in a need to level of effort within change the level of core existing schedule and activities required for the budget. Deloitte’s DDI efforts

6 Integrated Risk 7 7 49 H Conduct a thorough Management – Risk assessment of the analysis, mitigation, Risk Management management and processes. Ensure communications are not that this results in well integrated or strengthened and prioritized across the fully integrated different aspects of the processes with a high project – therefore some level of transparency. do not get adequately managed Develop an integrated risk management process that properly identifies high complexity and high uncertainty tasks that have a potential to negatively impact the latest DDI schedule, and ensure executive business leadership in decisions related to managing those risks.

Process Related Risks 11 Reasonableness of 7 6 42 M Be prepared to IE&ME Schedule judiciously evaluate - The additional design any change that will work yet to be completed have an impact on (target completion of the latest schedule March 2018) both on the published in IE&ME System side and December 2017, and Legacy Integration side eliminate sufficient may result in material number of non-critical change in schedule due / optional functionality to scope or level of or tasks to maintain required effort resulting in fidelity to the latest increased costs schedule. -

21 Limited Historical 6 7 42 M Conduct daily Legacy Documentation standup meetings - As a part of the Legacy between March and Readiness, there is October of 2018 to limited documentation for monitor progress and a number of the in-scope discuss legacy systems that may documentation negatively impact the related issues that schedule for some of the each of the legacy legacy remediation effort development teams may have run into, and assign issue owners and facilitator to expedite resolution of the same. 20 Unanticipated Legacy 5 7 35 M Conduct daily Remediation standup meetings Challenges - Due to the between March and volume and scale of October of 2018 to simultaneous changes to monitor progress and all the IE&ME related identify unanticipated legacy systems, the issues that each of probability of running into the legacy unforeseen obstacles development teams that may exceed the may have run into, planned timeline may be and assign issue material owners and facilitator to expedite resolution of the same.

Process Related Risks 8 Over Dependence on 6 5 30 M Strengthen Vendor Vendor Skills and Management with Resources – Deloitte, metrics that go driven by its own beyond those commercial imperatives, necessary for and fielding experienced Contract compliance. and assertive staff, may Have the QA vendor step in to drive or monitor the scope, influence certain critical focus, and extent of project decisions that the System should be strictly Integrator's role. reserved for the State and the project Ensure that all key leadership decisions are transparently and fully dealt with by the agreed governance processes.

Ensure that any Vendor recommendations and guidance are balanced and/or validated against third party advice and recommendations. 23 ET Project 5 6 30 M Complete the Requirements and detailed planning for Interdependencies - ET workstreams by The detailed tasks for March 2018, identify each of the ET all interdependencies workstreams have not yet and specific been fully defined. ET capabilities that need and ONE IE&ME to be delivered by interdependencies are IE&ME once in not yet fully defined production, and assess and manage the potential impact of those requirements on the planned IE&ME schedule and timeline.

Process Related Risks 3 Not Completely 4 5 20 L Update the formal Defining What Is Scope description Deloitte’s “Design, and submit for Development and approval by the Joint Implementation (DDI) Governance Board. Completion” What will constitute completion of the current ONE IE&ME Implementation Project and end of DDI?

4 Understanding what 4 5 20 L Include within the remains unaddressed formal Scope after Deloitte’s description and Completion of DDI: approval process Optimizing IE&ME for full identification of what achievement of the will remain Eligibility Transformation unaddressed after Vision Deloitte's Completion of DDI and an analysis of the impact of this remaining unaddressed.

4.3 People-Related Risks People-Related Risks, Scoring and Mitigation Strategies

People-Related Risks Risk Risk Name & Description Likelihood Impact Priority Priority Recommended # Score (L-M-H) Mitigation Strategy

7 Project Governance 8 7 56 H Consider Alignment – Project mechanisms to Governance and integrate the Leadership is complex and governance and not fully aligned in all areas oversight of key with the Transformation tasks and activities Vision. There is no overall defined by the ET Program with a scope that project into an overall fully includes the State’s IE/ME Program Eligibility Transformation Office. Lean / Business Process Reengineering Project Employing a clearly integrated with ONE Program IE&ME Implementation. Management Office model will keep a strong focus on an enterprise approach for the ONE IE&ME initiative and ensure effective sequencing of the planning, requirements and procurement efforts in accomplishing the eligibility transformation envisaged, help contain risks and maximize efficiency in the allocation and use of resources (see also Risk #22).

People-Related Risks 28 Number of Expert Legacy 7 7 49 H Continue to define Resources - The number and refine the of expert legacy systems resource resources who can support requirements for the required Legacy legacy system Readiness effort needed in remediation activities a relatively short period of with additional level time is limited, and of granularity and therefore resource resource contention becomes a key assignments, so that challenge to overcome the potential needs of multiple legacy systems for the same key experts during the same time frame becomes clear, and can be avoided.

13 Earned Value Reporting 7 6 42 M Incorporate earned for Project Control - value reporting (i.e. Although expenditure and planned and earned percentage of value matrices) that completeness is being shows the value tracked, Earned Value is received for the not being visibly tracked budget expended to and used to manage the date into the regular Project and pace of value reporting for the delivery in relations to project. expenditure.

15 Resource Shortage for 6 7 42 M Develop detailed test Testing - Potential for plans for required resource gap for Business resources during the SMEs and UAT testers for UAT to ensure the IE&ME project adherence to schedule.

People-Related Risks 26 DDI Contract Scope 5 8 40 M Ensure that scope is Management - Given the tightly managed by “not to exceed”, fixed cost the Business deliverable-based (as Sponsors and defined by DEDs) nature of Executives, and any the contract, and the changes are magnitude of the costs thoroughly vetted to involved, the most critical understand the risk related to the Deloitte impact on cost and DDI Costs is to keep the schedule (see also project within the agreed Risk #9). scope

People-Related Risks 10 Internal Software 8 2 16 L Initiate a project Development & Systems within OIS to Integration Effort formalize Estimation Process development and - The State is not publishing of employing a consistent and estimates at major repeatable estimation and milestones within time tracking process large projects (i.e. - The Legacy Readiness greater than $0.5M), team are not systematically capture each recording and tracking estimate in an AD estimates to be able to Repository, and measure (and improve) conduct actual vs. their estimation accuracy estimate post over time and be available mortem analysis at to provide input for further the completion of the estimation efforts. This project against each impacts the quality of the milestone estimate. continuous process of re- Establish formal time estimating (of all the work tracking for all IT already in play) and initial Project activities and and subsequent estimates resources to have a for work that has not yet factual baseline for been defined. the effort extended. - A standardized estimation model and process is not available across OIS, and any specialized model developed for specific projects will not have the opportunity to be leveraged in future projects without institutionalization of a consistent process and tools to capture the outcomes from the previous estimation effort

4.4 Product-Related Risks Product-Related Risks, Scoring and Mitigation Strategies

Product-Related Risks Risk Risk Name & Likelihood Impact Priority Priority Recommended # Description Score (L-M-H) Mitigation Strategy

19 Legacy Payment 6 8 48 H Vigorously monitor and Issuance System - The control the scope of legacy Readiness team required functionality, needs to add major and ensure appropriate functionality to one of the level of resources are legacy Cobol Payment assigned to ensure Issuance Systems (JV) timely completion and to enable payment testing of the new processes for a number functionality. of programs to continue

9 Scope Management - 5 8 40 M Ensure that anything Given the not-to-exceed, that threatens the fixed cost, deliverables- Scope that is validated, based nature of the agreed, and approved contract, and the (see items 3 and 4 magnitude of the costs above) is escalated to involved, the most the executive level of critical risk related to the the Governance Deloitte DDI Costs is to structure (see also Risk keep the project within #26). the agreed scope

24 Model of Practice 5 7 35 M ET should establish key Requirements - With business outcome the current ONE metrics (e.g. IE&ME’s pre-established processing times by and re-baselined program, case worker schedule, ET may not productivity, data have sufficient time to quality, etc.) related to define and incorporate the new integrated its requirements for an delivery model, and optimized Integrated continue on the path of Model of Practice into minimizing any the IE&ME finalized changes to Benefind's design documents functionality and capabilities available out of the box.

Product-Related Risks 25 Model of Practice 7 5 35 M ET should establish key Misalignment - ET may business outcome need to postpone metrics (e.g. desired changes to the processing times by ONE IE&ME model of program, case worker practice that is a part of productivity, data the Kentucky Benefind quality, etc.) related to solution till after the new integrated completion of the initial delivery model, and Production Release, and continue on the path of subsequent to the minimizing any capture of key process changes to Benefind's metrics in the centralized functionality and contact center and the capabilities available different size store front out of the box. operations

5 Adequacy of Project 4 7 28 M Consider moving to a Management Office Program Management (PMO) Capabilities – Office model for the State of Oregon may not ONE IE&ME initiative. have sufficient capabilities, mature Complete the processes, and critical procurement of skill sets in the PMO to additional PMO exert adequate control resources as soon as and employ timely and possible. effective risk management strategies To get the most form to avert potential these additional schedule delays and available resources undesirable budget complete a skills gap variances assessment across the PMO with consideration of migrating to a Program Management Office model. Use the additional available resources and staff development to address these gaps urgently.

Product-Related Risks 17 Legacy FDD 5 4 20 L Based on the March Completion - The State 2018 deadline to Legacy Readiness team complete all legacy needs to develop FDDs system FDDs, assess for an additional dozen the Legacy System or more of the less team's ability to meet complex legacy systems the October 2018 that are within the scope deadline for entry into of the project as of end Final SIT. of 2017. Since all legacy systems need to be ready for Final SIT in October of 2018, this effort may have a negative impact on schedule.

Summary of High Tier Risks and Mitigation Strategies  The critical (High Tier) risks to ONE IE&ME schedule and budget include: 1. Resource Allocation – Completion of all Legacy Readiness work by October of 2018 with potential issues related to resource contention and/or redirection 2. Anticipated Design True Up – Remaining ONE IE&ME and Legacy Readiness Design and Design True Up targeted for March 2018 can potentially impact the required resources and/or timeline 3. Business SME Availability – Availability of Business Subject Matter Exerts to effectively participate in all design discussions and decision-making sessions about the future operating model 4. Integrated Governance and Management – The project faces challenges with effective and integrated Program Governance and Management and Risk Management in that not all projects and initiatives impacting the success of the ONE IE&ME effort are clearly under an integrated governance model with clear decision-making authority. Migrating the current Project Management Office to a Program Management Office should be considered 5. Data Quality – The extent of legacy systems’ Data Quality issues are not yet well defined, and has the potential to require significant manual work to ensure successful transition of applicants to the ONE IE&ME system 6. ONE and Integrated ONE Merge – Verification of approach to and successful merging of enhancements to ONE functionality into ONE IE&ME requires further clarity, documentation and reviews

 Gartner recommended risk mitigation strategies include: 1. Resource Allocation – Develop strategies to assign appropriate subject matter experts (SMEs) resources to ONE IE&ME and Legacy Readiness with significant lead time to have SMEs available on a just in time basis, while avoiding over- allocating certain resources, and minimizing disruptions from other high priority, non-project related activities. This will require a comprehensive integrated work plan with resource loading to provide significant advance notice of when required SME resources are needed and maintain fidelity to the work plan 2. Anticipated Design True Up – Employ strong project governance and scope management techniques to identify ways to minimize the impact of any new work resulting from the Design True Up activities in March 2018, and reduce scope in other non-essential areas of the project to achieve a neutral impact on the ONE IE&ME overall project cost and schedule 3. Business SME Availability – Continue with establishing a formal process to identify weekly activities and resource requirements at a detailed level for the next 8-12 weeks, and work to avoid any duplicate booking of key resources by adjusting schedule and timing for meetings 4. Integrated Governance and Management – Consider mechanisms to integrate the governance and oversight of key tasks and activities defined by the ET project into the overall ONE IE&ME initiative to enforce proper integration and prioritization of tasks that impact the final model of practice being deployed. This effort can be enhanced by moving the Project Management Office approach to that of a Program Management Office model to include all projects and initiatives impacting the ONE IE&ME initiative under the Program Management Office 5. Data Quality – Place special emphasis on Data Profiling for legacy systems whose data is essential to be migrated to ONE IE&ME System, and identify the systems whose data represent the most significant risk to successful deployment, and develop mitigation plans to address the potential risks during 2018 6. ONE and Integrated ONE IE&ME Merge – Develop detailed traceability matrices and plans for testing and verification of incorporation of ONE Enhancements into the ONE IE&ME’s Benefind release. Develop key entry and exit criteria for each stage in the testing life cycle and pilot

5.0 Conclusions and Recommendations

5.1 Schedule and Budget Estimates Assessment Conclusions Based on the detailed review of the ONE IE&ME Project Business Case, Vision and Scope, the processes used by State and Deloitte teams to develop the schedule and budget, as well as the latest Integrated Project Schedule and the State’s overall budget published in December of 2017, it is Gartner’s opinion that:  The ONE IE&ME project incorporates the critical tasks and activities required to complete a robust Integrated Eligibility (IE) Solution for State of Oregon aligned with the State’s defined scope  The Solution selected by Oregon (Deloitte’s Benefind from Kentucky) is a proven solution that has been deployed into Production already in Kentucky (Full Integrated Eligibility) and Oregon (MAGI Medicaid), and its functional and technical components originate from the Systems Integrators’ work in IE for a number of other states in the last 12-15 years  The revised $164M Contract with Deloitte for Design, Development, and Implementation of Benefind is within the range of magnitude of investments observed by Gartner in other states that have invested in Integrated Eligibility solutions of similar scope in the last 7 years. In fact, it is on the low end of the scale based on Oregon’s decision to implement the solution as a “Transfer” with minimal customization  The State’s Legacy Readiness budget of $9.2M and an additional $1.4M in contingency funds is sufficient to cover the expenses related to the legacy systems remediation effort based on the completed planning and design work to date  The overall ONE IE&ME DDI budget of $345M (State and SI Costs) is on the low end of the range of total project expenditure observed in other states for comparable projects  Given the level of planning completed to date, Gartner’s view of the current budget and schedule forecast is that it is adequately revised and expanded to be within a margin of ±15% of the actual costs to achieve the deployment of current scope of functionality. This conclusion is based on Gartner’s review of completed planning and design documents for ONE IE&ME and Legacy Readiness, as well as industry research in to the quality and accuracy of project estimates at the end of current Systems Development lifecycle stage  State of Oregon should expect to require additional effort and expenditure for optimizing processes and functionality (Optimization Phase) in the first five years post the state- wide go live date, separate from Maintenance and Operations and Enhancement related expenditures  The revised ONE IE&ME schedule with a target completion date of July 2020 is a reasonable schedule in comparison to other projects of similar scale and scope in other states, and will provide State of Oregon with sufficient time to conduct adequate testing  The robust rules and policy management capabilities planned as a part of the ONE IE&ME solution can be expected to minimize the type of Medicaid audit findings that have been identified in previous years for OHA

5.2 Schedule and Budget Estimates Assessment Recommendations There are many aspects to the management and successful execution of such a large and complex effort like ONE IE&ME. From the Gartner review and findings, the State has in place many of the critical components essential for success. There are two key pillars for the continued success of the ONE IE&ME initiative and the State’s ability to address the challenges and risks identified through this third-party assessment of scope, schedule and budget. Figure 10 below depicts how the State can strengthen the ONE IE&ME effort through focused attention on the two key pillars in place to ensure continued momentum towards success and to identify, anticipate, and mitigate risks going forward — including what are identified as challenges/risks and related recommendations in this report. The two key pillars are:  Program Integration — Identification of clear roles and responsibilities for the Executive Governance Body (Joint Governance Board), Independent Project Director, and movement toward expanding the current Project Management Office to a Program Management Office (PMO) that will include all of the projects and initiatives that impact the ONE IE&ME initiative— focusing on Integrated Work plans; Resource Loading; Management of dependencies and Interdependencies; Proactive Resource Planning to mitigate resource contentions; Risk and Issues identification-logging-management  Contract and Vendor Management – The State has a contract in place with the System Integrator that reflects best practices in fixed price contracting. The State’s contract is based on a capped not to exceed amount driven by statements of work and deliverables expectation documents (DEDs) for Fixed Price Deliverables. Building on this contract structure, the State can enhance both contract and vendor management efforts to strengthen the ability to bring the ONE IE&ME project in on-schedule and within budget Figure 10. Pillars for ONE IE&ME Success

The following recommendations along with the suggested risk mitigation strategies presented in the body of this report will support the State of Oregon in moving forward with the successful delivery of the ONE IE&ME project: 1. Systems Integrator Partnership – State of Oregon should proceed with the implementation of ONE IE&ME project in collaboration with Deloitte. Gartner does not see any barriers or risks that would necessitate reconsideration of moving forward with the ONE IE&ME initiative 2. Program Management Office – The State of Oregon should consider elevating the current Project Management Office to a Program Management Office model with the related capabilities, mature processes, staffing and critical skill sets to exert adequate leadership and control over the project, and employ timely and effective risk management strategies over all of the projects and initiatives that impact the success of the One IE&ME effort, preventing any unplanned scope changes and avoid schedule delays 3. Strong Vendor Management and Performance Metrics – The Program Management Office, in collaboration with the Quality Assurance Vendor, must attempt to establish strong metrics to hold the Systems Integration vendor accountable to deliverable quality and timelines, and the State staff to conducting the reviews and sign-offs on a timely basis. This should be addressed through ensuring comprehensive Statements of Work and robust Deliverables Expectation Documents based on industry best practices for deliverable review and acceptance criteria 4. Reducing Data Quality Risk Impact – Since the deployment of ONE IE&ME is planned through a three wave, incremental geographical basis, it requires that any data quality and data conversion issues for all in-scope programs to be uncovered and resolved during a single pilot phase. ONE IE&ME project leadership should conduct a formal analysis of alternative approaches to deployment and data conversion as a contingency deployment strategy, including consideration of a multi-phase deployment strategy on a Program by Program (or Program Groups) basis, that could reduce the likelihood or extent of data quality issues that may arise 5. Eligibility Related Legacy Systems Evolution and Retirement – DHS leadership should continue to evaluate strategies to transform the family of Eligibility-related legacy mainframe and PowerBuilder based application systems that will not be replaced by Integrated ONE, with a view to enhance the underlying technology and integration architecture, in order to minimize complexity, and reduce technical debt and DHS/OHA IT infrastructure and application costs. As there are substantial data model differences between these legacy systems and Integrated ONE, even minor changes could require considerable time and cost to implement. The evolution of these legacy systems should enable a more functional, streamlined and less expensive model for functional enhancements to Integrated ONE going forward 6. M&O Strategy – State of Oregon should conduct a formal analysis of alternatives for the best value sourcing approach to Integrated ONE Application Maintenance and Operations (M&O) responsibilities. This analysis should consider a broad range of alternatives including a portion or a majority of these responsibilities being assigned to internal MS .NET Application Development resources vs. hiring an external vendor. Given the estimated $24M plus per year in Integrated ONE Application M&O costs, this will present the potential of considerable savings to DHS from a Total Cost of Ownership perspective, as well as additional employment opportunities for Oregonians

Appendices

Appendix A — Listing of Prioritized Risks

Risks to Schedule and Budget in Order of Priority (High to Low)

Prioritized Risks Risk Risk Name & Likelihoo Impac Priorit Priority Recommended Mitigation # Description d t y (L-M-H) Strategy Score 14 Resource Contention 8 8 64 H Formally remove all non- – Resource contention IE&ME related demands on for Legacy Design, the Legacy Readiness Development and development team until Testing activities and October 2018. related impact on schedule (e.g., Legacy Readiness are led by internal resources who can be interrupted to address other OHA/DHS projects) 18 Resource 8 8 64 H Formally remove all non- Redirection — The IE&ME related demands on Legacy Readiness the Legacy Readiness team is responsible for development team until the ongoing October 2018 (see Risk operations and #14). changes to all applications running on the Mainframe platform, and therefore can be “called away” to take on unanticipated day to day priorities and urgent requirements regarding recovery from unexpected service disruption or other mandated needs 2 Remaining Legacy 7 9 63 H Further develop System Design Work contingency plans by – There is also identifying any work that is additional design work not absolutely necessary to on Legacy systems meet the October 1 that may impact the deadline and additional resources necessary sources of contract for Legacy Readiness, developers to further and possibly may augment the Legacy have an impact on Readiness team if this Deloitte’s DDI becomes necessary. schedule and costs if it delays remediation of key legacy systems

Prioritized Risks that are a part of Legacy Readiness project, beyond October 1, 2018

27 Limited State 7 9 63 H Continue with establishing Subject Matter a formal process to identify Experts — There are weekly activities and limited State subject resource requirements at a matter experts who detailed level for the next 8- can participate in key 12 weeks, and work to information gathering avoid any duplicate booking and decision making of key resources by sessions, and even adjusting schedule and though the budget is timing for meetings. available to pay for these resources, there can be significant contention in a number of activities for the same resources 7 Project Governance 8 7 56 H Consider mechanisms to Alignment – Project integrate the governance Governance and and oversight of key tasks Leadership is complex and activities defined by the and not fully aligned in ET project into an overall all areas with the IE/ME Program Office. Transformation Vision. There is no overall Such a Program Office Program with a scope would keep focus on an that fully includes the enterprise approach and State’s Eligibility ensure effective Transformation Lean / sequencing of the planning, Business Process Re- requirements and engineering Project procurement clearly integrated with efforts in accomplishing the ONE IE&ME eligibility transformation Implementation. envisaged, help contain risks and maximize efficiency in the allocation and use of resources (see also Risk #22).

Prioritized Risks 12 Data Quality — At 8 7 56 H Placed special emphasis this point there is on Data Profiling for legacy incomplete knowledge systems whose data is of the Legacy Systems essential to be migrated to Data Quality and the IE&ME System, and full strategy for Data identify the systems whose Conversion has yet to data represent the most be established significant risk to successful deployment, and develop mitigation plans to address the potential risks in 2018. 16 Merging of ONE and 8 7 56 H Develop detailed IE&ME-Successful traceability matrices and merging of ONE MAGI plans for testing and Medicaid verification of incorporation Enhancements into of ONE Enhancements to Integrated ONE the original release. (IE&ME) is a critical Develop key entry and exit element of the criteria for each stage in the Integrated Project testing life cycle and pilot. Schedule. This will require further clarity regarding a comprehensive approach and process to verify that all ONE MAGI system functionality and related Oregon enhancements have been successfully ported into ONE IE&ME System 22 ET and ONE IE&ME 8 7 56 H Ensure business sponsors Coordination and and executives are Governance — ET regularly engaged in and ONE IE&ME have discussions related to separate governance status and and reporting interdependencies between structures and the two projects, and therefore will require provide the appropriate proactive and resources to ensure intentional planning success, and decide on and coordination by best path forward based on the respective discussion of tradeoffs (see leadership and staff of also Risk #7). each Project to manage interdependencies and keep both efforts synchronized

Prioritized Risks 1 Remaining IE&ME 7 7 49 H Review and update impact Design Work – There of final design changes on is additional design schedule and budget in work for ONE IE&ME April 2018 and identify yet to be completed strategies to fit new level of (targeted March 2018) effort within existing that may result in a schedule and budget. need to change the level of core activities required for the Deloitte’s DDI efforts 6 Integrated Risk 7 7 49 H Conduct a thorough Management – Risk assessment of the Risk analysis, mitigation, Management processes. management and Ensure that this results in communications are strengthened and fully not well integrated or integrated processes with a prioritized across the high level of transparency. different aspects of the project – therefore Develop an integrated risk some do not get management process that adequately managed properly identifies high complexity and high uncertainty tasks that have a potential to negatively impact the latest DDI schedule, and ensure executive business leadership in decisions related to managing those risks.

28 Number of Expert 7 7 49 H Continue to define and Legacy Resources — refine the resource The number of expert requirements for legacy legacy systems system remediation resources who can activities with additional support the required level of granularity and Legacy Readiness resource assignments, so effort needed in a that the potential needs of relatively short period multiple legacy systems for of time is limited, and the same key experts therefore resource during the same time frame contention becomes a becomes clear, and can be key challenge to avoided. overcome

Prioritized Risks 19 Legacy Payment 6 8 48 H Vigorously monitor and Issuance System — control the scope of The legacy Readiness required functionality, and team needs to add ensure appropriate level of major functionality to resources are assigned to one of the legacy ensure timely completion COBOL Payment and testing of the new Issuance Systems functionality. (JV) to enable payment processes for a number of programs to continue 11 Reasonableness of 7 6 42 M Be prepared to judiciously IE&ME Schedule evaluate any change that - The additional design will have an impact on the work yet to be latest schedule published in completed (target December 2017, and completion of March eliminate sufficient number 2018) both on the of non-critical / optional IE&ME System side functionality or tasks to and Legacy maintain fidelity to the latest Integration side may schedule. result in material change in schedule due to scope or level of required effort resulting in increased costs - 13 Earned Value 7 6 42 M Incorporate earned value Reporting for Project reporting (i.e., planned and Control — Although earned value matrices) that expenditure and shows the value received percentage of for the budget expended to completeness is being date into the regular tracked, Earned Value reporting for the project. is not being visibly tracked and used to manage the Project and pace of value delivery in relations to expenditure. 15 Resource Shortage 6 7 42 M Develop detailed test plans for Testing — for required resources Potential for resource during the UAT to ensure gap for Business adherence to schedule. SMEs and UAT testers for the IE&ME project

Prioritized Risks 21 Limited Historical 6 7 42 M Conduct daily standup Legacy meetings between March Documentation — As and October of 2018 to a part of the Legacy monitor progress and Readiness, there is discuss documentation limited documentation related issues that each of for a number of the in- the legacy development scope legacy systems teams may have run into, that may negatively and assign issue owners impact the schedule and facilitator to expedite for some of the legacy resolution of the same. remediation effort 9 Scope Management 5 8 40 M Ensure that anything that — Given the not-to- threatens the Scope that is exceed, fixed cost, validated, agreed, and deliverables-based approved (see items 3 and nature of the contract, 4 above) is escalated to the and the magnitude of executive level of the the costs involved, the Governance structure (see most critical risk also Risk #26). related to the Deloitte DDI Costs is to keep the project within the agreed scope 26 DDI Contract Scope 5 8 40 M Ensure that scope is tightly Management — managed by the Business Given the “not to Sponsors and Executives, exceed,” fixed cost and any changes are deliverable-based (as thoroughly vetted to defined by DEDs) understand the impact on nature of the contract, cost and schedule (see and the magnitude of also Risk #9). the costs involved, the most critical risk related to the Deloitte DDI Costs is to keep the project within the agreed scope 20 Unanticipated 5 7 35 M Conduct daily standup Legacy Remediation meetings between March Challenges — Due to and October of 2018 to the volume and scale monitor progress and of simultaneous identify unanticipated changes to all the issues that each of the IE&ME related legacy legacy development teams systems, the may have run into, and probability of running assign issue owners and into unforeseen facilitator to expedite obstacles that may resolution of the same. exceed the planned timeline may be material

Prioritized Risks 24 Model of Practice 5 7 35 M ET should establish key Requirements — business outcome metrics With the current ONE (e.g., processing times by IE&ME’s pre- program, case worker established and re- productivity, data quality, baselined schedule, etc.) related to the new ET may not have integrated delivery model, sufficient time to and continue on the path of define and incorporate minimizing any changes to its requirements for an Benefind's functionality and optimized Integrated capabilities available out of Model of Practice into the box. the IE&ME finalized design documents 25 Model of Practice 7 5 35 M ET should establish key Misalignment — ET business outcome metrics may need to postpone (e.g., processing times by desired changes to the program, case worker ONE IE&ME model of productivity, data quality, practice that is a part etc.) related to the new of the Kentucky integrated delivery model, Benefind solution till and continue on the path of after completion of the minimizing any changes to initial Production Benefind's functionality and Release, and capabilities available out of subsequent to the the box. capture of key process metrics in the centralized contact center and the different size store front operations 8 Over Dependence on 6 5 30 M Strengthen Vendor Vendor Skills and Management with metrics Resources – Deloitte, that go well beyond those driven by its own necessary for Contract commercial compliance. Have the QA imperatives, and vendor monitor the scope, fielding experienced focus, and extent of the and assertive staff, System Integrator's role. may step in to drive or influence certain Ensure that all important critical project decisions are transparently decisions that should and fully dealt with by the be strictly reserved for agreed governance the State and the processes. project leadership Ensure that any Vendor recommendations and guidance are balanced and/or validated against third party advice and recommendations.

Prioritized Risks 23 ET Project 5 6 30 M Complete the detailed Requirements and planning for ET Interdependencies — workstreams by March The detailed tasks for 2018, identify all each of the ET interdependencies and workstreams have not specific capabilities that yet been fully defined. need to be delivered by ET and ONE IE&ME IE&ME once in production, interdependencies are and assess and manage not yet fully defined the potential impact of those requirements on the planned IE&ME schedule and timeline. 5 Adequacy of Project 4 7 28 M Complete the procurement Management Office of additional PMO (PMO) Capabilities – resources as soon as State of Oregon may possible. not have sufficient capabilities, mature To get the most form these processes, and critical additional available skill sets in the PMO resources complete a skills to exert adequate gap assessment across the control and employ PMO. Use the additional timely and effective available resources and risk management staff development to strategies to avert address these gaps potential schedule urgently. delays and undesirable budget variances 3 Not Completely 4 5 20 L Update the formal Scope Defining What Is description and submit for Deloitte’s “Design, approval by the Joint Development and Governance Board. Implementation (DDI) Completion” What will constitute completion of the current ONE IE&ME Implementation Project and end of DDI? 4 Understanding what 4 5 20 L Include within the formal remains Scope description and unaddressed after approval process Deloitte’s identification of what will Completion of DDI: remain unaddressed after Optimizing IE&ME for Deloitte's Completion of full achievement of the DDI and an analysis of the Eligibility impact of this remaining Transformation Vision unaddressed.

Prioritized Risks 17 Legacy FDD 5 4 20 L Based on the March 2018 Completion — The deadline to complete all State Legacy legacy system FDDs, Readiness team assess the Legacy System needs to develop team's ability to meet the FDDs for an additional October 2018 deadline for dozen or more of the entry into Final SIT. less complex legacy systems that are within the scope of the project as of end of 2017. Since all legacy systems need to be ready for Final SIT in October of 2018, this effort may have a negative impact on schedule. 10 Internal Software 8 2 16 L Initiate a project within OIS Development & to formalize development Systems Integration and publishing of estimates Effort Estimation at major milestones within Process large projects (i.e., greater - The State is not than $0.5M), capture each employing a estimate in an AD consistent and Repository, and conduct repeatable estimation actual vs. estimate post and time tracking mortem analysis at the process completion of the project - The Legacy against each milestone Readiness team are estimate. Establish formal not systematically time tracking for all IT recording and tracking Project activities and estimates to be able to resources to have a factual measure (and baseline for the effort improve) their extended. estimation accuracy over time and be available to provide input for further estimation efforts. This impacts the quality of the continuous process of re-estimating (of all the work already in play) and initial and subsequent estimates for work that has not yet been defined. - A standardized estimation model and process is not available across OIS,

Prioritized Risks and any specialized model developed for specific projects will not have the opportunity to be leveraged in future projects without institutionalization of a consistent process and tools to capture the outcomes from the previous estimation effort

Appendix B — Legacy Readiness Scope and Effort

System Documentation Availability

Note: The above is a point in time representation of the status of DDI related documentation for the Legacy Systems in scope of Legacy Readiness portion of the ONE IE&ME project, as of January 2018. This list is being continuously updated as additional documentations related to Legacy Readiness are produced.

Legacy Readiness Effort Estimates (Bottom-Up Budget Validation)

Aging and People with Disability Systems

Self Sufficiency Systems

Figure 11. JV (Payment Issuance) Being Updated to Replace a Number of Legacy Payment Issuance Systems

Appendix C — Gartner’s Project Approach and Data Discovery

Foundation for Gartner’s Approach The Gartner team proposed for the review and validation effort, will leverage and apply:  Gartner’s Research around large IT project’s estimation best practices  Gartner Public Sector Health and Human Services Experience and Lesson Learned from work on similar integrated eligibility and enrollment and benefits management projects, including those that included the System Integrator for the ONE IE/ME project. Examples of such experiences include: ─ Arkansas, Florida, Georgia, Tennessee and Texas – Forensic Assessment of the states’ troubled Eligibility & Enrolment Projects ─ Connecticut, Florida, Georgia, Hawaii, Ohio and Tennessee – Integrated Health and Human Services Eligibility & Enrollment Go to Market Readiness Assessment, Requirements Development and Procurement Life Cycle Support ─ Vermont — Integrated Health and Human Services Eligibility & Enrollment Project Full Life Cycle Support – Planning, Requirements, Procurement Life Cycle Support and Quality Assurance / Independent Verification & Validation (QA / IV&V) Oversight (Includes Integrated Eligibility; Health Insurance Exchange, Integrated Health and Human Services (HHS) Solution Pattern, Care Management and Medicaid Modernization)

Approach Summary  Discovery – The Discovery phase of the effort will focus on Document Review and Key Stakeholder (State and System Integrator) Structured Interviews focusing on such things as (not meant to be fully inclusive):  ONE IE/ME Scope  Functional Requirements Development and Validation Process and Status  Non-Functional Requirements Development and Validation (Technical and Architecture Standards, Implementation and Software Development Life Cycle Approach, Performance and Service Levels, etc.) Process and Status  Detailed System Design (Functional and Technical) Development and Validation Process and Status  Interface Mapping and Development Plans  Legacy Systems Refactoring, Remediation, Enhancements, and Modernization Plans  ONE IE/ME Organizational Structure and Staffing (State and System Integrator)  System Integration Master Work Plan and Work Breakdown Structure  System Integrator’s Budget, including Labor Category and Rates, Staff Density and, if included in the Budget – Hardware, Environments, Software Licenses, etc.  Estimation Methodology(ies) Employed – Including Assumptions and Constraints (ONE and In-Scope Legacy Systems)

 Detailed Analysis of Schedule and Budget Analysis – Based on the results of the Discovery Phase; the application of Gartner HHS Team’s experience and lessons learned; Gartner Research; and Industry Best Practices for Schedule and Budget Estimation, complete a review and assessment and develop draft analysis and findings for State review and validation  Draft and Final One IE/ME Schedule and Budget Review Assessment and Validation Report – Based on the validated analysis and findings, develop a Draft Report for the State review and validation and based on State input and recommendation complete the Final Report that will include: – Executive Summary – Purpose of the Review and Assessment – Assumptions and Constraints – Methodology – Analysis Findings – Risks Identification – Recommendations

Figure 12. Gartner Project Three Phase Approach

 The project schedule will be finalized in partnership with the State to meet the February 2018 report back to the legislature  Efforts will be applied to accelerate the project activities including doing several tasks in parallel

 Gartner will leverage Gartner Research as appropriate through-out the project’s life cycle

Figure 13. Gartner Project – Preliminary Draft Schedule

Figure 14. List of Interviewees — Over 35 Individual and Group Interviews Conducted

Over 180 Project Related Documents Reviewed

1. "Joint Governance Board Charter ONE Integrated Eligibility & Medicaid Eligibility Project" 2. "DEL 2.2.1.1" — Functional Requirements Traceability Matrix - RTM 3. "DEL2.2.1.1.2 Functional Requirements Traceability Matrix (RTM) – Update – Iteration 2July 10, 2017Version 1.1" 4. Org Chart 5. Scope summary excerpt from the Business Case 6. ONE IE & ME Project Business Case 7. "Integrated Eligibility Project Implementation Advanced Planning Document (IAPD) Annual Update Federal Fiscal Year 2018" 8. ONE IE & ME Integrated Project Schedule 9. "Deliverable 2.2.4: Technology Management Plan (Design Phase Update)" 10. Copy of Legacy Design Deliverables tracker +JAD status_2017-10-19 11. Legacy Artifacts table_2017-10-20 12. Legacy Project 13. IE_Overview_January_2016_JWMIT_Presentation 14. Legacy_AJ_Interface_Control_Document 15. Legacy_AJ_Requirements_Traceability_Matrix 16. Legacy_AJ_System_Detailed_Design

17. Legacy_CBC_Interface_Control_Document_APPROVED 18. Legacy_CBC_Requirements_Traceability_Matrix_APPROVED 19. Legacy_CBC_System_Detailed_Design_APPROVED 20. Legacy_CEP_Interface_Control_Document_APPROVED 21. Legacy_CEP_Requirements_Traceability_Matrix_APPROVED 22. Legacy_CEP_System_Detailed_Design_APPROVED 23. Legacy_CI_Data_Conversion_Design_APPROVED 24. Legacy_CI_Interface_Control_Document_APPROVED 25. Legacy_CI_Requirements_Traceability_Matrix_APPROVED 26. Legacy_CI_System_Detailed_Design_APPROVED 27. Legacy_CM_Data_Conversion_Design 28. Legacy_CM_Requirements_Traceability_Matrix 29. Legacy_CM_System_Detailed_Design 30. Legacy_CPI_Interface_Control_Document_APPROVED 31. Legacy_CPI_Requirements_Traceability_Matrix_APPROVED 32. Legacy_CPI_System_Detailed_Design_APPROVED 33. Legacy_CRAI_Requirements_Traceability_Matrix 34. Legacy_CRAI_System_Detailed_Design 35. Legacy_EB_Requirements_Traceability_Matrix 36. Legacy_EB_System_Detailed_Design 37. Legacy_FR(ACF 204)_Requirements_Traceability_Matrix 38. Legacy_FR(ACF196R)_System_Detailed_Design 39. Legacy_FR(ACF204)_System_Detailed_Design 40. Legacy_FR(ACF801)_System_Detailed_Design 41. Legacy_FR_Requirements_Traceability_Matrix 42. Legacy_FR_System_Detailed_Design 43. Legacy_FSMIS_Data_Conversion_Design 44. Legacy_FSMIS_Requirements_Traceability_Matrix 45. Legacy_FSMIS_System_Detailed_Design 46. Legacy_JV_Interface_Control_Document 47. Legacy_JV_Requirements_Traceability_Matrix 48. Legacy_JV_System_Detailed_Design 49. Legacy_OA_Data_Conversion_Design 50. Legacy_OA_Interface_Control_Document 51. Legacy_OA_Requirements_Traceability_Matrix 52. Legacy_OA_System_Detailed_Design 53. Legacy_PP_Interface_Control_Document

54. Legacy_PP_Requirements_Traceability_Matrix 55. Legacy_PP_System_Detailed_Design 56. Legacy_SA_Interface_Control_Document 57. Legacy_SA_Requirements_Traceability_Matrix 58. Legacy_SA_System_Detailed_Design 59. Legacy_SJC_Interface_Control_Document_APPROVED 60. Legacy_SJC_Requirements_Traceability_Matrix_APPROVED 61. Legacy_SJC_System_Detailed_Design_APPROVED 62. Legacy_SP_Requirements_Traceability_Matrix 63. Legacy_SP_System_Detailed_Design 64. Legacy_TRACS_Requirements_Traceability_Matrix_APPROVED 65. Legacy_TRACS_System_Detailed_Design_APPROVED 66. Steering Committee Project Charter 20170725 67. ONE IE and ME Integrated Schedule IPS 68. ONE IE and ME September Quality Assurance Report 69. Legacy Schedule_20171205 70. Legacy Project - Estimating 71. Assessment Overview 72. Del 1.5b Legacy Process Assessment Report v1.0 73. IE_Legacy_Scedule_draft_001-5 74. Overview_draft2 75. ESS Audit Process 76. IE ME PMP v1.5 77. IE_CommPlan_1.5_Signed 78. IE Proper JAD Tracker 79. Legacy Design Deliverables tracker +JAD status 80. 8-24-17 IE-Legacy-DR Detailed Spend Plans by FFY 81. 8-28-17 IE-Legacy-DR Detailed Spend Plans by FFY wo contingency 82. 2016-02-01 Approval of OR SNAP DDI QA Amendments 83. 2016-02-02 Approval of OR SNAP IAPD 84. 2017-09-22 Approval of OR SNAP ONE IE ME Project Annual IAPDU-signed 85. A9 State Sole Source Exception Request Final 86. CAM TOOL_OR_IE_Draft_Projections_MA_SNAP_TANF_ERDC_Draft After Fit Gap 87. Copy of IE 17-19 072216 with case 2 DDI 88. IAPD Doc 89. IAPD FFY18-19 8-30-17 Updated Oregon Response 90. OR 2017 OAPD Approval Letter Final 1

91. OR APD Approval Letter_as_of_2016-03-04 92. OR Deloitte Approval Letter 11-17-17 93. OR_2017_07_26_EE_APD_Approval_Letter 94. Deloitte_SOW4 one pager_2016-06-10 95. IT_Proj_Stage_Gate_FuncRefModel 96. DASPS-1402-15 - OHA MAGI Medicaid EE SI & SOW1 97. DASPS-1402-15 Amendment 1 - SOW2 98. DASPS-1402-15 Amendment 2 - SOW3 99. DASPS-1402-15 Amendment 3 100. DASPS-1402-15 Amendment 4 - SOW4 101. DASPS-1402-15 Amendment 5 102. DASPS-1402-15 SOW1 CO1 103. DASPS-1402-15 SOW1 CO2 104. DASPS-1402-15 SOW1 CO3 105. DASPS-1402-15 SOW1 CO4 106. DASPS-1402-15 SOW1 CO5 107. DASPS-1402-15 SOW1 CO6 108. DASPS-1402-15 SOW1 CO7 109. DASPS-1402-15 SOW1 CO8 110. DASPS-1402-15 SOW3 CO1 111. DASPS-1402-15 SOW3 CO2 112. DASPS-1402-15 SOW3 CO3 113. DASPS-1402-15 SOW3 CO4 114. DASPS-1402-15 SOW3 CO5 115. DASPS-1402-15 SOW3 CO6 116. DASPS-1402-15 SOW4 CO1 117. DASPS-1402-15 SOW4 CO2 118. DASPS-1402-15 SOW4 CO3 119. DASPS-1402-15 SOW4 CO4 120. DASPS-1402-15 SOW4 CO5 121. DASPS-1402-15 SOW4 CO6 122. DASPS-1402-15 SOW4 CO7 123. DASPS-1439-16 Amendment 1 124. DASPS-1439-16 Amendment 1_Appendix F_RTM 125. DASPS-1439-16 Amendment 2_SOW3 126. DASPS-1439-16 Amendment 3 127. DASPS-1439-16 DHS Integrated Eligibility SI and Support

128. DASPS-1439-16 SOW2 CO1 129. DASPS-1439-16 SOW2 CO2 130. DASPS-1439-16 SOW2 CO3 131. DASPS-1439-16 SOW2 CO4 132. DASPS-1439-16 SOW3 CO1 133. DASPS-1439-16 SOW3 CO2 134. Deloitte_Amd_4_Final 135. Deloitte_SOW4 one pager_2016-06-10 136. History Document (2018 needed) 137. POP 201 Updated-Rob and Nate 138. DEL2.1.3.32_DEL3.1.2.17 Bi-Weekly Status Report_2017-12-08_v1.1 139. DEL2.2.5 Data Conversion Design (DCD)_2017-11-21_v1.3 140. DEL2.2.5 Data Conversion Design - RTM_2017-11-13_v1.2 141. DEL2.2.6 Database Design Data Dictionary Logical Design and Physical Design (Preliminary)_2017-12-09_v1.4 142. DHS IE DDI Integrated Project Schedule_20171219 143. DEL2.1.1 Project Management Framework and Plan_2017-01-04_v1.3 144. Generic Governance Layers Presentation KNW 20171116 2 145. Legacy Project -Payment System Options - Final v1.1 146. OISApplications 147. SDLC Process 148. DHS IE DDI Integrated Project Schedule (IPS)_2017-12-21_dc 149. RFP DASPS-2184-17-PM Consulting services 150. RFP-DASPS-2184-17 Addendum(1)_12-14-17 151. SP-810-15_Special Procurement_Approval 152. SP-810-15_Special_Procurement_Amendment__1 153. OIS Practices from Debbie Estabrook_compiled document 154. QA D5.1 Initial Risk Assessment v1.2 155. 3.1.6 October Quarterly Status and Improvement Report v0.9 156. 3.1.5 July Quarterly v1.0 157. 3.1.4 April Quarterly Report v1.0 158. 3.1.3 Quarterly QA Status Report v1.0 159. 3.1.2 Quarterly QA Status Report v1.0 160. 3.1 Quarterly QA Status Report Factual Review Draft v1.0 161. 2.3b(12) September Quality Status Report v1.0 162. 2.3b(10) June Monthly Report v1.0 163. 2.3b(9) May Monthly Report v1.0

164. 2.3b(8) March Monthly Report v1.0 165. 2.3b(7) February Monthly Report v1.0 166. 2.3b(6) December Monthly Report v1.1 167. 2.3b(5) November Monthly Report v1.1 168. 2.3b(4) Monthly Quality Status Report v1.1-1 169. 2.2.2 FDD Activity Summary Report v1.0 170. 1.5b Legacy Process Assessment Report v1.0 171. 2.3b(11) August Monthly Report v1.0 172. 2017-12-21 Integrated Project Schedule (IPS) 173. 2017-12-21 IPS - Critical Path View 174. 2017-12-21 IPS - Milestone List View 175. 2017-12-21 IPS - Overall Master Plan 176. 2017-12-21 IPS - Revisions in Progress 177. Integrated Project Schedule (IPS) Amendment 4 Overview 178. 2017.12.13_ET_CharterPlan_DraftV3 179. HL_ET_TimelineV8 180. OHP_Integration_Timeline_Draft 181. IE and Legacy Details Nov 2017 182. IE Mthly Nov 2017 01032018 183. FDD Baseline 184. FDD Update TrueUp 185. Budget Policy Option Package 811 LegApprovedBudget 186. Budget_Staffing tables 187. Budget_TCO as of October 2017 draft 121217b (002) 188. IE Mthly Oct 12122017a w 50-50 PMO draft (version 1)

Any questions regarding this Report should be addressed to: Frank Petrus Senior Managing Partner Gartner, Inc. Telephone: +1 617-851-6800 Email: [email protected]

This Report was prepared for Oregon OSCIO, DHS & OHA : Alex Z. Pettit State Chief Information Officer Oregon OSCIO Telephone: +1 503-378-2128 Email: [email protected]