United States Government Accountability Office

Report to Congressional Committees

May 2017 NASA

Assessments of Major Projects

GAO-17-303SP

May 2017 NASA Assessments of Major Projects

Highlights of GAO-17-303SP, a report to congressional committees

Why GAO Did This Study What GAO Found This report provides GAO’s annual The cost and schedule performance of the National Aeronautics and Space snapshot for 2017 of how well NASA is Administration’s (NASA) portfolio of major projects continued to improve, as planning and executing its major shown in the figure below, but this trend may be difficult to sustain. The current acquisition projects. In March 2016, trend is driven by two main factors: (1) most projects are being executed within GAO found that projects continued a their cost and schedule baselines; and (2) new projects, which are less likely to general positive trend of limiting cost have experienced cost and schedule growth, were added to the portfolio. and schedule growth, maturing However, two projects—a Mars seismology instrument and lander and an technologies, and stabilizing designs, upgrade to the NASA’s space communications network—experienced significant but that NASA faced several cost or schedule growth in 2016. In addition, eight projects are in the phase of challenges that could affect its ability to development when cost and schedule problems are most likely to occur, effectively manage its portfolio. including the Orion Multi-Purpose Crew Vehicle and Space Launch System, The explanatory statement of the NASA’s most expensive human spaceflight programs. House Committee on Appropriations accompanying the Omnibus NASA’s Major Project Portfolio Cost and Schedule Performance Has Continued to Improve Appropriations Act, 2009 included a provision for GAO to prepare status reports on selected large-scale NASA programs, projects, and activities. This is GAO’s ninth annual assessment. This report describes (1) the cost and schedule performance of NASA’s portfolio of major projects, (2) the maturity of technologies and stability of project designs at key milestones, and (3) NASA’s progress in implementing initiatives to manage acquisition risk and potential challenges for project management and oversight. This report also includes assessments of NASA’s 21 major projects, each with a life-cycle cost of over $250 million. To NASA has generally maintained recent improvements in the technology maturity conduct its review, GAO analyzed cost, and design stability of its projects, even as the number of new technologies in its schedule, technology maturity, design most recent projects has increased. Three of the four major projects that passed stability, and other data; reviewed preliminary design review in the past year matured all technologies to the level monthly project status reports; and recommended by GAO best practices. Several NASA projects experienced late interviewed NASA officials. design changes, but the overall level of these changes remained relatively low and other design stability measures remained unchanged. What GAO Recommends NASA continues to improve project management tools to manage acquisition In prior reports, GAO has made related risks but faces workforce and funding challenges. NASA has not implemented a recommendations that NASA generally best practice for monitoring contractor performance, as GAO recommended in agreed with, but has not yet fully November 2012, due to resource constraints. Other NASA workforce analyses addressed. GAO continues to believe have identified gaps in key areas, such as scheduling. NASA dissolved its they should be fully addressed. NASA independent assessment office in October 2015, in part to bolster its mission generally agreed with GAO’s findings. directorate and center workforces. GAO is monitoring the effect this change could have on project oversight. Finally, several major projects experienced View GAO-17-303SP. For more information, funding-related challenges, such as working to schedules that were not contact Cristina Chaplain at (202) 512-4841 or supported by NASA’s budget plans, which could affect cost and schedule [email protected]. performance.

United States Government Accountability Office

Contents

Letter 1 Background 3 Portfolio Cost and Schedule Performance Continues to Improve 9 NASA Has Generally Maintained Improvements in Technology Maturity and Design Stability 20 NASA Has Implemented New Tools to Reduce Acquisition Risks, but Workforce and Funding Challenges Are Emerging Issues 32 Project Assessments 37 Asteroid Redirect Robotic Mission 39 Commercial Crew Program 41 Europa Clipper 43 Exploration Ground Systems 45 Gravity Recovery and Climate Experiment Follow-On 47 Ice, Cloud, and Land Elevation Satellite-2 49 Interior Exploration using Seismic Investigations, Geodesy, and Heat Transport 51 Ionospheric Connection Explorer 53 James Webb Space Telescope 55 Landsat 9 57 Mars 2020 59 NASA ISRO – Synthetic Aperture Radar 61 Orion Multi-Purpose Crew Vehicle 63 Plankton, Aerosol, Cloud, ocean Ecosystem 65 Radiation Budget Instrument 67 Solar Probe Plus 69 Space Launch System 71 Space Network Ground Segment Sustainment 73 Surface Water and Ocean Topography 75 Transiting Exoplanet Survey Satellite 77 Wide-Field Infrared Survey Telescope 79 Agency Comments and Our Evaluation 81

Appendix I Objectives, Scope, and Methodology 83

Appendix II Major NASA Projects Reviewed in GAO’s Annual Assessments 91

Appendix III Technology Readiness Levels 94

Page i GAO-17-303SP Assessments of Major NASA Projects

Appendix IV Elements of a Sound Business Case 96

Appendix V Comments from the National Aeronautics and Space Administration 98

Appendix VI GAO Contact and Staff Acknowledgments 100

Tables Table 1: Development Cost and Schedule Performance of Selected Major NASA Projects Currently in the Implementation Phase 15 Table 2: Current Projects between Critical Design Review and Systems Integration Review 17 Table 3: Development Cost Changes from Project Confirmation to February 2017 for Select Science Mission Directorate Projects 18 Table 4: Projects Developing Critical Technologies by Confirmation Year through 2016 25 Table 5: Characteristics of NASA Major Projects That Launched within Their Cost and Schedule Baselines from 2009 through 2017 30 Table 6: Characteristics of NASA Major Projects That Have Rebaselined from 2007 through 2017 31 Table 7: Characteristics of Technology Readiness Levels 94

Figures Figure 1: NASA’s Life Cycle for Space Flight Projects 3 Figure 2: Notional Distribution of Cost Reserves for a Project Budgeted at the 70 Percent Confidence Level 5 Figure 3: Major NASA Projects Reviewed in GAO’s 2017 Assessment 8 Figure 4: Development Cost Performance and Average Launch Delay for Major NASA Projects from 2009 through 2017 10 Figure 5: Total Number and Development Cost Growth for NASA Major Projects with Established Cost Baselines from 2009 through 2017 12

Page ii GAO-17-303SP Assessments of Major NASA Projects

Figure 6: Average Age of NASA Major Projects from 2009 through 2017 13 Figure 7: Number of NASA’s Major Projects Attaining Technology Maturity by Preliminary Design Review from 2010 through 2017 22 Figure 8: Average Number of Critical Technologies Reported by NASA for Major Projects in Development from 2009 through 2017 24 Figure 9: Average Percentage of Engineering Drawing Growth after Critical Design Review for NASA Major Projects from 2010 through 2017 27 Figure 10: Number of NASA Major Projects That Released over 90 Percent of Engineering Drawings at Critical Design Review from 2010 through 2017 28 Figure 11: Illustration of a Sample Project Assessment 38 Figure 12: Major NASA Projects Reviewed in GAO’s Annual Assessments 92

Abbreviations

ARRM Asteroid Redirect Robotic Mission ATLAS Advanced Topographic Laser Altimeter System CCP Commercial Crew Program CNES Centre National d’Etudes Spatiales DCI data collection instrument DHU data handling unit DLR German Aerospace Center EGS Exploration Ground Systems EM-1 Exploration Mission 1 EM-2 Exploration Mission 2 ESM European Service Module ESA European Space Agency EVM earned value management GFAS Ground Flight Application Software GFZ German Research Centre for Geosciences GRACE-FO Gravity Recovery and Climate Experiment Follow-On GSLV Geosynchronous Satellite Launch Vehicle HP3 Heat Flow and Physical Properties Probe ICESat-2 Ice, Cloud, and Land Elevation Satellite-2 ICON Ionospheric Connection Explorer

Page iii GAO-17-303SP Assessments of Major NASA Projects

InSight Interior Exploration using Seismic Investigations, Geodesy, and Heat Transport ISRO Indian Space Research Organisation ISS International Space Station JCL joint cost and schedule confidence level JPSS-2 Joint Polar Satellite System 2 JWST James Webb Space Telescope KaRIn Ka-band Radar Interferometer KDP key decision point NASA National Aeronautics and Space Administration NISAR NASA ISRO Synthetic Aperture Radar NOAA National Oceanic and Atmospheric Administration NPR NASA Procedural Requirements OCI Ocean Color Instrument OLI-2 Operational Land Imager 2 Orion Orion Multi-Purpose Crew Vehicle OSIRIS-REx Origins-Spectral Interpretation-Resource Identification Security-Regolith Explorer PACE Plankton, Aerosol, Cloud, ocean Ecosystem RBI Radiation Budget Instrument RFU radio frequency unit SCaN Space Communication and Navigation SEIS Seismic Experiment for Interior Structure SEP Solar Electric Propulsion SGSS Space Network Ground Segment Sustainment SLS Space Launch System SMAP Soil Moisture Active Passive SPP Solar Probe Plus SWOT Surface Water and Ocean Topography TESS Transiting Exoplanet Survey Satellite TIRS-2 Thermal Infrared Sensor 2 ULA United Launch Alliance WFIRST Wide-Field Infrared Survey Telescope

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.

Page iv GAO-17-303SP Assessments of Major NASA Projects Letter

441 G St. N.W. Washington, DC 20548

May 16, 2017

Congressional Committees

In fiscal year 2017, the National Aeronautics and Space Administration (NASA) plans to spend over $6 billion on 22 major projects, with each having a life-cycle cost of over $250 million. In total, these projects represent an expected investment of over $59 billion to continue exploring Earth and the solar system as well as extending human presence beyond low Earth orbit. This report provides an overview of NASA’s planning and execution of these major acquisitions—an area that has been on GAO’s high risk list since 1990.1 It includes assessments of NASA’s key projects across mission areas, such as the Space Launch System for human exploration, Mars 2020 for planetary science, and Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2) for Earth science.

The explanatory statement of the House Committee on Appropriations accompanying the Omnibus Appropriations Act, 2009 includes a provision for us to prepare project status reports on selected large-scale NASA programs, projects, and activities.2 This is our ninth annual report responding to that mandate. This report assesses (1) the cost and schedule performance of NASA’s portfolio of major projects, (2) the maturity of critical technologies and stability of project designs at key points in the development process, and (3) NASA’s progress in implementing initiatives to reduce acquisition risk and potential challenges that could affect project management, execution, and oversight. This report also includes individual assessments of 21 major NASA projects. When NASA determines that a project has an estimated life-cycle cost of over $250 million, we include that project in our annual review up through launch or completion. There are 22 major projects, but we did not develop an assessment for the Origins-Spectral Interpretation-Resource Identification-Security-Regolith Explorer (OSIRIS-REx) project because it launched in September 2016.

1GAO, High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others, GAO-17-317 (Washington, D.C.: Feb. 15, 2017). 2See Explanatory Statement, 155 Cong. Rec. H1653, 1824-25 (daily ed., Feb. 23, 2009), on H.R. 1105, the Omnibus Appropriations Act, 2009, which became Pub. L. No. 111-8. In this report, we refer to these projects as major projects rather than large-scale projects as this is the term used by NASA.

Page 1 GAO-17-303SP Assessments of Major NASA Projects

To assess the cost and schedule performance, technology maturity, and design stability of NASA’s major projects, we collected information on these areas from projects using a data collection instrument, analyzed projects’ monthly status reports, interviewed NASA project and headquarters officials, and reviewed project documentation. There are 22 major projects in total, but the information available depends on where a project is in its life cycle.3 For the 16 projects in the implementation phase we compared current cost and schedule estimates to their original cost and schedule baselines, identified the number of technologies being developed and assessed their technology maturity against GAO-identified best practices and NASA policy, and compared the number of releasable design drawings at the critical design review against GAO-identified best practices and analyzed subsequent design drawings changes. We reviewed historical data on cost and schedule performance, technology maturity, and design stability for major projects from our prior reports and compared it to the performance of NASA’s current portfolio of major projects. We also analyzed prime operations cost data for projects that have launched and were included in our prior annual reviews of NASA’s major projects. To assess NASA’s progress in reducing acquisition management risk, we met with officials and analyzed information pertinent to ongoing NASA initiatives, including its efforts to improve cost and schedule estimation and earned value management capabilities. We also followed up on other potential acquisition management challenges that we identified during our review, such as NASA’s transition to a new independent assessment model, emerging workforce issues, and project funding and budget phasing. Finally, to conduct our project assessments, we analyzed information provided by project officials, such as monthly status reports, and interviewed project officials to identify major sources of risk and the strategies that projects are using to mitigate them. Appendix I contains detailed information on our scope and methodology.

We conducted this performance audit from April 2016 to May 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our

3Five projects were in an early stage of development called formulation when there are still unknowns about requirements, technology, and design. For those projects, we reported preliminary cost ranges and schedule estimates. The Commercial Crew Program has a tailored project life cycle and project management requirements. As a result, it was excluded from our cost and schedule performance, technology maturity, and design stability analyses.

Page 2 GAO-17-303SP Assessments of Major NASA Projects

findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

The life cycle for NASA space flight projects consists of two phases— Background formulation, which takes a project from concept to preliminary design, and implementation, which includes building, launching, and operating the system, among other activities. NASA further divides formulation and implementation into phase A through phase F. Major projects must get approval from senior NASA officials at key decision points before they can enter each new phase. Figure 1 depicts NASA’s life cycle for space flight projects.

Figure 1: NASA’s Life Cycle for Space Flight Projects

Project formulation consists of phases A and B, during which the projects develop and define requirements, cost and schedule estimates, and the system’s design for implementation. NASA Procedural Requirements 7120.5E, NASA Space Flight Program and Project Management Requirements, specifies that during formulation, the project must complete a formulation agreement to establish the technical and acquisition work that needs to be conducted during this phase and define the schedule and funding requirements for that work. The formulation

Page 3 GAO-17-303SP Assessments of Major NASA Projects

agreement should identify new technologies and their planned development, the use of heritage technologies, risk mitigation plans, and testing plans to ensure that technologies will work as intended in a relevant environment. Prior to entering phase B, projects develop a range of the project’s expected cost and schedule which is used to inform the budget planning for that project. During Phase B, the project also develops programmatic measures and technical leading indicators, which track various project metrics such as requirement changes, staffing demands, and mass and power utilization. Near the end of formulation, leading up to the preliminary design review, the project team completes technology development and its preliminary design.

Formulation culminates in a review at key decision point C, known as project confirmation, where cost and schedule baselines are established and documented in the decision memorandum. The decision memorandum outlines the management agreement and the agency baseline commitment. The management agreement can be viewed as a contract between the agency and the project manager. The project manager has the authority to manage the project within the parameters outlined in the agreement. The agency baseline commitment includes the cost and schedule baselines against which the agency’s performance on a project may be measured.

To inform the management agreement and the agency baseline commitment, each project with a life-cycle cost estimated to be greater than $250 million must also develop a joint cost and schedule confidence level (JCL). The JCL initiative, adopted in January 2009, produces a point-in-time estimate that includes, among other things, all cost and schedule elements in phases A through D, incorporates and quantifies known risks, assesses the impacts of cost and schedule to date, and addresses available annual resources. NASA policy requires that projects be baselined and budgeted at the 70 percent confidence level and funded at a level equivalent to at least the 50 percent confidence level for the project.4

4NASA Procedural Requirements (NPR) 7120.5E, NASA Space Flight Program and Project Management Requirements paras 2.4.4 and 2.4.4.2 (Aug. 14, 2012) (hereinafter cited as NPR 7120.5E (Aug. 14, 2012)). The decision authority for a project can approve it to move forward at less than the 70 percent confidence level. That decision must be justified and documented.

Page 4 GAO-17-303SP Assessments of Major NASA Projects

The management agreement and agency baseline commitment include cost and schedule reserves held at the project and NASA headquarters- level, respectively.5 Cost reserves are for costs that are expected to be incurred—for instance, to address project risks—but are not yet allocated to a specific part of the project. Schedule reserves are extra time in project schedules that can be allocated to specific activities, elements, and major subsystems to mitigate delays or address unforeseen risks. Project-held cost and schedule reserves are within the project manager’s control. If the project requires additional time or money beyond the management agreement—for example, if a project needs additional funds for an issue outside of the project’s control—NASA headquarters may allocate headquarters-held reserves. Figure 2 notionally depicts how NASA would distribute cost reserves for a project that was baselined in accordance with its JCL policy.

Figure 2: Notional Distribution of Cost Reserves for a Project Budgeted at the 70 Percent Confidence Level

The total amount of cost and schedule reserves held at the project level varies based on where the project is in its life cycle. Four of the six NASA centers, which are responsible for managing 18 of the 22 major projects,

5NASA refers to cost reserves as unallocated future expenses.

Page 5 GAO-17-303SP Assessments of Major NASA Projects

require or recommend that projects hold a certain level of reserves at key project milestones.6 For example, at the Goddard Space Flight Center, projects are required to hold cost reserves equal to at least 25 percent of the estimated cost remaining at the project confirmation review, and 10 percent at the time of delivery to the launch site. Projects track their reserves between phases to help ensure they hold reserves consistent with these requirements.

At project confirmation, NASA approves a project budget profile. This includes how much funding is needed each fiscal year through project closeout to manage the project to the agency baseline commitment. According to GAO’s Cost Estimating and Assessment Guide, project cost estimates are time phased because they usually span many years.7 Time phasing spreads a program’s expected costs over the years in which they are anticipated to aid in developing a proper budget. Depending on the activities in the schedule for each year, some years may have more costs than others. When a project’s budget does not align with its budget profile for a given fiscal year, it can experience budget phasing issues. We have previously found that budget phasing issues can lead to development delays and cost increases.8

After a project is confirmed, it begins implementation, consisting of phases C, D, E, and F. In this report, we refer to projects in phase C and D as being in development. A critical design review is held during the latter half of phase C in order to determine if the design is stable enough to support proceeding with the final design and fabrication. After the critical design review and just prior to beginning phase D, the project

6NASA, Goddard Procedural Requirements 7120.7A, Schedule and Budget Margins for Flight Projects (Feb. 28, 2017); Marshall Procedural Requirements 7120.1, Marshall Space Flight Center Engineering and Program/Project Management Requirements (Aug. 26, 2014); Langley Research Center, Space Flight Project Practices Handbook, LPR 7120.5 B-1 (Mar. 17, 2014); and Jet Propulsion Laboratory, Flight Project Practices, Rev. 8 (Oct. 6, 2010). The Kennedy Space Center and Johnson Space Center do not have center-specific guidance for reserves. The Johns Hopkins University Applied Physics Laboratory manages the Solar Probe Plus (SPP) project and has guidelines for schedule reserves, but not for cost reserves. The Johns Hopkins University Applied Physics Laboratory SD-QP-012, Rev. b, Space Exploration Sector (SES) Quality Procedure: Earned Value Management System (EVMS) Project Management Control System (PMCS) (Apr. 4, 2017). 7GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: Mar. 2, 2009). 8GAO-09-3SP.

Page 6 GAO-17-303SP Assessments of Major NASA Projects

completes a system integration review to evaluate the readiness of the project and associated supporting infrastructure to begin system assembly, integration and test. In phase D, the project performs system assembly, integration, test, and launch activities. Phases E and F consist of operations and sustainment and project closeout.

NASA Projects Reviewed NASA’s portfolio of major projects ranges from satellites equipped with in GAO’s Annual advanced sensors to study the Earth to a rover that plans to collect soil or Assessment rock samples on Mars to telescopes intended to explore the universe to spacecraft to transport humans and cargo beyond low-Earth orbit. When NASA determines that a project will have an estimated life-cycle cost of more than $250 million, we include that project in our annual review. After a project launches or reaches full operational capability, we no longer include an assessment of it in our annual report. This report includes assessments of 21 major NASA projects. Four projects are being assessed for the first time this year: Landsat 9; Plankton, Aerosol, Cloud, ocean Ecosystem (PACE); the Radiation Budget Instrument (RBI); and Wide-Field Infrared Survey Telescope (WFIRST). We did not develop a project assessment for one major project, the OSIRIS-REx project, because it launched in September 2016. Figure 3 includes more information on the 21 projects we assessed. Appendix II includes a list of all the projects that we have reviewed from 2009 to 2017.

Page 7 GAO-17-303SP Assessments of Major NASA Projects

Figure 3: Major NASA Projects Reviewed in GAO’s 2017 Assessment

Note: The life cycle for NASA space flight projects consists of two phases—formulation, which takes a project from concept to preliminary design, and implementation, which includes building, launching, and operating the system, among other activities. aThe ICESat-2 project experienced technical challenges with its instrument and the project has determined it will not launch by its committed launch readiness date of June 2018. Information on a new launch readiness date was not available at the time of our review.

Page 8 GAO-17-303SP Assessments of Major NASA Projects

bIn 2016, NASA reclassified the Space Network Ground Segment Sustainment (SGSS) as a hybrid sustainment effort, rather than a major project. A hybrid sustainment effort still includes development work. As a result, we continue to include SGSS in our assessment. Cost and schedule information in the figure reflects SGSS’s 2015 approved baseline. Its current cost and schedule are under review. cBoth Commercial Crew Program contractors have experienced delays and will not be able to meet their original 2017 certification dates and both expect certification to be delayed until 2018. The Commercial Crew Program is implementing a tailored version of NASA’s space flight project life cycle, but it is currently completing development activities typically associated with implementation.

NASA has other projects in formulation that have a life-cycle cost estimate greater than $250 million that were not included in our assessment this year because they are technology demonstration missions. They include the Restore-L satellite servicing and Laser Communication Relay Demonstration projects within the Space Technology Mission Directorate, and X-plane projects with the Aeronautics Research Mission Directorate. We may include these projects in future reviews, but before doing so, we wanted to gain a better understanding of how these projects are being managed by NASA.

The cost and schedule performance of NASA’s portfolio of major projects Portfolio Cost and continues to improve. The trend has been driven by two main factors: Schedule most projects are being executed within their cost and schedule baselines and new projects, which are less likely to have experienced cost and Performance schedule growth, have been added to the portfolio. NASA could have Continues to Improve difficulty sustaining this trend. Half of its major projects in development are at the point in the acquisition process when cost and schedule problems are most likely to occur. These projects include the largest projects in the portfolio—the Orion Multi-Purpose Crew Vehicle (Orion) and Space Launch System—both of which face significant development risk. Lastly, while NASA has improved its development cost performance, our analysis of recently launched projects found that the majority of these projects have experienced operations cost growth.

Overall Cost and Schedule The overall cost and schedule performance of NASA’s portfolio of major Performance Has projects continues to improve—a trend that began in 2013. In 2017, the Improved Over the Last overall development cost growth for the portfolio of 16 development projects declined to 15.6 percent, down from 17.3 percent in 2016. In Five Years addition, the average launch delay declined to 7 months, down from 8 months in 2016, due to OSIRIS-REx launching 1 month before its

Page 9 GAO-17-303SP Assessments of Major NASA Projects

committed launch date. Both measures are at or near the lowest levels we have reported since we began our annual review in 2009 (see fig. 4).9

Figure 4: Development Cost Performance and Average Launch Delay for Major NASA Projects from 2009 through 2017

Note: Includes projects in development.

The overall cost and schedule performance of the NASA major projects portfolio has improved, in part, due to the continued addition of new development projects. The cost and schedule performance of any

9We have historically presented cost and schedule performance including and excluding the James Webb Space Telescope (JWST) because, prior to 2015, it had a development cost baseline significantly larger than other projects and the magnitude of its cost growth masked the performance of the remainder of the portfolio. Now that there are other projects, such as Orion and the Space Launch System, with large development cost baselines, we decided to no longer present cost performance trends excluding JWST.

Page 10 GAO-17-303SP Assessments of Major NASA Projects

portfolio is partially driven by its composition—the number, size, and average age of projects. The addition of new projects generally helps improve portfolio performance because they are less likely to have experienced cost and schedule growth than older ones. New projects can also improve a portfolio’s cost performance by increasing the development baseline against which portfolio cost growth is measured. In other words, increases in the development cost baseline—which is the denominator when calculating the percentage cost growth in the portfolio—can help drive cost growth percentages down. There are four new projects in NASA’s major project portfolio in 2017. These projects increased the development baseline of the portfolio by $3.1 billion or approximately 14 percent, and have not incurred any cost growth to date. If we exclude these new projects from the portfolio, development cost growth increases from 15.6 percent to 17.8 percent in 2017. We have reported similar findings on the effects of new projects on cost performance in our prior two annual assessments.10 Figure 5 illustrates several key characteristics of NASA’s portfolio of major projects in development, including the number of projects, and their combined development cost baseline.

10GAO, NASA: Assessments of Major Projects, GAO-16-309SP (Washington, D.C.: Mar. 30, 2016) and NASA: Assessments of Selected Large-Scale Projects [Reissued on March 26, 2015], GAO-15-320SP (Washington, D.C.: Mar. 24, 2015).

Page 11 GAO-17-303SP Assessments of Major NASA Projects

Figure 5: Total Number and Development Cost Growth for NASA Major Projects with Established Cost Baselines from 2009 through 2017

Note: Includes projects in development.

We have also previously found that cost and schedule performance collectively improved as the projects in the portfolio have become, on average, younger.11 Project age is calculated based on the length of time a project has been in development and is another method of capturing the newness of the portfolio. In 2017, the average age of the portfolio increased from 29 months to 34 months and cost performance still improved (see. fig. 6). If this trend continues, it would indicate that NASA’s development cost performance improvements may be sustainable even when the portfolio consists of older projects that are more likely to experience cost growth.

11GAO-16-309SP.

Page 12 GAO-17-303SP Assessments of Major NASA Projects

Figure 6: Average Age of NASA Major Projects from 2009 through 2017

Note: The average age of projects is the average length of time projects in the portfolio have been in development.

Most Projects Performed Most NASA major projects stayed within cost and schedule estimates in Well in 2016, but Several 2016, but several projects have experienced or will soon experience Projects Have significant cost growth or schedule delays. On the positive side, the OSIRIS-REx project was launched in September 2016—1 month before Experienced or Will its committed date—and completed development for $157 million or 20 Experience Cost or percent less than was estimated in its baseline. The largest contributors Schedule Growth to the underrun reported in 2016 were lower than expected use of reserves, which are funds set aside to address potential risks or problems, and launch vehicle costs. Several other projects did not fare well.

• The Interior Exploration using Seismic Investigations, Geodesy, and Heat Transport (InSight) project was replanned in 2016 after it missed its committed launch date due to technical issues with its primary

Page 13 GAO-17-303SP Assessments of Major NASA Projects

science instrument, which was contributed by France.12 As a result of the problems, NASA delayed the project’s launch by 26 months from March 2016 to May 2018. Development costs have increased by $131.7 million or 24 percent, primarily due to the project needing to recomplete the assembly, test, and launch operations phase. • The Space Network Ground Segment Sustainment (SGSS) project’s estimated cost grew by at least $53 million in 2016 due to continued problems with contractor performance. The rising cost of the project also led it to defer work; the project will only deploy the system at one of three planned sites. The project is currently reviewing its cost and schedule and expects there to be additional cost growth and schedule delays. SGSS was previously rebaselined in 2015. • The ICESat-2 project’s cost and schedule is under review due to technical issues with its only instrument, the Advanced Topographic Laser Altimeter System (ATLAS). A key part in the instrument’s lasers failed during ATLAS thermal vacuum testing, which is the last step before system integration begins. The project is working to repair the lasers. The project expects to miss its committed launch date—June 2018—by at least 3 months due to the laser problems. ICESat-2 was previously rebaselined in 2014. • Both Commercial Crew Program contractors have notified NASA that they will not be able to complete the development and certification of their crew transportation systems by 2017, the date originally established in their contracts. The contractors must provide NASA evidence that their crew transportation system meets its performance and safety requirements to be certified. Both contractors have experienced challenges in development and expect certification to be delayed until at least 2018. Table 1 provides data on the cost and schedule performance for the 16 major projects in development that have cost and schedule baselines.

12A replan is a process initiated if development cost growth is 15 percent or more. A replan does not require a new project baseline to be established. A rebaseline is a process initiated if development cost growth is more than 30 percent. Both processes require the NASA Administrator to transmit a report to the Committee on Science, Space, and Technology of the House of Representatives and the Committee on Commerce, Science, and Transportation of the Senate. In addition, if a project or program milestone is likely to be delayed by 6 months or more, a report to the committees is required.

Page 14 GAO-17-303SP Assessments of Major NASA Projects

Table 1: Development Cost and Schedule Performance of Selected Major NASA Projects Currently in the Implementation Phase

Overall performance Project Confirmation 2016 performance Cumulative performance date Year Cost Schedule Cost Schedule (millions) (months) (millions) (months) Lower than expected cost OSIRIS-REx 2013 -$79.6 -1 -$157.8 -1 SPP 2014 $0.0 0 -$5.4 0 TESS 2014 $0.0 0 -$26.8 0 GRACE-FO 2014 $0.0 0 -$0.6 0 Oriona 2015 $4.7 0 -$151.7 0 Within baseline ICONa 2014 $0.2 0 $0.0 0 SLS 2014 $0.0 0 $0.0 0 EGSa 2014 $0.0 0 $3.6 0 SWOT 2016 $0.0 0 $0.0 0 Mars 2020 2016 $0.0 0 $0.0 0 RBI 2016 $0.0 0 $0.0 0 NISAR 2016 $0.0 0 $0.0 0 Higher than expected cost InSight 2014 $131.7 26 $131.7 26 Rebaseline JWST 2008 $0.0 0 $3,607.7 52 ICESat-2b 2012 $0.0 0 $204.9 13 SGSSb 2013 $53.4 0 $362.1 27 Total: $110.4 25 $3,967.7 117

Legend: OSIRIS-REx: Origins-Spectral Interpretation-Resource Identification-Security-Regolith Explorer; SPP: Solar Probe Plus; ICON: Ionospheric Connection Explorer; TESS: Transiting Exoplanet Survey Satellite; GRACE-FO: Gravity Recovery and Climate Experiment Follow-On; Orion: Orion Multi-Purpose Crew Vehicle; SLS: Space Launch System; EGS: Exploration Ground Systems; SWOT: Surface Water and Ocean Topography; RBI: Radiation Budget Instrument; NISAR: NASA Indian Space Research Organisation – Synthetic Aperture Radar; InSight: Interior Exploration using Seismic Investigations, Geodesy, and Heat Transport; JWST: James Webb Space Telescope; ICESat-2: Ice, Cloud, and Land Elevation Satellite-2; SGSS: Space Network Ground Segment Sustainment Source: GAO analysis of National Aeronautics and Space Administration data. | GAO-17-303SP Note: Positive values indicate cost growth or launch delays. Negative values indicate cost decreases or earlier than planned launch dates. aThe total estimated costs of the Orion, EGS, and ICON projects remain unchanged. The cost changes in the table reflect changes to how the project allocated project funding between the development and formulation phases. bThe ICESat-2 and SGSS projects expect to experience additional cost growth and schedule delays, but the exact magnitude is unknown. The projects were reevaluating their cost and schedules at the time of our review.

Page 15 GAO-17-303SP Assessments of Major NASA Projects

NASA’s Highest Cost As we found in 2016, the projects in NASA’s current portfolio with the Projects Are in the Stage highest development costs, including Orion and the Space Launch 13 When Most Rebaselines System, are at the stage when most rebaselines occur. Projects appear most likely to rebaseline between their critical design review and system Occur integration reviews—the riskiest point in the development cycle. NASA has rebaselined one major project each year for 8 out of the last 10 years. All eight projects were rebaselined after their critical design reviews and most of the projects that held a systems integration review were rebaselined before doing so. Table 2 lists the eight projects that plan to be in that phase of development in 2017. If a rebaseline occurs on any of these projects, it could add at least between $97 million and $2.1 billion to the development cost of the portfolio. This range is based on 30.1 percent development cost growth—as a rebaseline is triggered when development cost growth exceeds 30 percent—for the projects with the lowest and highest development cost in table 2.

13GAO-16-309SP.

Page 16 GAO-17-303SP Assessments of Major NASA Projects

Table 2: Current Projects between Critical Design Review and Systems Integration Review

Project Critical design review date Systems integration review date Exploration Ground Systems (EGS) December 2015 May 2017 Transiting Exoplanet Survey Satellite (TESS) August 2015 May 2017 Interior Exploration using Seismic Investigations, May 2014 June 2017 Geodesy, and Heat Transport (InSight)a James Webb Space Telescope (JWST)b March 2010 September 2017 Space Launch System (SLS)c July 2015 August 2018 Orion Multi-Purpose Crew Vehicle (Orion)d October 2015 September 2020 Space Network Ground Segment Sustainment June 2013 TBD (SGSS)b Ice, Cloud, and Land Elevation Satellite-2 February 2014 TBD (ICESat-2)e

Source: GAO analysis of National Aeronautics and Space Administration data. | GAO-17-303SP Note: Projects shaded in gray have been rebaselined previously. aInSight will hold a second system integration review. The project has to recomplete the system assembly, test, and launch operations phase due to technical issues with its primary instrument. bThe system integration review dates are for the project’s rebaselined schedule. cThe Space Launch System does not have a system integration review. The program’s next major system engineering milestone is its planned August 2018 design certification review, which is intended to ensure that the design meets functional and performance requirements and is ready for operation. dThe system integration review date for Orion is for Exploration Mission-2, the mission for which the program established its cost and schedule baseline. Orion held a system integration review for Exploration Mission-1 in November 2016. eICESat-2 postponed its system integration review indefinitely due to technical problems with its primary instrument.

Three of the largest projects in this critical stage of development— Exploration Ground Systems, Orion, and the Space Launch System— continue to face cost, schedule, and technical risks. In April 2017, we found that the first integrated test flight of these systems, known as Exploration Mission-1, will likely be delayed beyond November 2018.14 NASA concurred with our findings and is currently conducting an assessment to establish a new launch date. Because NASA’s assessment is ongoing, the cost implications of the schedule delay and its effect on the projects’ baselines are still unknown. However, given that these three human space exploration programs represent more than half of NASA’s current portfolio development cost baseline, a cost increase or delay could have substantial repercussions not only for these programs but NASA’s entire portfolio.

14GAO, NASA Human Exploration Programs: Delay Likely for First Exploration Mission, GAO-17-414 (Washington, D.C.: Apr. 27, 2017).

Page 17 GAO-17-303SP Assessments of Major NASA Projects

Payload Development Multiple independent studies have historically found that instrument Was the Biggest Area of development problems are the largest element of cost growth within the 15 Cost Risk for Science control of a project. We found that still to be the case for seven current Science Mission Directorate projects that we examined.16 The total Projects estimated payload development costs for the seven projects increased by $250.1 million or 67 percent since they were baselined, which was more than any other cost category that NASA tracked (see table 3).

Table 3: Development Cost Changes from Project Confirmation to February 2017 for Select Science Mission Directorate Projects

Development categories Baseline costs February 2017 cost Dollar change Percentage change (in millions) (in millions) (in millions) Payloads $373.4 $623.5 $250.1 67% Ground Systems $114.4 $151.3 $36.9 32% Aircraft and Spacecraft $866.6 $1,079.0 $212.4 25% Launch Vehicle $1,116.5 $1,044.3 ($72.2) -6% All Other Categories $1,247.3 $966.04 ($281.3) -23% Total Development Cost $3,718.1 $3,864.1 $146.0 4%

Source: GAO analysis of National Aeronautics and Space Administration data. | GAO-17-303SP Note: “All other categories” include system integration and test, science and technology, and other direct costs. Other direct costs include project and headquarters-held cost reserves.

The total development cost for these seven projects increased by a smaller amount because launch vehicle costs were less than anticipated and the individual projects have been able to use their cost reserves to cover overruns in other development cost categories. Project officials have told us that they use higher launch vehicle cost estimates in their

15For example, National Research Council of the National Academies, Controlling Cost Growth of NASA Earth and Space Science Missions (Washington, D.C.: National Academies Press, 2010); Bob Bitten, Perspectives on NASA Mission Cost and Schedule Performance Trends (El Segundo, Calif.: The Aerospace Corporation, 2008); and Claude Freaner, An Assessment of the Inherent Optimism in Early Conceptual Designs and its Effect on Cost and Schedule Growth (El Segundo, Calif.: The Aerospace Corporation, 2008). 16The seven projects are Gravity Recovery and Climate Experiment Follow-On (GRACE- FO), ICESat-2, InSight, Ionospheric Connection Explorer (ICON), OSIRIS-REx, SPP, and Transition Exoplanet Survey Satellite (TESS). We choose these projects because they reported their development costs using similar cost categories and at least 6 months had passed since their cost baseline was approved. See appendix I for more information about our scope and methodology.

Page 18 GAO-17-303SP Assessments of Major NASA Projects

baselines to account for the most expensive launch vehicle option under consideration and inflation. Once a launch vehicle is selected and a contract is signed, these costs can decrease. Competition in the space launch industry may also be contributing to lower than expected prices. Cost reserves are included in the “All Other Categories” line in table 3. The amount in that category decreases as projects allocate reserves to pay for increases in categories, such as payloads and aircraft and spacecraft. The changes in most categories were primarily driven by a few projects. For example, ICESat-2 had development problems with its only instrument, resulting in a $144.7 million increase in its payload category, while OSIRIS-REx had a $45.2 million decrease in the launch vehicle category.

Operations Costs for NASA has improved the development cost performance of its projects, Launched Projects Have but operations cost performance continues to be a challenge.17 We Increased examined projects that were included in our annual assessments from 2009 to 2016 and found that 20 previously launched science missions experienced cumulative operations cost growth of $177.7 million (11 percent), up from $114.6 million (8 percent) in 2016. This increase was driven mostly by the OSIRIS-REx project. OSIRIS-REx launched in 2016 and its operations costs were $100 million higher than previously estimated. The project determined it needed additional funding for operations, for example, to increase staff levels for activities such as on- orbit system check out and payload calibration. The project has not exceeded its cost baseline because it underran its estimated costs for development. On the positive side, the estimated operations cost for the Mars Atmosphere and Volatile EvolutioN project decreased by $31.5 million in 2016 because the mission operated more efficiently than anticipated and needed less funding to address risks. The project is still exceeding its operations cost estimate overall.

Of the 20 previously launched science missions we examined, 15 projects experienced operations cost growth and, for 3 projects those operations cost increases caused the project to exceed the cost baseline for the project as a whole. For example, the Juno project experienced operations

17Development costs include Phases C and D of the NASA project life cycle. Our analysis of operations costs focused on prime operations costs, which are the project’s planned mission operations in Phase E. The agency may elect to undertake a period of extended operations if a system is still operational after the prime mission is fulfilled. We did not consider extended operations costs in this analysis as they are not part of the project’s cost baseline.

Page 19 GAO-17-303SP Assessments of Major NASA Projects

cost growth in 2016 that resulted in the project exceeding its overall cost baseline. The project’s operations costs increased by $22.8 million because it needed to change the orbit duration from 11 days to 14 days in order to reduce mission risk during operations. This change resulted in increasing the project’s time in prime operations by approximately 4 months over the initial plan.

NASA has acknowledged that establishing operations cost baselines at project confirmation is a challenging task, and has efforts underway to improve operations cost estimates. For example, NASA updated the Mission Operations Cost Estimating Tool to include historical project information as well as more rigorous statistical methods. Science Mission Directorate officials have also previously told us that they have initiated studies to examine how operations costs are estimated and asked standing review boards to more carefully review these estimates at a project’s confirmation review.

NASA has generally maintained improvements in the technology maturity NASA Has Generally and design stability of its major projects, even as the number of new Maintained technologies in its most recent projects has increased. Three out of the four projects that held preliminary design reviews in 2016 matured all their Improvements in technologies to the level recommended by GAO best practices. Further, Technology Maturity these four projects had a combined total of 17 critical technologies— nearly the same number developed by the previous 10 major projects to and Design Stability hold a preliminary design review. None of NASA’s major projects held critical design reviews in 2016. Design changes on the projects that previously held this review increased this year, although the overall magnitude of the changes remains relatively low. Further, we found that NASA major projects that have launched within their cost and schedule baselines since 2009 tended to meet best practices for maturing technologies and minimizing design changes. In contrast, rebaselined projects were less likely to do so.

Page 20 GAO-17-303SP Assessments of Major NASA Projects

NASA Continues to Most of NASA’s major projects met GAO best practices for technology Improve the Technology maturity in 2017, even as the average number of critical technologies in Maturity of Its Projects, projects increased. Our best practices work has shown that reaching technology readiness level 6—which includes demonstrating a Even as the Number of representative prototype of the technology in a relevant environment that Critical Technologies simulates the harsh conditions of space—by the preliminary design Increased review can minimize risks for systems entering product development.18 Projects falling short of this standard may experience subsequent technical problems, which can result in cost growth and schedule delays. Figure 7 shows that NASA has continued its trend to improve the technology maturity of its projects.

18Appendix IV contains information about GAO’s product development best practices and the project attributes and knowledge-based metrics that we assess projects against at each stage of a system’s development.

Page 21 GAO-17-303SP Assessments of Major NASA Projects

Figure 7: Number of NASA’s Major Projects Attaining Technology Maturity by Preliminary Design Review from 2010 through 2017

Note: Includes projects that completed preliminary design review and identified critical or heritage technologies. For example, for 2017, 15 of 22 NASA major projects had held this review and identified critical or heritage technologies.

Three of the four projects that held a preliminary design review in 2016— NASA Indian Space Research Organisation – Synthetic Aperture Radar (NISAR), RBI, and Surface Water and Ocean Topography (SWOT)— matured their heritage and critical technologies to a technology readiness level 6 by that review.19 The remaining project, Mars 2020, matured two

19NASA distinguishes critical, or new, technologies from heritage technologies. Our product development best practices do not make this distinction. We describe critical technologies as those that are required for the project to successfully meet customer requirements, which can include both existing or heritage technology or new technology. Therefore, to assess overall technology maturity, we analyzed the maturity of heritage and critical technologies that NASA reported for projects in our data collection instrument. In other analyses, which focus on the number of new technologies being used by programs, we maintain NASA’s distinction between critical and heritage technologies. Appendix III provides a description of technology readiness levels, which are the metrics used to assess technology maturity.

Page 22 GAO-17-303SP Assessments of Major NASA Projects

of its seven critical technologies and one of its two heritage technologies to the recommended level. A Mars 2020 project official said NASA held the review with immature technologies to stay on schedule and avoid delaying progress on other parts of the project. Schedule is a key driver for Mars 2020. If the project misses the 2020 launch window, it must wait 26 months before another launch opportunity is available. To help mitigate the risk of moving forward with technologies that were not mature, the project allocated a significant portion of its reserve funding to cover potential cost growth for its new and modified instruments. As of January 2017, the project stated that all its critical technologies had reached a technology readiness level 6, but it had not yet matured the one remaining heritage technology. The remaining technology is an improved sensor on Mars 2020’s entry, descent, and landing instrument that could be descoped and replaced with a heritage sensor if necessary.

In 2017, NASA’s average number of critical technologies increased from prior years to a level not seen in a portfolio since 2011 (see fig. 8). The increase resulted from four projects—Mars 2020, NISAR, RBI, and SWOT—that held preliminary design reviews in 2016 with a combined total of 17 critical technologies, nearly the same number developed by the previous 10 major projects to hold preliminary design review. Mars 2020 has seven critical technologies; SWOT has four critical technologies; and RBI and NISAR each have three critical technologies.

Page 23 GAO-17-303SP Assessments of Major NASA Projects

Figure 8: Average Number of Critical Technologies Reported by NASA for Major Projects in Development from 2009 through 2017

Most major projects we have reviewed since 2009 have not pushed technological innovation through the development of critical technologies. Until Mars 2020 and SWOT, the previous two major NASA projects to develop more than three critical technologies were the James Webb Space Telescope and Solar Probe Plus (SPP) which entered implementation in 2008 and 2014, respectively. About three-quarters of all NASA major projects we have reviewed had fewer than three critical technologies (see table 4). Over half of all major projects we have reviewed had either one critical technology or none at all. Several of these are upgrade or follow-on missions, which rely heavily on heritage technologies. For example, in this year’s portfolio, the Gravity Recovery and Climate Experiment Follow-On (GRACE-FO) project—a follow-on to the original GRACE mission—is not developing any critical technologies but is employing technologies developed for the original GRACE mission.

Page 24 GAO-17-303SP Assessments of Major NASA Projects

Table 4: Projects Developing Critical Technologies by Confirmation Year through 2016

Confirmation Number of projects developing Number of projects developing year three or fewer critical four or more critical technologies technologies 2009 or earlier 15 6 2010 3 0 2011 0 0 2012 2 0 2013 2 0 2014 5 1 2015 1 0 2016 2 2 Total 30 9

Source: GAO analysis of National Aeronautics and Space Administration data. | GAO-17-303SP Note: EGS did not report any critical or heritage technologies, so it was omitted from this analysis.

GAO’s best practices criteria do not focus on the number of new technologies, but rather their maturity, when considering their effect on cost and schedule risk. Therefore, the issue is not whether to push innovation through technology development, but rather the steps projects take to increase the likelihood of mission success by maturing these technologies to a high level prior to entering the implementation phase. We have previously reported that maturing technologies in separate technology development efforts is a best practice that can reduce risk and improve cost and schedule outcomes in product or system development.20

NASA’s technology demonstration missions program, which began in 2010, aims to implement this best practice of maturing new technologies outside of projects. The program’s goal is to mature a technology from a technology readiness level 5 to technology readiness level 7 or greater. After the technologies are matured, they are to be transferred or infused into other NASA, partner, or commercial projects. For example, NASA is maturing advanced controls, sensors, and robotics technologies in its Restore-L mission, a satellite servicing mission, and anticipates using Restore-L technologies on multiple projects including the Asteroid

20GAO, Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes, GAO/NSIAD-99-162 (Washington, D.C.: July 30, 1999).

Page 25 GAO-17-303SP Assessments of Major NASA Projects

Redirect Robotic Mission (ARRM) and Orion. Likewise, NASA is maturing Deep Space and Near Earth operational systems with the optical communications technologies in its Laser Communication Relay Demonstration to be infused into other NASA and partner projects.

Several NASA Projects No NASA major projects held a critical design review during this reporting Experienced Late Design cycle, but several projects already past this milestone reported late 21 Changes, but Other design changes. The critical design review is the time in the project’s life cycle when the integrity of the project design and its ability to meet Design Stability Measures mission requirements is assessed. If a project experiences a large Remained Unchanged amount of engineering drawing growth after this review, it may be an indicator of instability in the project design late in the development cycle. Design changes at this point can be costly to the project in terms of time and funding because hardware may need to be reengineered or reworked as a result. For the 11 projects in development that have held critical design reviews, drawing growth after the review was 16 percent, which is higher than the last 2 years, but relatively low historically (see fig. 9). The increase in 2017 was due to primarily to three projects—James Webb Space Telescope, Transiting Exoplanet Survey Satellite (TESS), and InSight. For example, the number of InSight drawings grew by 15 percent after a key instrument failed late in testing. As a result, the project added new hardware and took on work that was previously managed by its French partner.

21The Mars 2020 project held its critical design review after we completed our analysis for this reporting cycle.

Page 26 GAO-17-303SP Assessments of Major NASA Projects

Figure 9: Average Percentage of Engineering Drawing Growth after Critical Design Review for NASA Major Projects from 2010 through 2017

Note: Drawing growth in 2010 was primarily attributed to the Solar Dynamics Observatory (SDO) because it did not have a stable design at its critical design review and drawings for SDO’s instruments were not included in this review. The project launched in 2010 and exited the portfolio.

Designs for NASA’s major projects have remained generally stable after their critical design reviews, but often fall short of a key GAO best practice leading up this review. Our work on product development best practices shows that at least 90 percent of engineering drawings should be releasable by the critical design review to lower the risk of subsequent cost and schedule growth.22 The NASA Systems Engineering Handbook also includes this metric. Projects that do not achieve design stability by critical design review may experience design changes and manufacturing problems. Fewer than half of NASA’s current major projects released at least 90 percent of their engineering drawings by the critical design review, although projects have performed better in recent years (see fig.

22Engineering drawings are considered to be a good measure of the demonstrated stability of a product’s design because the drawings represent the language used by engineers to communicate to the manufacturers the details of a new product design— what it looks like, how its components interface, how it functions, how to build it, and what critical materials and processes are required to fabricate and test it. Once the design of a product is finalized, the drawing is “releasable.”

Page 27 GAO-17-303SP Assessments of Major NASA Projects

10). The average percentage of engineering drawings released by the projects was 69 percent, roughly the same as it has been since 2013. Two projects—Mars 2020 and RBI—expect to hold critical design reviews in 2017. The RBI project has released all of its expected design drawings in advance of its planned June 2017 critical design review. The Mars 2020 project had released 72 percent of its design drawings as of January 2017, and planned to release about 80 percent by its late February 2017 critical design review. In January 2017, the NASA Inspector General recommended that the Mars 2020 project manager ensure the project met the 90 percent drawing release criteria before holding its critical design review.23 NASA concurred with the recommendation, but stated that the current rate of drawing release on the project is acceptable.

Figure 10: Number of NASA Major Projects That Released over 90 Percent of Engineering Drawings at Critical Design Review from 2010 through 2017

Note: Includes projects that completed critical design review and had engineering drawings. For example, for 2017, 11 of 22 NASA major projects had held this review and had engineering drawings.

23National Aeronautics and Space Administration Office of Inspector General, NASA’s Mars 2020 Project, IG-17-009 (Washington, D.C.: Jan. 30, 2017).

Page 28 GAO-17-303SP Assessments of Major NASA Projects

NASA projects have performed better against other design stability metrics at their critical design review. A panel of experts convened by GAO in 2013 identified several metrics as good indicators of design stability, including maintaining adequate mass and power margins and completing requirements validation and verification plans. The NASA Systems Engineering Handbook also includes these metrics. We found that all projects for which mass and power requirements were applicable met these requirements at their critical design reviews and have continued to maintain the required levels of margin.24 Further, all projects that were required to complete a validation and verification plan by their critical design review met the requirement. NASA requires that projects complete a validation and verification plan by this review to ensure that projects have a plan in place to track the completion of verification and validation events and activities.25

NASA Project NASA’s major projects that have launched within their cost and schedule Performance Reflects baselines tended to have fewer critical technologies, fully mature their Adherence to Best critical and heritage technologies before their preliminary design reviews, and minimize late design changes, although these are not the only factors Practices for Maturing that contribute to cost and schedule performance. Since we started Technologies and reporting on NASA’s major projects in 2009, six NASA missions have Minimizing Design launched within cost and schedule baselines. All of these projects had Changes three or fewer critical technologies and all but two matured their technologies to technology readiness level 6 by the preliminary design review (see table 5). In addition, these projects minimized design changes after critical design review with drawing growth below 20 percent for all but one project. Even when projects did not meet best practices, they took other steps to mitigate potential risks. For example, we previously reported that the Gravity Recovery and Interior Laboratory

24Mass is a measurement of how much matter is in an object. It is related to an object’s weight, which is mathematically equal to mass multiplied by acceleration due to gravity. Margin is the spare amount of mass or power allowed or given for contingencies or special situations. Some centers provide guidance on the percentage of mass margin required at various points in project development, with required margins ranging from 30 to 0 percent, depending on where a project is in the development cycle. 25Validation is defined as the continuous process of ensuring that requirements are well- formed (clear and unambiguous), complete (agrees with customer and stakeholder needs and expectations), consistent (conflict free), and individually verifiable and traceable to a higher level requirement or goal. Verification is defined as proof of compliance with requirements and specifications.

Page 29 GAO-17-303SP Assessments of Major NASA Projects

project held its preliminary design review with an immature technology but set aside adequate reserves to cover subsequent delays.26

Table 5: Characteristics of NASA Major Projects That Launched within Their Cost and Schedule Baselines from 2009 through 2017

Project Critical Technologies matured at Drawing growth after technologies preliminary design review critical design review Juno 0 Yes 47% Gravity Recovery and Interior Laboratory 1 No 18% Mars Atmosphere and Volatile EvolutioN 1 Yes 3% Soil Moisture Active Passive 1 No 0% Landsat Data Continuity Mission 3 Yes 16% Origins-Spectral Interpretation-Resource 3 Yes 3% Identification-Security-Regolith Explorer

Source: GAO analysis of National Aeronautics and Space Administration data. | GAO-17-303SP

In contrast, major projects that have rebaselined tended to have more critical technologies, not fully mature their critical and heritage technologies before their preliminary design reviews, and experience more late design changes. Of the eight major projects that have rebaselined over the last 10 years, five projects developed between four and nine critical technologies and none of the five projects matured their technologies prior to their preliminary design review (see table 6). In addition, the projects experienced more drawing growth—on average 54 percent—after their critical design reviews.27 None of the projects included in this analysis met the best practice of releasing 90 percent or more of their design drawings at their critical design reviews. Therefore, we were not able to make a comparison between the projects that launched within cost and schedule and those that were rebaselined for that best practice.

26GAO, NASA: Assessments of Selected Large-Scale Projects, GAO-11-239SP (Washington, D.C.: Mar. 3, 2011). 27We excluded SGSS from the drawing growth calculation as this project does not use drawings.

Page 30 GAO-17-303SP Assessments of Major NASA Projects

Table 6: Characteristics of NASA Major Projects That Have Rebaselined from 2007 through 2017

Project Critical Technologies matured at Drawing growth after technologies preliminary design review critical design review Orbiting Carbon Observatory-2a 0 Yes 8% Space Network Ground Segment Sustainment 0 Yesb Not applicable Ice, Cloud, and Land Elevation Satellite-2 2 Yes 15% Glory 4 No 27% National Polar-orbiting Operational 6 No 41% Environmental Satellite System Preparatory Projectc Mars Science Laboratory 7 No 147% Stratospheric Observatory for Infrared 9 No 93% Astronomy James Webb Space Telescope 9 No 44%

Source: GAO analysis of National Aeronautics and Space Administration data. | GAO-17-303SP aThe Orbiting Carbon Observatory-2 rebaseline was driven by launch vehicle failures which were external to the project. bIn April 2013, we reported that although SGSS officials reported that the project’s technologies were mature, a management review board determined that two heritage technologies were not at an appropriate level of maturity at the project’s preliminary design review. cThe current name of the National Polar-orbiting Operational Environmental Satellite System Preparatory Project is the Suomi-National Partnership Program.

Both we and NASA’s Inspector General have found that design and technical problems, among other factors, contributed to cost and schedule increases with these rebaselined projects. For example, the Mars Science Laboratory’s launch was delayed 26 months and its development costs increased by almost $800 million due in part to technical challenges. The project did not mature any of its seven critical technologies at its preliminary design review, including its motor actuators. The actuators are responsible for moving and controlling instruments on the rover and are considered high risk because they have a complex design involving hundreds of parts. The project had to redesign the actuators after determining that the initial design was not durable enough, which significantly contributed to the project’s cost and schedule growth.28

28GAO-16-309SP; GAO-11-239SP; GAO, NASA: Assessments of Selected Large-Scale Projects, GAO-09-306SP (Washington, D.C.: Mar. 2, 2009); IG-17-009; and NASA’s Challenges to Meeting Cost, Schedule, and Performance Goals, IG-12-021 (Washington, D.C.: Sept. 27, 2012).

Page 31 GAO-17-303SP Assessments of Major NASA Projects

NASA has implemented improved project management tools to manage NASA Has acquisition risks, but capacity and funding challenges are emerging Implemented New issues. In October 2015, NASA decided to decentralize its independent assessment function and deploy the staff to the agency’s centers, in part, Tools to Reduce to better use its workforce to meet program needs in areas such as Acquisition Risks, but program management, cost estimating, and resource analysis, and to fill gaps in program analysis skills at the center level. Last year, we reported Workforce and on the potential risks that this change could pose for project oversight, but Funding Challenges the transition is still ongoing and it is too early to assess its effect on areas, such as independence, the robustness of reviews, and information Are Emerging Issues sharing. Finally, almost one-third of the projects we assessed reported funding challenges that have or could affect their cost and schedule. In several cases, including Orion, NASA’s budget does not support the project’s internal schedules.

NASA Continues to NASA continues to implement cost, schedule, and earned value Improve Its Cost and management (EVM) tools designed to improve estimation practices and Schedule Management reduce acquisition risk, but is not following several key best practices in some of these areas. For several years, we have reported that high- Tools, but Lacks the quality joint cost and schedule confidence level (JCL) estimates that are Workforce Capacity to based on reliable supporting cost and schedule estimates and monitored Fully Implement Best using EVM data are critical to reducing acquisition risks.29 But NASA may Practices not have the workforce capacity to fully implement these management tools.

Updating joint cost and schedule confidence level estimates: NASA’s current major projects have developed JCL estimates as required, but most of these projects did not update their JCL estimates unless they were rebaselined. A JCL is a tool that assigns a confidence level, or likelihood, of a project meeting its cost and schedule estimates. NASA requires that programs and projects with estimated life-cycle costs of more than $250 million develop a JCL prior to project confirmation to ensure that cost and schedule estimates are realistic and projects thoroughly plan for anticipated risks. NASA policy does not require projects to update their JCLs as they progress through development and new risks emerge. However, GAO cost estimating best practices recommend that cost estimates be updated to reflect changes to a

29GAO-16-309SP and GAO, High-Risk Series: An Update, GAO-15-290 (Washington, D.C.: Feb. 11, 2015).

Page 32 GAO-17-303SP Assessments of Major NASA Projects

program or kept current as it moves through milestones.30 NASA has stated that it relies on other tools to assess progress and the adequacy of resources after project confirmation. One major project—InSight—did update its JCL in 2016 after missing its committed launch date. By doing so, the project was able to provide additional information to decision makers about the probability it will meet its revised cost and schedule estimates.

Creating reliable cost and schedule estimates: NASA has updated tools aimed at improving cost and schedule estimates, but individual projects have not met best practices for producing reliable cost estimates. Workforce capacity in these disciplines is also a challenge. High-quality cost and schedule estimates are the basis for a high-quality JCL, so it is critical that these estimates are rigorous and follow best practices. Accurate cost and schedule estimates and their associated confidence levels are crucial for decision makers who must be kept informed of the true cost and schedule in order for the projects to be positioned to succeed. In 2016, NASA updated its Cost Engineering Database to provide analysts with more data, such as historical project cost information, that can be used to inform cost estimates for similar projects. However, despite this and other NASA efforts to improve cost and schedule estimating, we recently found that the cost estimate for the most expensive project in the portfolio—Orion—did not include the necessary support and the schedule estimate did not include the level of detail required for high-quality estimates.31 NASA did not agree to update its cost and schedule estimates and JCL analysis as we recommended because it said the program was still performing within its cost and schedule baseline and updating them was not yet warranted. We continue to believe this recommendation should be addressed because without sound cost and schedule estimates, decision makers lack the information they need to make programmatic decisions.

In addition, NASA has identified workforce gaps that could affect its ability to develop reliable cost and schedule estimates. In 2016, NASA completed an assessment to ascertain the agency’s current health in the areas of cost and schedule estimating, cost and schedule assessment,

30GAO-09-3SP. 31GAO, Orion Multi-Purpose Crew Vehicle: Action Needed to Improve Visibility into Cost, Schedule, and Capacity to Resolve Technical Challenges, GAO-16-620 (Washington, D.C.: July 27, 2016).

Page 33 GAO-17-303SP Assessments of Major NASA Projects

EVM, and programmatic assessment, among other areas. The assessment revealed gaps in key skill areas like schedule estimation. NASA previously decentralized its independent assessment function in 2015, in part, to bolster the agency’s workforce capacity at the center level in key areas, such as cost and schedule estimating.

Implementing earned value management: NASA has made progress implementing EVM analysis—another key project management tool—but the agency has not yet fully implemented a formal EVM surveillance plan in accordance with NASA and GAO best practices.32 When implemented well, EVM integrates information on a project’s cost, schedule, and technical efforts for management and decision makers by measuring the value of work accomplished in a given period and comparing it with the planned value of work scheduled for that period and the actual cost of work accomplished. In November 2012, we recommended that NASA update its procedural requirements to include a formal EVM surveillance program in order to improve the reliability of EVM data collected by NASA programs.33 NASA agreed with our conclusion that EVM data reliability needed improvement, but it has yet to implement a formal surveillance requirement due to resource constraints.34 NASA has taken steps to improve the reliability of EVM data on individual projects, such as the James Webb Space Telescope, where we have previously found EVM deficiencies. We continue to believe that this recommendation should be addressed because without implementing proper surveillance agency- wide, projects may be utilizing unreliable EVM data to inform their decision making.35

32GAO, Schedule Assessment Guide: Best Practices for Project Schedules, GAO-16-89G (Washington, D.C.: Dec. 22, 2015); GAO-09-3SP; and National Aeronautics and Space Administration, Earned Value Management Implementation Handbook, NASA/SP-2012- 599 (Washington, D.C.: Feb. 15, 2013). 33GAO, NASA: Earned Value Management Implementation across Major Spaceflight Projects Is Uneven, GAO-13-22 (Washington, D.C.: Nov. 19, 2012). 34A formal surveillance plan involves establishing an independent surveillance organization with members who have practical experience using EVM. This organization then conducts periodic surveillance reviews to ensure the integrity of the contractor’s EVM system and where necessary discusses corrective actions to mitigate risks and manage cost and schedule performance. 35GAO, James Webb Space Telescope: Project on Track but May Benefit from Improved Contractor Data to Better Understand Costs, GAO-16-112 (Washington, D.C.: Dec. 17, 2015).

Page 34 GAO-17-303SP Assessments of Major NASA Projects

Implementation of NASA’s NASA’s transition away from a centralized independent assessment New Independent function is ongoing, and the agency is still developing and updating Assessment Approach Is policies, procedures, and guidance over a year later. In October 2015, NASA dissolved the Independent Project Assessment Office and Ongoing devolved the responsibility for independent assessments down to mission directorates, which oversee the projects being assessed. Independent reviews provide unbiased and comprehensive assessments of the technical, schedule, cost, and risk posture of NASA’s projects. They are also a key acquisition best practice that we have highlighted in prior reports.36 NASA issued a white paper that explains the general framework for the new independent assessment model in June 2016. At the same time, the Human Exploration and Operations Mission Directorate completed an implementation plan that explains how the independent assessment function will be carried out for projects within the directorate. In addition, the Office of the Chief Financial Officer completed its implementation plan in February 2017. Other plans have lagged. The Space Technology Mission Directorate and Science Mission Directorate plan to complete implementation plans in spring 2017. NASA also updated its standing review board handbook in December 2016, which serves as a guide for projects and review boards regarding expectations, processes, and products, and outlines guidance for membership selection, reviews, and reporting results. Finally, NASA is developing a new standard operating procedure for independent assessments that will include instructions for implementing the new model and plans to complete it by March 2017—17 months after the change was announced.

In March 2016, we highlighted three areas that could be negatively affected by the reorganization of the independent assessment function— independence, the robustness of reviews, and information sharing.37 NASA has taken some steps to address these areas. For example, to help maintain the robustness of the reviews, the Office of the Chief Financial Officer told us that it will review programmatic assessments produced by the mission directorates for review boards, such as JCL analyses, to ensure they are in line with agency expectations. The transition is still ongoing, so it is too early to assess the effectiveness of many of these actions.

36For example, see GAO, Defense Acquisitions: Assessments of Selected Weapon Programs [Reissued on April 9, 2015], GAO-15-342SP (Washington, D.C.: Mar. 12, 2015). 37GAO-16-309SP.

Page 35 GAO-17-303SP Assessments of Major NASA Projects

Several major projects have held reviews using the new assessment model and offered initial thoughts on its benefits and challenges. One project stated that placing responsibility for designing the terms of reference—the ground rules for standing reviews—in the implementing organization was a positive change. Another project said that its progress was slowed because of difficulties with standing review board staffing. We will continue to monitor the potential effects of the reorganization as more projects begin to use the new model.

Funding Challenges Could At least 7 of the 22 major projects we reviewed experienced funding Affect NASA’s Ability to challenges—specifically, funding phasing issues— in 2016. The cost and Efficiently and Effectively schedule performance of a project depends, in part, on receiving funding in line with its budget plan. Funding phasing issues—a mismatch between Manage Its Major Projects the money needed in a given time period to complete work to keep the project on schedule and the money currently or projected to be available—can lead to development delays and cost increases. Funding phasing issues can arise for a variety of reasons. For example, an agency may not receive enough funding in a given year to execute all of its projects in the most efficient manner or significant cost growth could occur and force decision makers to make trade-offs between major projects.

The projects we assessed experienced several types of funding-related challenges.

• The Europa Clipper project and Orion program are working to schedules not supported by NASA’s budget plans, which could result in funding shortfalls at the project-level. In recent years, Congress has provided more funding to these projects than NASA has requested. Up until recently, the Space Launch System has also been working to a schedule not supported by NASA’s budget plans. If the projects do not continue to receive appropriations that match their plans, they may have to take actions, such as revising their current schedules. We previously found that working to an unrealistic schedule may increase a project’s risk of exceeding its cost and schedule baseline.38

• Other projects have already experienced the effects of funding shortfalls. Prior to project confirmation, the NISAR project reported funding shortfalls in 2017 and 2018. As a result, the project stretched out its schedule by 1 year when it established its cost and schedule

38GAO-16-620.

Page 36 GAO-17-303SP Assessments of Major NASA Projects

baselines. The project initially thought it could address the shortfalls by delaying procurements, but a detailed study showed that this would not be possible due to the need to integrate several components in parallel. Funding shortfalls have also been a problem for other projects, such as the ARRM and SGSS projects. • In the case of WFIRST, a project early in formulation, the availability of funding in the near-term could drastically affect its eventual cost and launch date. According to project officials, receiving significant additional funding in fiscal year 2018 would allow the project to optimize the development schedule and launch in 2024, which would reduce the cost of the mission.

The individual assessments of the 21 projects we reviewed provide a two- Project Assessments page profile of each project. Each assessment includes a description of the project’s objectives, information about the NASA centers and international partners involved in the project, the project’s cost and schedule performance, a timeline identifying key project dates, budget information, and a brief narrative describing the current status of the project.39 The budget information is based on NASA’s fiscal year 2017 budget. The budget covers fiscal years 2017 to 2021; however, NASA describes funding numbers beyond fiscal year 2017 as notional. On the first page, the project profile presents the standard information listed above. On the second page of the assessment, we provide an analysis of the project challenges, and outline the extent to which each project faces cost, schedule, or performance risks because of these challenges, if applicable. NASA project offices were provided an opportunity to review drafts of the assessments prior to their inclusion in the final product, and the projects provided both technical corrections and more general comments. We integrated the technical corrections as appropriate and summarized the general comments at the end of each project assessment.

See figure 11 for an illustration of a sample assessment layout.

39The manifested launch date is the launch date which the project is working toward, and when a launch vehicle is available to launch the project. This date is only a goal launch date for the project, not a commitment that they will launch on this date. The committed launch readiness date is determined through a launch readiness review that verifies that the launch system and spacecraft/payloads are ready for launch.

Page 37 GAO-17-303SP Assessments of Major NASA Projects

Figure 11: Illustration of a Sample Project Assessment

Page 38 GAO-17-303SP Assessments of Major NASA Projects Asteroid Redirect Robotic Mission

The Asteroid Redirect Robotic Mission (ARRM) will retrieve a boulder from an asteroid and place it into lunar orbit for future human exploration. ARRM and the planned follow-on crewed mission are capability demonstrations— designed to develop systems and provide the types of operational experiences required for future human and robotic exploration of Mars. ARRM will demonstrate technologies for longer-duration, deep-space missions, such as advanced solar electric propulsion (SEP). The mission will also demonstrate an asteroid deflection technique by gravitationally altering the asteroid’s trajectory.

Source: AMA Studios. | GAO-17-303SP

TION 04/15 05/16 01/17 12/17 03/18 05/19 06/20 12/21 Formulation System GAO Preliminary Project Critical System Projected start requirements/ review design confirmation design integration launch

FORMULA mission review review review readiness

definition review IMPLEMEN TA TION date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Jet Propulsion In August 2016, the ARRM project entered the preliminary design and technology Laboratory completion phase with a higher cost and longer schedule than previously International Partner: TBD estimated. The project’s estimated costs increased primarily because of a 1-year delay in its planned launch readiness date, which was driven by near-term funding Launch Location: TBD constraints. Multiple NASA mission directorates are developing technologies for Launch Vehicle: TBD ARRM. Maturing technologies in separate technology development efforts is a best practice, but it could pose a challenge for the project if schedule, funding, or Mission Duration: 6 years other issues arise with these related efforts. The ARRM project is tracking a variety Requirement Derived from: NASA Strategic of design and development risks related to its critical technologies, including the Plan boulder capture system, which will require changes to the project’s requirements. The ARRM project plans to select its spacecraft, payloads, and launch vehicle in Budget Portfolio: Exploration, Exploration Research and Development 2017, but those plans could be affected if adequate funding is not available. The President’s 2018 Budget Blueprint has proposed canceling the Asteroid Redirect Mission.

PROJECT BUDGET INFORMATION PRELIMINARY COST a PRELIMINARY SCHEDULE percent of current portfolio over next 5 years then-year dollars in millions

5.6% 12/21 PROJECTED LAUNCH READINESS DATE

ARRM

Other major projects $1,672 – $1,822 (includes launch vehicle, excludes mission operations) 94.4%

total funded to date then-year dollars in millions 04/15 aThis estimate is preliminary, as the project is in formulation and there is uncertainty regarding the costs associated with the design options LATEST ESTIMATE $111.8 being explored. NASA uses these estimates for planning purposes. FEB. 2017

common name: ARRM 39 Assessments of Major NASA Projects GAO-17-303SP ASTEROID REDIRECT ROBOTIC MISSION

Cost and Schedule Status In August 2016, NASA approved the ARRM project to enter The ARRM project plans to select its spacecraft, the preliminary design and technology completion phase, payloads, and launch vehicle in 2017, but those plans but with a higher cost and longer schedule than previously could be affected if adequate funding is not available. planned. NASA increased the development cost cap for The project has decided to use a commercially available the project from $1.25 billion to $1.4 billion, not including spacecraft to reduce project costs. The project received launch costs. The cost primarily increased because of a proposals in October 2016 and plans to award the 1-year delay in ARRM’s planned launch readiness date spacecraft development contract in May 2017. NASA is from December 2020 to December 2021, due to near-term also considering adding hosted payloads on ARRM that funding constraints. In March 2017, the President’s 2018 could decrease mission risk. The project issued a call for Budget Blueprint proposed canceling the Asteroid Redirect hosted payloads in September 2016 and expects to make Mission, which includes the ARRM project. selections in June 2017. Finally, project officials stated that they expect to select a launch vehicle by preliminary design Technology review. The project plans to maintain compatibility with The ARRM project is continuing to mature its critical the Delta IV Heavy, Falcon 9 Heavy, and Space Launch technologies, but it is dependent on other projects for System vehicles until selection. many of its key technology development efforts. Project officials said they expect to mature ARRM’s four critical technologies to a technology readiness level 6 before its planned December 2017 preliminary design review. Maturing critical technologies to this level is a best practice, which helps minimize risks for space systems entering product development. Multiple NASA mission directorates are developing technologies for ARRM. From the Space Technology Mission Directorate, the project plans to leverage advanced controls, sensors, and robotics technologies in development for the Restore-L project, as well as high-powered solar electric propulsion (SEP) technologies. If significant schedule, funding, or other issues arise with the dependent technologies, ARRM could experience cost and schedule growth. The Restore-L and SEP schedules currently align with the ARRM project’s need dates, but the President’s 2018 Budget Blueprint proposed restructuring the Restore-L project. According to the ARRM project executive, the project has developed a memorandum of understanding related to these dependencies that includes actions it can take if there are schedule delays.

Design The ARRM project is tracking a variety of design and development risks related to its propulsion and boulder capture systems. According to a project official, the project is primarily focused on the risks associated with the capture system’s ability to successfully grip and retrieve a boulder from the asteroid. The project’s independent review board raised concerns about the system’s ability to meet current planned ARRM requirements. The project has completed a study of the capture system and has made changes to the PROJECT OFFICE COMMENTS design, but it will also have to make requirements changes. The ARRM project provided technical comments to a draft of this assessment, which were incorporated as appropriate.

common name: ARRM 40 Assessments of Major NASA Projects GAO-17-303SP Commercial Crew Program

The Commercial Crew Program facilitates and oversees the development of safe, reliable, and cost-effective crew transportation systems by commercial companies to carry NASA astronauts to and from the International Space Station (ISS). The program is a multi-phase effort that started in 2010. During the current phase, the program is working with two contractors—Boeing and SpaceX—that will design, develop, test, and operate the crew transportation systems. Once NASA determines the systems meet its standards for human spaceflight—a process called certification—the companies will fly up to six crewed missions to ISS.

Source: NASA. | GAO-17-303SP

TION 09/14 01/17 11/17 05/18 06/18 08/18 2018 2018 – 2024 Transportation GAO SpaceX SpaceX Boeing Boeing Final Post-certification Capabilities phase review uncrewed crewed uncrewed crewed certification missions

FORMULA contract awards test flight test flight test flight test flight review

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Kennedy Space Center Both of the Commercial Crew Program’s contractors have made progress developing their crew transportation systems, but have aggressive development Commercial Partners: Boeing, SpaceX, Blue Origina, Sierra Nevada Corporationa schedules that are increasingly under pressure. Boeing and SpaceX have determined that they will not be able to meet their original 2017 certification dates Launch Location: Boeing-Cape Canaveral and expect certification to be delayed until 2018. In addition, the Commercial Air Force Station, FL; SpaceX-Kennedy Crew Program is tracking risks that both contractors could experience additional Space Center, FL schedule delays. In February 2017, NASA reached an agreement with Boeing Launch Vehicle: Boeing-Atlas V; SpaceX- giving NASA the option of acquiring crew transportation from Boeing on Russian Falcon 9 Soyuz flights to the ISS in 2019, to protect against Commercial Crew program Requirement Derived from: NASA Strategic delays or contractor problems in certification. The program has also identified the Plan ability of NASA and its contractors to meet crew safety requirements as one of its top risks. Program officials told us their main focus is to work with the contractors Budget Portfolio: Space Operations, Space Transportation to ensure that the spacecraft designs are robust from a safety perspective.

aBlue Origin and Sierra Nevada Corporation do not have contracts for the current phase and therefore were not included in this assessment.

c PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions

Q4 2018 10.5% Q3

MONTHS 2018 MONTHS $4,275.5 14 1.1% 08/17 15

$4,229.6 LEAS T CCP

AT 04/17 LEAS T

Other AT major projects $2,646.5 1.8% $2,599.0 Current Current proposed maximum certification 89.5% contract value review date (as of Feb Original 2017) certification b total needed to be funded Original review date then-year dollars in millions maximum contract value (as of fiscal $6,922.0 year 2014) 09/14 09/14 BOEING SPACEX BOEING SPACEX $4,762.0 cIncludes contract costs for development, operations, and b Includes costs to fund the contracts. special studies.

common name: CCP 41 Assessments of Major NASA Projects GAO-17-303SP COMMERCIAL CREW PROGRAM

Cost and Schedule Status Both of the Commercial Crew Program’s contractors crew” metric as a way to measure the safety of a crew have made progress developing their crew transportation transportation system. The metric captures the probability systems, but have aggressive development schedules of death or permanent disability to one or more crew that are increasingly under pressure. The contractors members. Under the current contracts, the loss of crew were originally required to provide NASA all the evidence requirement is 1 in 270, meaning that the contractors’ it needed to certify that their systems met its requirements systems must carry no more than a 1 in 270 probability of by 2017. In February 2017, we reported neither Boeing incurring loss of crew. Program officials told us their main nor SpaceX can meet their original certification dates and focus is to work with the contractors to ensure that the both now expect certification to be delayed until 2018.a spacecraft designs are robust from a safety perspective. Boeing has proposed moving its certification review out For example, the program has identified the spacecrafts’ to the fourth quarter of 2018—at least 14 months later ability to tolerate the micrometeoroid and orbital debris than initially planned. SpaceX has moved its certification environment as the most significant driver of the loss review to the third quarter of 2018—at least 15 months of crew metric. Both contractors have lowered this risk later than initially planned. The Commercial Crew Program through testing, which provides insight into how well their is tracking risks that both contractors could experience systems perform in these environments, and by making additional schedule delays and its own analysis indicates design changes. If the contractors have to make future that certification is likely to slip into 2019. NASA currently design changes to improve their spacecraft’s performance relies on the Russian space agency to transport astronauts in this environment, certification could be further delayed. to the ISS. In February 2017, NASA reached an agreement with Boeing giving NASA the option of acquiring crew transportation from Boeing on Russian Soyuz flights in 2019, to protect against Commercial Crew program delays or problems in certification.

NASA has also made changes to the contracts that have increased their value. While the contracts are firm-fixed- price, their values can increase if NASA adds to the scope of the work or otherwise changes requirements. As of January 2017, the value of the design, development, and test portion of the contracts had increased by $47 million and $91 million, respectively for Boeing and SpaceX. PROJECT OFFICE COMMENTS According to a program official, these contract increases In commenting on a draft of this assessment, were covered by program cost reserves. Commercial Crew Program officials stated that the assessment adequately represents the program’s Other Issues to Be Monitored current status. The officials emphasized that In addition to Boeing and SpaceX’s schedule challenges, competition is essential to achieving a safe and both contractors face other risks that will need to be productive program and that having at least two addressed to support their certification. The Commercial companies developing disparate crew transportation Crew Program’s top programmatic and safety risks for Boeing are, in part, related to having adequate information systems provides benefits in redundancy, innovation, on certain systems, including its launch vehicle’s main and cost effectiveness. Further, the officials noted engine, to support certification. The Commercial Crew that the Commercial Crew Program was not funded Program’s top programmatic and safety risks for SpaceX at the levels in the President’s Budget request are, in part, related to ongoing launch vehicle design and during fiscal years 2011-2015, which were critical development efforts, including changes that could result years for design and development; adequate, timely from a September 2016 pre-launch mishap. funding remains essential to ensuring successful program performance. Finally, the Commercial Crew The Commercial Crew Program has identified the ability Program continues to support the crew transportation of it and its contractors to meet crew safety requirements development activities of Blue Origin and Sierra as one of its top risks. NASA established the “loss of Nevada Corporation through Space Act Agreements. Program officials also provided technical comments, aGAO, NASA Commercial Crew Program: Schedule Pressure Increases as Contractors Delay Key Events, GAO-17-137 (Washington, D.C.: Feb. 16, 2017). which were incorporated as appropriate. common name: CCP 42 Assessments of Major NASA Projects GAO-17-303SP Europa Clipper

The Europa Clipper mission aims to investigate whether the Jupiter moon could harbor conditions suitable for life. The project plans to launch a spacecraft in the 2020s, place it in orbit around Jupiter, and conduct a series of investigatory flybys of Europa. The mission’s planned objectives include characterizing Europa’s ice shell and any subsurface water, analyzing the composition and chemistry of its surface and ionosphere, understanding the formation of its surface features, and surveying sites for a potential landed mission. We did not assess the proposed Europa lander mission, which NASA expects to manage as a separate project.

Source: Europa Project Personnel, California Institute of Technology, Jet Propulsion Laboratory. | GAO-17-303SP

TION 06/15 01/17 01/17 08/18 10/18 11/19 09/20 06/22 07/23 Formulation System GAO Preliminary Project Critical System Project Agency start requirements/ review design confirmation design integration launch launch

FORMULA mission review review review readiness readiness

definition review IMPLEMEN TA TION date date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Jet Propulsion The Europa Clipper project entered the preliminary design and technology Laboratory completion phase in February 2017 and updated its preliminary cost and International Partner: None schedule estimates. Payload and spacecraft costs have increased, but the high end of the project’s cost range did not change. The project’s scope has Launch Location: Kennedy Space Center, also grown. At the project’s most recent decision review, its independent review FL board stated that it was at risk of exceeding its preliminary cost and schedule Launch Vehicle: TBD ranges unless its scope or complexity was reduced. Growth in the mass of and power needed to operate the Europa Clipper and its instruments, as well as Mission Duration: 3 year science mission Jupiter’s harsh radiation environment, also pose challenges for the project. The Requirement Derived from: 2010 Decadal Consolidated Appropriations Act, 2016 requires the project to use NASA’s Space Survey Launch System (SLS) and to plan for a 2022 launch, which could pose risks. SLS Budget Portfolio: Science, Planetary is still in development and may not have its initial launch until November 2018 or Science later. Project officials said the project needs to select a launch vehicle by the end of 2018 to maintain a 2022 launch date.

PROJECT BUDGET INFORMATION PRELIMINARY COST a PRELIMINARY SCHEDULE percent of current portfolio over next 5 years then-year dollars in millions

07/23 2.4% $3,100 – $4,000 PROJECTED (including launch LAUNCH vehicle) READINESS DATE

Europa Clipper Other major projects

97.6%

total funded to date then-year dollars in millions 06/15 aThis estimate is preliminary, as the project is in formulation and there is uncertainty regarding the costs associated with the design options LATEST ESTIMATE $257.4 being explored. NASA uses these estimates for planning purposes. FEB. 2017

common name: CLIPPER 43 Assessments of Major NASA Projects GAO-17-303SP EUROPA CLIPPER

Cost and Schedule Status The Europa Clipper project entered the preliminary design least this review. Project officials have also stated that the and technology completion phase in February 2017 and project needs to select a launch vehicle by the end of 2018 updated its preliminary cost and schedule estimates. to maintain a 2022 launch date. Payload and spacecraft costs have increased, but the high end of the project’s cost range did not change. Funding The project’s scope has also grown. NASA added two The Europa Clipper project has been working to a target new project requirements—the capability to serve as launch date in 2022 that has not been supported by NASA’s communication relay for a lander and investigate plumes budget plans to date. For fiscal year 2017, the project emitted from Europa’s surface. At the project’s most recent needs approximately $215 million to maintain the 2022 decision review, its independent review board stated that launch date, but NASA only requested $49.6 million. The the project was at risk of exceeding its preliminary cost Consolidated Appropriations Act, 2016 required that NASA and schedule ranges unless its scope or complexity was plan for a launch no later than 2022, and include the 5-year reduced. The board recommended that the project identify funding profile in its fiscal year 2017 budget necessary potential instruments and science requirements that could to achieve this and other goals. In February 2017, NASA be removed or reduced for the payload, which the project officials signed a decision memorandum for the project that plans to do prior to entering implementation in 2018. included a funding profile supporting a 2022 launch, but the agency has not yet released its fiscal year 2018 budget Technology and Design plan requesting these amounts. Growth in the mass and power needed to operate the Europa Clipper and its instruments poses a continuing challenge for the project. NASA selected 10 instruments for the mission—2 more than initially planned. Several of these instruments, including a mapping spectrometer and ice-penetrating radar, experienced mass and power growth and cost increases after being selected. Spacecraft costs also increased to accommodate the growth. For example, the project added two panels to its solar arrays to increase power, although it is working to reduce its power needs. The project plans to establish mass and power caps for each instrument and the spacecraft to mitigate these issues going forward.

The radiation environment around Jupiter poses one of the biggest technical challenges for the project. The project is working to reduce or better understand the negative effects of radiation—such as diminished performance and loss of science data—by hardening its technologies and testing materials and parts, among other actions. The project is considering adding more shielding to the vault that will protect its electronics, but doing so would increase mass.

Launch According to the Europa Clipper project, its top risk involves availability of a qualified and well-understood launch vehicle. The Consolidated Appropriations Act, 2016 requires the project to use NASA’s SLS. However, SLS is in development and will not have its initial launch until November 2018 or later. The Europa Clipper project plans PROJECT OFFICE COMMENTS to hold its preliminary design review in August 2018, at Europa Clipper project officials provided technical which point projects prefer to select a launch vehicle. The comments on a draft of this assessment, which were project plans to maintain compatibility with the Delta IV incorporated as appropriate. Heavy, Falcon Heavy, and SLS launch vehicles, until at common name: CLIPPER 44 Assessments of Major NASA Projects GAO-17-303SP Exploration Ground Systems

The Exploration Ground Systems (EGS) program is modernizing and upgrading infrastructure at the Kennedy Space Center and developing software needed to integrate, process, and launch the Space Launch System (SLS) and Orion Multi-Purpose Crew Vehicle (Orion). The EGS program consists of several major construction and facilities projects including the Mobile Launcher, Crawler Transporter, Vehicle Assembly Building, and launch pad, all of which need to be complete before the first uncrewed exploration mission (EM-1) using the SLS and Orion vehicles.

Source: NASA. | GAO-17-303SP (Note: Vehicle Assembly Building pictured above.)

TION 02/12 08/12 03/14 05/14 12/15 01/17 05/17 11/18 Formulation System Preliminary Project Critical GAO System Committed start requirements/ design confirmation design review integration launch

FORMULA mission review review review readiness

definition review IMPLEMEN TA TION date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Kennedy Space Center The EGS program’s schedule is deteriorating and it is at increased risk of exceeding its cost baseline and missing its November 2018 launch readiness International Partner: None date. The program has completed several of its projects, but it continues to Requirement Derived from: NASA experience technical challenges and delays and has limited cost and schedule Authorization Act of 2010 reserves remaining to address them. The EGS program has used 5 months of Budget Portfolio: Exploration, Exploration reserve in the past year, and as of February 2017, it had 1 month of schedule Systems Development reserve remaining. According to program officials, this reserve is being held for the program’s integrated test period with SLS and Orion. As a result, the EGS program has no reserves left to address any issues or challenges associated with its own development or integration and test phase. Program officials said NASA management continues to work to mitigate the risk. The program is tracking schedule risks for the Mobile Launcher and one of its two major software development efforts.

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 0 $2,812.9 0% $2,812.9 MONTHS CHANGE 4.4% 11/18 11/18 $1,843.5 $1,847.1 LAUNCH LAUNCH DATE DATE

EGS

Other major projects

95.6% $969.4 $965.8 Operations total needed to be funded Development then-year dollars in millions 02/12 02/12 Formulation $2,812.9 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2014 FEB. 2017 FY 2014 FEB. 2017 $913.4

common name: EGS 45 Assessments of Major NASA Projects GAO-17-303SP EXPLORATION GROUND SYSTEMS

Cost and Schedule Status The EGS program’s schedule is deteriorating and it is at program officials, some content has been deferred due in increased risk of exceeding its cost baseline and missing its part to Orion and SLS software delays. Due to the ongoing November 2018 launch readiness date. The program has delays, the program may complete GFAS development experienced multiple technical challenges and delays and as late as August 2018—6 months later than previously has limited cost and schedule reserves remaining to address planned. EGS officials said this delay will not interfere issues as they arise. It has used 5 months of schedule with integrated testing and operations with SLS and Orion reserve over the past year and, as of February 2017, it had 1 because key parts of the software, such as the launch month of schedule reserve remaining. According to program countdown capability, will not be needed until later in the officials, this reserve is being held for the program’s integrated integration process. The EGS program is still tracking a risk test period with SLS and Orion. As a result, the EGS program that GFAS may not be ready when needed and it has hired has no reserves left to address any issues or challenges additional staff as part of its efforts to address this risk. associated with its own development or integration and test phase. Program officials said NASA management continues to work to mitigate the risk. The program’s schedule is also being revisited. Program officials stated they are working with NASA to conduct a schedule analysis for the Orion, SLS, and EGS programs that takes into account the 7-month continuing budget resolution for fiscal year 2017, a delay in the delivery of a key piece of the Orion spacecraft to Kennedy Space Center for integration and testing, and continued EGS technical challenges, among other issues. According to senior NASA officials, this analysis is likely to lead to a delay in the planned EM-1 launch date.

Design and Technical The EGS program has completed several projects, including the Crawler Transporter and Launch Vehicle Offline Processing facility, but technical challenges and delays with the remaining work could put the overall schedule at risk. The Mobile Launcher—which supports the assembly, testing, and servicing of the SLS rocket and provides the platform on which SLS and Orion will launch—has experienced multiple technical challenges and delays. For example, in last year’s assessment, we reported that there were design issues with a connection between the Mobile Launcher and SLS. According to EGS program officials, these issues are still being resolved. The program is also tracking several schedule risks related to launch equipment testing, ground support equipment installation, and analysis and testing of the Mobile Launcher. The EGS program does not have any reserves to accommodate further Mobile Launcher delays, and program officials stated that they have nearly exhausted other options to stay on schedule, such as adding extra shifts.

The EGS program has also experienced delays with the development and testing of one of its two main software efforts. The Ground Flight Application Software (GFAS)— software that will interface with flight systems and ground PROJECT OFFICE COMMENTS crews—continues to face challenges due to delays in the EGS project officials provided technical comments on other main EGS software effort and late deliveries from the a draft of this assessment, which were incorporated SLS and Orion programs. Over half of the GFAS software as appropriate. releases have been delivered; however according to EGS common name: EGS 46 Assessments of Major NASA Projects GAO-17-303SP Gravity Recovery and Climate Experiment Follow-On

The Gravity Recovery and Climate Experiment Follow-On (GRACE-FO) will continue and expand upon the 2002 GRACE mission, which remains in operation. The system, which consists of two spacecraft working together to obtain scientific measurements, will provide high-resolution models of Earth’s gravity field and insight into water movement on and beneath the Earth’s surface for up to 5 years. These models will provide rates of ground water depletion and polar ice melt and enable improved planning for droughts and floods. GRACE-FO is a collaborative effort with the German Research Centre for Geosciences (GFZ).

Source: NASA/JPL Caltech. | GAO-17-303SP

TION 01/12 07/12 01/14 02/14 02/15 07/15 01/17 12/17 - 02/18 02/18 Formulation System Preliminary Project Critical System GAO Manifested Committed start requirements/ design confirmation design integration review launch launch

FORMULA mission review review review readiness readiness

definition review IMPLEMEN TA TION date date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Jet Propulsion The GRACE-FO project has successfully integrated both of its spacecraft and Laboratory plans to launch by its February 2018 committed launch date despite a late International Partner: German Research change to its launch vehicle and launch site. In February 2016, the Russian Centre for Geosciences (Germany) Federal Space Agency notified GFZ that the Dnepr launch vehicle, which is manufactured by a Russian firm in conjunction with a Ukrainian firm, was Launch Location: Vandenberg Air Force no longer available for GRACE-FO due to Russian restrictions on Ukrainian Base, CA personnel accessing the launch site. The project now plans to launch on a Launch Vehicle: Falcon 9 SpaceX Falcon 9 with commercial satellites in a shared ride arrangement. Mission Duration: 5 years The launch vehicle change resulted in at least a 4-month delay in the project’s planned launch, but the project has adequate cost reserves to cover the cost of Requirement Derived from: NASA 2010 the delay. The project told us that they plan to use the extra time in the schedule Climate Plan to conduct additional risk reduction activities. Budget Portfolio: Science, Earth Science

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 0 $431.9 0% $431.9 MONTHS 0.4% CHANGE 02/18 02/18 LAUNCH LAUNCH DATE DATE

GRACE-FO

Other major projects

$60.5 $61.1 99.6% $264.0 $263.4

Operations total needed to be funded then-year dollars in millions Development $107.4 $107.4 01/12 01/12 Formulation $431.9 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2014 FEB. 2017 FY 2014 FEB. 2017 $104.3

common name: GRACE-FO 47 Assessments of Major NASA Projects GAO-17-303SP GRAVITY RECOVERY AND CLIMATE EXPERIMENT FOLLOW-ON Cost and Schedule Status The GRACE-FO project plans to launch by its committed February 2018 launch date despite a late change to its launch vehicle and launch site. The launch vehicle change resulted in at least a 4-month delay in the project’s planned launch. The project has adequate reserves to cover the cost of the delay.

Launch and Developmental Partner The GRACE-FO project has delayed its launch readiness date by at least 4 months, from August 2017 to December 2017, due to issues with its planned launch vehicle and launch site. The launch vehicle is the responsibility of NASA’s partner on the project—GFZ. In February 2016, the Russian Federal Space Agency notified GFZ that the Dnepr launch vehicle was no longer available for GRACE-FO due to Russian restrictions on launch site access by Ukrainian personnel who provided support for the launch vehicle. As a result, GFZ entered into a shared ride agreement with Iridium Communications Inc. to launch the two GRACE-FO spacecraft, along with five Iridium satellites, on a Falcon 9 from Vandenberg Air Force Base in California. GFZ also signed a contract with Airbus Defence and Space to provide launch service management and develop a new multi-satellite dispenser. The multi-satellite dispenser, which project officials told us is different from the one planned for use with the Dnepr, releases the satellites at the right time into the correct orbit.

GRACE-FO officials told us that they are aiming to prevent any significant design changes to its spacecraft because of the late launch vehicle change. Project officials only anticipate minor design changes, such as how the spacecraft attaches to the multi-satellite dispenser. The project is coordinating with GFZ to determine if the project needs to conduct any additional risk mitigation activities or environmental testing because of the launch vehicle and satellite dispenser changes.

Integration and Test The GRACE-FO project has finished building both spacecraft and has remained on schedule with its integration and testing activities. The project expects to begin integrated system testing on both spacecraft in January 2017. The project said they plan to use the extra time in the schedule due to the launch delays to conduct additional risk reduction activities during system-level integration and test. PROJECT OFFICE COMMENTS GRACE-FO project officials provided technical comments on a draft of this assessment, which were incorporated as appropriate.

common name: GRACE-FO 48 Assessments of Major NASA Projects GAO-17-303SP Ice, Cloud, and Land Elevation Satellite-2

The Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2) is a follow-on mission to ICESat that will measure changes in polar ice-sheet mass and elevation. The measurements will provide researchers a better understanding of the mechanisms that drive polar ice changes and their effect on global sea level. The ICESat-2’s upgraded laser instrument will allow the satellite to make more frequent measurements and provide better elevation estimates over certain types of terrain than ICESat.

Source: Orbital Sciences Corporation. | GAO-17-303SP

TION 12/09 05/11 10/12 12/12 02/14 05/14 01/17 TBD 06/18 09/18 Formulation System Preliminary Project Critical Project GAO System Committed Manifested start requirements/ design confirmation design rebaseline review integration launch launch

FORMULA mission review review review readiness readiness

definition review IMPLEMEN TA TION date date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Goddard Space Flight The ICESat-2 project has encountered problems with the flight lasers in its sole Center instrument—the Advanced Topographic Laser Altimeter System (ATLAS)—that International Partner: None will likely cause it to miss its committed launch date and could cause it to exceed its current cost baseline. The project was previously rebaselined in 2014 because Launch Location: Vandenberg Air Force of ATLAS-related development and design issues. During ATLAS environmental Base, CA testing in July 2016, the project observed a cracked crystal within the laser optical Launch Vehicle: Delta II module, and it is now working to repair the lasers. A spare laser encountered the same problem during earlier testing, which indicates a systemic problem. The Mission Duration: 3 years instrument carries two flight lasers and one must be working for the mission to Requirement Derived from: 2007 Decadal succeed. Due primarily to the laser problems, the project now plans to launch Survey no earlier than September 2018, which is 11 months later than its previously Budget Portfolio: Science, Earth Science planned October 2017 launch date, and 3 months later than its committed launch date. The project plans to complete the laser repair in parallel with system-level integration and testing in an effort to mitigate the effect of the delays.

a PROJECT BUDGET INFORMATION COST PERFORMANCE a SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 16 $860.3 23.6% $1,063.6 MONTHS 1.1% CHANGE 09/18 LAUNCH DATE 05/17 LAUNCH DATE ICESat-2

Other major projects

$50.7 98.9% $52.6 $763.7 $558.8 Operations total needed to be funded then-year dollars in millions Development $248.8 $249.1 Formulation 12/09 12/09 $1,063.6 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2013 FEB. 2017 FY 2013 FEB. 2017 $216.7 aThe ICESat-2 project expects to experience additional cost growth and schedule delays, and was re-evaluating their cost and schedule at the time of our review.

common name: ICESAT-2 49 Assessments of Major NASA Projects GAO-17-303SP ICE, CLOUD, AND LAND ELEVATION SATELLITE-2 Cost and Schedule Status The ICESat-2 project is likely to miss its committed launch that replicates the weight of a second laser, so that testing date and could exceed its current cost baseline. During could continue while the other lasers are repaired. The environmental testing for the project’s only instrument—the project will conduct additional testing after the repaired Advanced Topographic Laser Altimeter System (ATLAS)—in lasers are delivered, which is currently planned for August July 2016, it experienced a problem with a flight laser that 2017. required repairs and led to significant schedule delays. Due primarily to the laser problems, the project now plans to Launch launch no earlier than September 2018, which is 11 months The ICESat-2 launch delays could also pose challenges later than its previously planned October 2017 launch date, related to its launch vehicle. According to NASA and project and 3 months later than the committed launch date in its 2014 officials, ICESat-2 will be the last flight of United Launch baseline. The project was rebaselined in 2014, at which time Alliance’s (ULA) Delta II vehicle. NASA officials said staff its estimated costs increased by $203 million and its launch retention could be a concern because ULA may need to date was delayed by 13 months. According to project officials, reassign staff to other areas while the project resolves the the project will spend about $8 million per month for each ongoing laser issues. The ICESat-2 Delta II is already built month past its previously planned October 2017 launch; this and ULA will place it in storage. In addition, project officials includes penalties NASA must pay to the launch provider. said the project will incur a $2 million penalty each month For example, if the project launches in September 2018, its the launch is delayed past the previously planned October estimated cost would increase by about $88 million, which 2017 launch. would exhaust all the cost reserves for the project. The project is currently re-evaluating its cost and schedule and expects to complete that process in spring 2017.

Technical and Design The ICESat-2 project has continued to experience technical and design issues with the ATLAS instrument, which has posed challenges since the beginning of the project. The project is investigating an anomaly in one of its two flight lasers that occurred during environmental testing in July 2016. The project observed a cracked crystal within the laser optical module, removed the laser from ATLAS, and shipped it to the manufacturer’s facility for investigation and repair. The manufacturer determined the primary cause of the anomaly was a flaw in the design of the mount that ensures a component of the optical module remains in a specific, precise position. The spare flight laser encountered the same problem during earlier testing, which indicates a systemic problem. The project has encountered other problems with the lasers in the past. For example, the project conducted additional testing in 2016 to mitigate a risk that a crystal in the each of ATLAS’s two lasers may become de-bonded from its mount, which would result in loss of laser function. ICESat-2 only needs one laser for mission success, but will carry two for redundancy. According to officials, it could take at least 8 to 10 months to repair, reintegrate, and realign the two ATLAS flight lasers to address the most recent issues, which will likely cause the project to miss its committed launch date.

Integration and Test PROJECT OFFICE COMMENTS The ICESat-2 project has taken steps in integration and ICESat-2 project officials provided technical testing to try to mitigate the flight laser delays. The project comments on a draft of this assessment, which were shipped ATLAS to Orbital ATK in October 2016 for system- incorporated as appropriate. level integration and testing with one flight laser and a model common name: ICESAT-2 50 Assessments of Major NASA Projects GAO-17-303SP Interior Exploration using Seismic Investigations, Geodesy, and Heat Transport Project

The Interior Exploration using Seismic Investigations, Geodesy, and Heat Transport (InSight) is a Mars lander with two primary objectives. It is intended to further understanding of the formation and evolution of terrestrial planets by determining Mars’s size, its composition, and the physical state of the core; the thickness of the crust; and the composition and structure of the mantle, as well as the thermal state of the interior. It will also determine the present level of tectonic activity and the meteorite impact rate on Mars. InSight is based on the Phoenix lander design. Phoenix successfully landed on Mars in 2008.

Source: © California Institute of Technology. | GAO-17-303SP

TION 07/11 10/13 03/14 05/14 02/15 01/17 06/17 05/18 Formulation Preliminary Project Critical System GAO System Committed start design confirmation design integration review integration launch

FORMULA review review review review #2 readiness

IMPLEMEN TA TION date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Jet Propulsion The InSight project missed its committed launch date of March 2016 and Laboratory exceeded its cost baseline due to technical issues with its primary science International Partners: Centre National payload—the Seismic Experiment for Interior Structure (SEIS) instrument—which d’Etudes Spatiales (France) and German is contributed by the French space agency (CNES). NASA delayed the InSight Aerospace Center (Germany) launch by 26 months to May 2018, which is the next available planetary launch window, and increased the project’s estimated cost by almost $154 million. Launch Location: Vandenberg Air Force Base, CA Due to the earlier problems, NASA has taken over design and fabrication of the SEIS container from CNES. The SEIS container had to be redesigned to hold Launch Vehicle: Atlas V a vacuum, which is required for the instrument to perform properly. The project Mission Duration: 29 months plans to deliver the rebuilt SEIS to system-level integration and test in June 2017. The project has used the additional time from the delays to mitigate other Requirement Derived from: 2010 Decadal project risks, such as addressing anomalies identified late in testing on its other Survey instrument—the German-contributed Heat Flow and Physical Properties Probe, Budget Portfolio: Science, Planetary which drills into the Martian surface to collect temperature data. Science

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 26 $675.1 22.8% $828.9 MONTHS 0.9% CHANGE 05/18 LAUNCH DATE

$56.6 InSight $673.5 $34.4 03/16 LAUNCH Other DATE major $541.8 projects

99.1%

Operations total needed to be funded then-year dollars in millions Development $98.9 $98.9 07/11 07/11 Formulation $828.9 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2014 FEB. 2017 FY 2014 FEB. 2017 $175.8

common name: INSIGHT 51 Assessments of Major NASA Projects GAO-17-303SP INTERIOR EXPLORATION USING SEISMIC INVESTIGATIONS, GEODESY, AND HEAT TRANSPORT PROJECT

Cost and Schedule Status The InSight project missed its committed launch date of The InSight project has used the additional development March 2016 and exceeded its development cost baseline time it gained from the launch delay to mitigate other due to technical issues with the Seismic Experiment for project risks. For example, the Heat Flow and Physical Interior Structure (SEIS) instrument, the project’s primary Properties Probe (HP3)—a contributed instrument from the science instrument. In August 2016, NASA approved German Aerospace Center (DLR) that drills into the Martian the project’s replan—a process initiated if development surface to take temperature measurements—experienced costs increase by 15 percent or more and requires a anomalies late in testing when the hammering mechanism report to congressional committees. NASA delayed the damaged internal wiring. The project redesigned the InSight launch by 26 months to May 2018, which is the internal wiring and started to fabricate and integrate next available planetary launch window, and increased the new HP3 in January 2017. If the redesigned HP3 the project’s total estimated cost by almost $154 million, demonstrates improved robustness during life tests, the or almost 23 percent. The development cost increases project will use it for the mission. The project has also are primarily due to having to recomplete system-level identified operational workarounds to minimize the effects assembly, test, and launch operations activities. The project of the anomaly if it occurs again. has also increased its operations cost estimate by $22 million as it has gained a better understanding of what it will take to safely deploy InSight’s instruments on the ground with less risk.

Design and Developmental Partner NASA has redesigned the SEIS container to address the technical issues that caused the launch delays. Prior to its planned March 2016 launch, the InSight project experienced vacuum seal leaks in the SEIS container, which prevented it from properly sealing and holding a vacuum. A seal is required for the instrument to perform properly. The SEIS, a seismometer contributed by the Centre National d’Etudes Spatiales (CNES), is critical for meeting three of the project’s six top-level science requirements. Project analysis indicated the failure was related to a feedthrough leak. Feedthroughs provide a leak-free path for the wiring that connects the SEIS to the spacecraft computer. The project selected a new feedthrough vendor in September 2016 and received and tested the flight parts in December 2016, which resulted in a container delivery two months later than planned. The project adjusted its schedule to accommodate the delay, and plans to deliver the integrated SEIS to system-level integration and test in June 2017. The SEIS schedule remains one of the project’s top risks.

After the launch delay, NASA and CNES redefined their roles and responsibilities with regard to the SEIS. CNES is responsible for integration and test of the flight SEIS instrument, but NASA has taken over the design, testing, and development of the SEIS container. In addition, CNES reduced the role of the contractor that had been responsible for the container. According to project officials, PROJECT OFFICE COMMENTS CNES and NASA will provide on-site support to the InSight project officials were provided a draft of this contractor for its remaining SEIS work because, in addition assessment and did not have any comments. to the container failure, it had previously experienced development delays and quality assurance issues that could have caused contamination of flight hardware. common name: INSIGHT 52 Assessments of Major NASA Projects GAO-17-303SP Ionospheric Connection Explorer

The Ionospheric Connection Explorer (ICON) observatory will orbit Earth to explore its ionosphere—the boundary region between Earth and space where ionized plasma and neutral gas collide and react. Its four instruments will make direct measurements and use remote sensing to further researchers’ understanding of Earth’s upper atmosphere, the Earth-Sun connection, and the ways in which Earth weather drives space weather.

Source: University of California, Berkeley. | GAO-17-303SP

TION 09/11 01/14 07/14 10/14 04/15 08/16 01/17 07/17 10/17 Formulation System Preliminary Project Critical System GAO Manifested Committed start requirements/ design confirmation design integration review launch launch

FORMULA mission review review review readiness readiness

definition review IMPLEMEN TA TION date date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Goddard Space Flight The ICON project has experienced technical issues and delays in system Center integration and testing, but it still on track to launch in July 2017—3 months International Partner: None earlier than its committed launch date. The most significant delay occurred when a primary component of the spacecraft’s electronics control system failed Launch Location: Kwajalein (Marshall during system-level testing. The project delayed system-level testing activities Islands) by about 3 months, while it made design changes to the failed part and retested Launch Vehicle: Pegasus it. The project delays led to scheduling conflicts at the contractor’s test facility. The contractor was able to move testing to an alternate location, which helped Mission Duration: 2 years prevent additional delays. The project plans to complete system-level integration Requirement Derived from: 2013 ̵ 2022 and testing in April 2017 and ship the spacecraft for launch by May 2017. Decadal Survey in Solar Space Physics

Budget Portfolio: Science, Heliophysics

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 0 $252.7 0% $252.7 MONTHS 0.3% CHANGE 10/17 10/17 LAUNCH LAUNCH DATE DATE

ICON

Other major projects

99.7% $14.3 $17.9 $196.0 $196.0 Operations total needed to be funded Development then-year dollars in millions 09/11 09/11 $42.4 $38.8 Formulation $252.7 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2015 FEB. 2017 FY 2015 FEB. 2017 $64.2

common name: ICON 53 Assessments of Major NASA Projects GAO-17-303SP IONOSPHERIC CONNECTION EXPLORER

Cost and Schedule Status The ICON project plans to launch in July 2017—3 months According to project officials, the models were worth the earlier than its committed launch date. The project held investment because they saved time and money by retiring its system integration review and entered the assembly, integration risks. integration, and test phase in August 2016. The project plans to complete system-level integration and testing in April 2017 and ship the spacecraft for launch no earlier than May 2017. The project is holding cost and schedule reserves consistent with the level required by NASA center policy.

Integration and Test The ICON project discovered a design issue in system- level integration and testing that has caused delays, but has not affected the project’s planned launch date. Part of the spacecraft’s electronics control system experienced a failure during its first system-level performance test. Specifically, the electronics control system’s actuator board failed, due to a design error, and was damaged. To resolve the issue, the project modified the design and produced a replacement board. To minimize the effect on the schedule, the project used an engineering model to continue testing during the 4 months it took to address the issue. In January 2017, the electronics control system was shipped back to the contractor test facility to be reintegrated with the spacecraft and tested. The project planned to have all system-level testing complete in January 2017, but the electronics failure delayed the completion of testing by 3 months to April 2017. This delay, in part, led to contractor test facility conflicts with other programs and forced the project to move testing to an alternate location, which helped prevent further delays.

The ICON project experienced other technical issues with its spacecraft and payload in 2016, but had sufficient cost and schedule reserves to address them. For example, the project encountered prior technical problems with the electronics control system during spacecraft integration and test. The issue was isolated to a specific module and PROJECT OFFICE COMMENTS only occurred when the unit reached certain temperatures. In commenting on a draft of this assessment, ICON The project used 30 days of schedule reserve to test other project officials reported the project continues to components to reproduce the event and determined the progress toward the end of system-level testing, and configuration of its ground support equipment was the the ground segment has completed its first operational cause. readiness test. Project officials also said there are no In addition to maintaining adequate levels of reserves to known technical risks at this time on the spacecraft, address issues when they arose, the ICON project took payload, or ground segment and that sufficient steps to reduce integration-related risks by identifying reserves exist to achieve its planned launch date. The potential challenges early. To do so, the project built full- project recently moved its planned launch date from scale, high-fidelity models of every payload component June to July 2017 due to a delay in the delivery of two including the payload interface plate. The project used of the three segments of the Pegasus launch vehicle. the model to route harnesses, determine accessibility to Project officials also provided technical comments, instruments, design flight blankets, and determine the which were incorporated as appropriate. integration sequence prior to receiving the flight hardware. common name: ICON 54 Assessments of Major NASA Projects GAO-17-303SP James Webb Space Telescope

The James Webb Space Telescope (JWST) is a large, infrared-optimized space telescope designed to help understand the origin and destiny of the universe, the creation and evolution of the first stars and galaxies, and the formation of stars and planetary systems. It will also help further the search for Earth-like planets. JWST will have a large primary mirror composed of 18 smaller mirrors and a sunshield the size of a tennis court. Both the mirror and sunshield are folded for launch and open once JWST is in space. JWST will reside in an orbit about 1 million miles from the Earth.

Source: NASA. | GAO-17-303SP

TION 03/99 01/06 03/08 07/08 03/10 09/11 01/17 09/17 10/18 Formulation Mission/ Preliminary Project Critical Project GAO System Committed start system design confirmation design rebaseline review integration launch

FORMULA definition review review review readiness

review IMPLEMEN TA TION date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Goddard Space Flight The JWST project is on track for its planned October 2018 launch, but it still Center must complete a series of challenging integration and test activities. The project International Partners: European Space is holding schedule reserves above NASA center requirements, but it must Agency, Canadian Space Agency complete three of five major integration and test efforts, one of which has not yet begun. Integration and test is when problems are often identified and schedules Launch Location: Kourou, French Guiana tend to slip. The project is also facing cost pressures, due to cost increases on Launch Vehicle: Ariane 5 the observatory development and integration contract with Northrop Grumman. The contractor has maintained a larger workforce than planned in order to Mission Duration: 5 years (10 year goal) address technical issues while minimizing impacts to the project’s schedule. In Requirement Derived from: 2001 July 2016, the contractor submitted a proposal to NASA for the cost overruns. Astrophysics Decadal Survey The project is currently evaluating the proposal, including its effect, if any, on Budget Portfolio: Science, Astrophysics project costs; it does not expect to conclude negotiations before early 2017.

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 52 $4,963.6 77.8% $8,825.4 MONTHS CHANGE 10/18 8.5% LAUNCH DATE

JWST 06/14 $836.6 LAUNCH DATE Other $6,188.8 major projects

$582.4 91.5% $2,581.1

Operations total needed to be funded then-year dollars in millions $1,800.1 $1,800.1 Development Formulation 03/99 03/99 $8,825.4 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2009 FEB. 2017 FY 2009 FEB. 2017 $2,214.7

common name: JWST 55 Assessments of Major NASA Projects GAO-17-303SP JAMES WEBB SPACE TELESCOPE

Cost and Schedule Status The JWST project continues to operate within its 2011 The project is currently evaluating the proposal, including cost and schedule baseline. The project has used about 9 the effect on its cost reserves, and does not expect to months of the 14 months of schedule reserve established conclude negotiations before early 2017. in its 2011 replan, including about 1 month in early 2017 to address technical challenges. The project’s remaining Launch schedule reserves are at the level required by NASA’s The JWST project has been working with its launch Goddard Space Flight Center. The project will likely have to vehicle provider to expand its potential launch window in use some of its cost reserves to cover cost overruns on its the event the project experiences delays. According to observatory development and integration contract, but it is program officials, prior mass reduction efforts have made still negotiating the magnitude of the increase. the observatory lighter and resulted in more flexibility in launch dates. At its former expected mass, JWST could Integration and Test not launch for a period before and after the solstices, due The JWST project is currently in its riskiest phase of to its planned trajectory and its relationship to the moon. If development—integration and test. Some integration and the project missed its planned October 2018 launch date, test activities have taken longer than planned and the it would have had to wait until February 2019 for another project has had to use schedule reserves to keep the overall opportunity. schedule on track. In 2016, the project completed two of five major integration and test efforts for the instrument module and telescope. Two other efforts—integrating the telescope and instrument module together and integrating the spacecraft—are underway. In early 2017, the project used over 1 month of schedule reserves to address test anomalies on the telescope and instrument module element and to replace spacecraft propulsion system components that were damaged during testing. The final integration and test effort, which involves the entire observatory, is scheduled to begin in fall 2017.

In December 2016, we found that as integration and testing moves forward, the project will need to be able to reduce a significant amount of risk in a timely manner to stay on schedule.a Many of these risks relate to the project’s numerous deployments, such as the unfolding of the sunshield, or the over 100 single point failures, which, if they occur, could threaten the mission.

Contractor In December 2016, we found that the primary threat to the JWST project continues to be the ability of the observatory development and integration contractor, Northrop Grumman, to control its costs.b The contractor has continued to maintain a larger workforce than planned in order to address technical issues while minimizing impacts to the project’s schedule. Based on its projections at the beginning of the fiscal year, Northrop Grumman exceeded its monthly workforce projections for fiscal year 2016 by about 37 percent. In July 2016, the contractor submitted a proposal to NASA for the agency to cover the cost overruns, which was PROJECT OFFICE COMMENTS the first such proposal since the project’s replan in 2011. JWST project officials were provided a draft of this assessment and did not have any comments. aGAO, James Webb Space Telescope: Project Meeting Cost and Schedule Commitments but Continues to Use Reserves to Address Challenges, GAO-17-71 (Washington, D.C.: Dec. 7, 2016).

bGAO-17-71. common name: JWST 56 Assessments of Major NASA Projects GAO-17-303SP Landsat 9

Landsat 9 is the next satellite in the Landsat Program, which provides a continuous space-based record of land surface observations to study, predict, and understand the consequences of land surface dynamics, such as deforestation. The program is a joint mission between NASA and the U.S. Geological Survey. The Landsat data archive constitutes the longest continuous moderate-resolution record of the global land surface as viewed from space and is used by many fields, such as agriculture, mapping, forestry, and geology.

Source: Orbital ATK (artist rendering). | GAO-17-303SP

TION 03/15 06/16 01/17 08/17 10/17 03/18 08/19 12/20 Formulation System GAO Preliminary Project Critical System Projected start requirements/ review design confirmation design integration launch

FORMULA mission review review review readiness

definition review IMPLEMEN TA TION date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Goddard Space Flight The Landsat 9 project is on track to begin implementation in October 2017 with Center a low level of technology risk, but an ambitious schedule. Due to direction in the International Partner: None Explanatory Statement accompanying the Consolidated Appropriations Act, 2016, the project is working to an accelerated launch readiness date of December Launch Location: Vandenberg Air Force 2020, which is about 3 years earlier than originally planned. According to officials, Base, CA the earlier launch date will help ensure that the Landsat program is able to Launch Vehicle: TBD maintain two satellites on orbit simultaneously, which is a goal of the scientific community. The project stated that this schedule is aggressive, but plausible Mission Duration: 5 years because of the high use of heritage technologies, mature instrument designs, Requirement Derived from: National Plan and the extensive use of hardware from the prior land data continuity mission, for Civil Earth Observations Landsat 8. Landsat 9 will use the same two primary instruments as Landsat Budget Portfolio: Science, Earth Science 8; however, the project plans to make several design changes to its Thermal Infrared Sensor 2 (TIRS-2) instrument to improve its reliability and performance. The project plans to hold its confirmation review in October 2017, at which point it will establish its cost and schedule baseline.

PROJECT BUDGET INFORMATION PRELIMINARY COST a PRELIMINARY SCHEDULE percent of current portfolio over next 5 years then-year dollars in millions

3.5% 12/20 – 11/21 PROJECTED LAUNCH READINESS Landsat 9 DATE

Other major projects

96.5% $851 – $928

total funded to date then-year dollars in millions 03/15 aThis estimate is preliminary, as the project is in formulation and there is uncertainty regarding the costs associated with the design options LATEST ESTIMATE $116.4 being explored. NASA uses these estimates for planning purposes. FEB. 2017

common name: LANDSAT 9 57 Assessments of Major NASA Projects GAO-17-303SP LANDSAT 9 Cost and Schedule Status The Landsat 9 project entered the preliminary design The project also plans to use spare flight hardware from and technology completion phase in August 2016 with an Landsat 8, which provides cost, schedule, and technical ambitious schedule. Due to direction in the Explanatory benefits and poses risks. For example, the project identified Statement accompanying the Consolidated Appropriations and then mitigated a risk that the performance of the Act, 2016, the Landsat 9 project is working to an spare Landsat 8 focal plane modules for OLI-2 could accelerated launch readiness date of December 2020, have degraded in the interim, which would have affected which is about 3 years earlier than originally planned. mission performance. The project tested the spares and According to officials, the earlier launch date will help found that they met or exceeded Landsat 9 performance ensure that the Landsat program is able to maintain requirements. two satellites on-orbit simultaneously, which is a goal of the scientific community, but not a requirement for the program. The project plans to hold its confirmation review in October 2017, at which point it will formally establish its cost and schedule baseline. If the project is baselined with a December 2020 launch readiness date, the project may move into implementation with less schedule reserves than required by NASA center policy. As a result, the program may have less time than recommended to address known and unknown risks.

Project officials consider the accelerated schedule aggressive, but plausible because of the high use of heritage technologies, mature instrument designs, and the extensive use of hardware from the prior land data continuity mission, Landsat 8. For example, the spacecraft development schedule is expected to be 8 months shorter than it was for Landsat 8, but according to the project, having more mature and stable requirements and interfaces helps mitigate this risk.

Technology and Design The Landsat 9 project will enter implementation with mature technologies and stable instrument designs, which helps to minimize development risk. The design of Landsat 9 is largely based on Landsat 8 and uses existing or heritage technology. Utilizing heritage technologies and designs on projects can help to reduce risk and control costs when they are used for similar purposes in similar environments. The project has two primary instruments—the Operational Land Imager 2 (OLI-2) and the Thermal Infrared Sensor 2 (TIRS-2). The OLI-2 instrument is being built to the same design and by the same contractor used for Landsat 8, but at a lower cost. According to project officials, the cost for Ball Aerospace to build the OLI-2 instrument is approximately $76 million lower for Landsat 9. NASA plans to make several changes to the TIRS-2 instrument design, which will be built in-house at the Goddard Space Flight Center, to improve its reliability and performance. For PROJECT OFFICE COMMENTS example, the project added redundant electronics systems Landsat 9 project officials provided technical and survival heaters, improved telescopes, and better comments on a draft of this assessment, which were protection from orbital debris. The project expects to hold incorporated as appropriate. critical design reviews for both instruments by February 2017. common name: LANDSAT 9 58 Assessments of Major NASA Projects GAO-17-303SP Mars 2020

Mars 2020 is part of the Mars Exploration Program, which seeks to further understand whether Mars was, is, or can be a habitable planet. Its rover and science instruments will explore Mars and conduct geological assessments, search for signs of ancient life, determine potential environmental habitability, and prepare soil and rock samples for potential future return to Earth. The rover will include a technology demonstration instrument designed to convert carbon dioxide into oxygen. Mars 2020 is based heavily on the Mars Science Laboratory, or Curiosity, which landed on Mars in 2012 and remains in operation.

Source: NASA/JPL-Caltech. | GAO-17-303SP

TION 11/13 10/14 02/16 06/16 01/17 02/17 11/17 07/20 Formulation System Preliminary Project GAO Critical System Committed start requirements/ design confirmation review design integration launch

FORMULA mission review review review readiness

definition review IMPLEMEN TA TION date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Jet Propulsion The Mars 2020 project has not met key best practices for reducing product Laboratory development risk, but it has set aside significant cost and schedule reserves to International Partners: Centre National address its most challenging developments. The project held its preliminary and d’Etudes Spatiales (France), Centro de critical design reviews, in February 2016 and February 2017 respectively, with Astrobiología (Spain), Norwegian Defence a lower level of technology and design knowledge than recommended by best Research Establishment (Norway) practices. The project has since matured all but one of its technologies, but other Launch Location: Eastern Range, FL technical challenges and design-related issues still pose cost and schedule risks. For example, according to the project, substantial technical challenges remain Launch Vehicle: Atlas V with one of the instruments that will be used to detect organic compounds and Mission Duration: 2 years it may not be ready for integration as planned. Mass will also continue to be a risk. Finally, the project plans to conduct additional testing on its parachute Requirement Derived from: 2011 National design, which was previously used for the Mars Science Laboratory, and develop Research Council Decadal Survey a strengthened parachute, after parachute test failures on an unrelated NASA Budget Portfolio: Science, Planetary project. Science

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 0 $2,443.5 0% $2,443.5 MONTHS CHANGE 8.3% 07/20 07/20 LAUNCH LAUNCH DATE DATE

$376.0 $376.0 Mars 2020

Other $1,674.7 $1,674.7 major projects

91.7%

Operations total needed to be funded Development then-year dollars in millions $392.9 $392.9 11/13 11/13 Formulation $2,443.5 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2016 FEB. 2017 FY 2016 FEB. 2017 $1,834.2

common name: MARS 2020 59 Assessments of Major NASA Projects GAO-17-303SP MARS 2020

Cost and Schedule Status The Mars 2020 project entered the implementation exceed its mass requirements. It previously increased phase in June 2016 and formally established its cost and the maximum allowable mass from 980kg to 1050kg and schedule baselines. The cost baseline is higher than the formed an assessment team to evaluate the effect of any project’s preliminary estimate range because contributions future increases. Additional capabilities could still be added from outside the Science Mission Directorate, such as to the project. NASA has proposed including a helicopter a technology demonstration designed to convert carbon technology demonstration. The helicopter could assist dioxide into oxygen, were not included in earlier estimates. with route planning for the rover. NASA planned to make The project is holding cost and schedule reserves a decision on the helicopter by the Mars 2020 critical consistent with the level required by NASA center policy. design review, but the project was still studying how to accommodate it within the design. Technology The Mars 2020 project entered implementation with a The Mars 2020 project is also exploring whether it needs lower level of technology maturity than recommended by to make changes to its parachute design. NASA’s Low- best practices, but most technologies have since matured. Density Supersonic Decelerator parachute, which is The project held its preliminary design review in February different than the one that will be used for Mars 2020, 2016 with two of its seven critical technologies and one unexpectedly failed two tests. The project is concerned of its two heritage technologies matured to technology that there may be unknown parachute performance readiness level 6. Maturing technologies to this level is characteristics that could affect Mars 2020. To mitigate this a best practice, which helps minimize risks for space risk, the project plans to conduct additional testing on its systems entering product development. A Mars 2020 existing design and begin development on a strengthened project official said the project conducted the review with parachute. immature technologies to avoid delaying progress on other parts of the project. Schedule is a key driver for Mars 2020. If the project misses the 2020 launch window, it must wait 26 months before another launch opportunity is available. The project set aside a significant portion of its reserves to cover potential cost growth for its new and modified instruments, which would help mitigate remaining technology risk. As of January 2017, the project had not yet matured one technology—an improved sensor on the entry, descent, and landing instrument—but it could be replaced with a heritage sensor, if necessary.

The project has made progress addressing technical risks on its instruments, but several of its new developments still present cost and schedule risks. For example, the project has addressed prior concerns about the performance of the laser that will be used to detect organic compounds, but according to the project, substantial technical challenges remain with that instrument and it may not be ready for integration as planned in fall 2018. PROJECT OFFICE COMMENTS In commenting on a draft of this assessment, Mars Design 2020 project officials stated that by the project’s The Mars 2020 design is almost stable, but the project critical design review the project matured all of its will have to continue to manage mass closely. The project critical technologies and the project released 80 planned to release 80 percent of its design drawings percent of their design drawings, which is above by its late February 2017 critical design review. Best average for large projects. Project officials also stated practices show that releasing less than 90 percent of that rover mass is no longer considered a key risk design drawings by this review can increase the project’s and the project has approximately 80 kilograms of risk of cost growth and schedule delays. The project also continues to track a risk that the combined mass of the unallocated mass margin to address unforeseen rover, payload, and sampling and caching system could issues. Project officials also provided technical comments, which were incorporated as appropriate. common name: MARS 2020 60 Assessments of Major NASA Projects GAO-17-303SP NASA ISRO – Synthetic Aperture Radar

The NASA Indian Space Research Organisation (ISRO) - Synthetic Aperture Radar (NISAR) is a joint project between NASA and ISRO that will study the solid Earth, ice masses, and ecosystems. It aims to address questions related to climate change, Earth’s carbon cycle, and natural hazards, such as earthquakes and volcanoes. The project will include the first dual frequency synthetic aperture radar instrument, which will use advanced radar imaging to construct large-scale data sets of the Earth’s movements. NISAR represents the first major aerospace science partnership between NASA and ISRO.

Source: © 2016 California Institute of Technology/Jet Propulsion Laboratory. | GAO-17-303SP

TION 03/14 12/14 06/16 08/16 01/17 10/18 12/19 12/21 09/22 Formulation System Preliminary Project GAO Critical System Manifested Committed start requirements/ design confirmation review design integration launch launch

FORMULA mission review review review readiness readiness

definition review IMPLEMEN TA TION date date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Jet Propulsion The NISAR project took steps to reduce its development risk prior to entering Laboratory implementation in August 2016, but it was baselined at a higher than expected International Partner: Indian Space cost and with a later than expected launch date. The primary drivers of the higher Research Organisation (India) cost estimate were near-term funding shortfalls that stretched out the schedule by 1 year and additional risk reduction activities for the radar reflector boom Launch Location: Satish Dhawan Space assembly. The radar reflector boom was one of three critical technologies the Centre, India project matured to the level recommended by best practices by its preliminary Launch Vehicle: Geosynchronous Satellite design review. This reduced the project’s overall development risk, but several Launch Vehicle Mark II boom assembly risks remain. The project is also tracking risks related to Mission Duration: 3 years differences in the ways NASA and ISRO manage projects, and the reliability of its ISRO-contributed launch vehicle, but has made progress mitigating these risks. Requirement Derived from: National Research Council Decadal Survey

Budget Portfolio: Science, Earth Science

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 0 $866.9 0% $866.9 MONTHS 2.5% CHANGE 09/22 09/22 LAUNCH LAUNCH DATE DATE $88.9 $88.9

NISAR $661.0 $661.0

Other major projects

97.5%

Operations total needed to be funded Development then-year dollars in millions $117.0 $117.0 03/14 03/14 Formulation $866.9 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2016 FEB. 2017 FY 2016 FEB. 2017 $664.4

common name: NISAR 61 Assessments of Major NASA Projects GAO-17-303SP NASA ISRO – SYNTHETIC APERTURE RADAR

Cost and Schedule Status Developmental Partner In August 2016, the NISAR project held its project Because NISAR is the first major partnership between confirmation review and established its cost and schedule NASA and ISRO, the project continues to track a risk that baselines. The cost baseline is $59 million higher than the differences in NASA and ISRO management processes top end of the project’s preliminary cost estimates, primarily could negatively affect cost and schedule. The project due to near-term funding shortfalls that stretched out the established a joint management plan that outlines roles and schedule by 1 year and additional risk reduction activities responsibilities for each agency and the cooperative project for the radar reflector boom—a complex and critical path plan to help mitigate this risk. development—and radar reflector. The boom will be used to deploy the radar reflector as part of the radar reflector One of the main ISRO contributions to the NISAR project boom assembly that will be used to transmit and receive is the launch vehicle, but the Geosynchronous Satellite radar signals. The launch date was also delayed by 1 Launch Vehicle (GSLV) Mark II must meet all NASA-ISRO year. The project would have needed more funding than agreed-upon criteria before it can be used. In September planned in fiscal years 2017 and 2018 to meet its original 2016, the GSLV Mark II had its third consecutive successful launch date. The project is currently holding cost and launch, which is one of the criteria. schedule reserves at the amount required by Jet Propulsion Laboratory policy.

Technology The NISAR project met a key best practice by maturing its three critical technologies to a technology readiness level 6 by its preliminary design review. Maturing critical technologies—including the radar reflector boom—to this level helps minimize risks for space systems entering product development. The project spent several years and approximately $63 million prior to beginning the concept and technology development phase to mature the technology associated with the synthetic aperture radar and reduce associated risks.

Design The NISAR project is working to address risks related to its radar reflector boom assembly and mass as it finalizes the system design. The project is redesigning the boom assembly to increase its reliability. The project identified several single point failures on the boom that could prevent it from latching in place, which could compromise the mission. The project changed its electronics and hinge designs to mitigate this risk. The project also added environmental and deployment tests for the radar reflector boom assembly to reduce risk prior to system integration and testing.

The NISAR project is not meeting project mass margin requirements and is tracking the system’s mass as a significant risk. Maintaining adequate mass margins is a key indicator of design stability. Since the project’s preliminary design review, ISRO has reported a spacecraft mass increase and the project has made design changes, PROJECT OFFICE COMMENTS such as increasing the thickness of its electronics box, that In commenting on a draft of this assessment, have also increased mass. The project is working to better NISAR project officials stated that they agreed with understand the spacecraft changes and reduce mass in the assessment, and did not have any technical other areas. comments. common name: NISAR 62 Assessments of Major NASA Projects GAO-17-303SP Orion Multi-Purpose Crew Vehicle

The Orion Multi-Purpose Crew Vehicle (Orion) is being developed to transport and support astronauts beyond low-Earth orbit, including traveling to Mars or an asteroid. The Orion program is continuing to advance development of the human safety features, designs, and systems started under the Constellation program, which was canceled in 2010. Orion is planned to launch atop NASA’s Space Launch System (SLS). The current design of Orion consists of a crew module, service module, and launch abort system.

Source: NASA. | GAO-17-303SP

07/06 02/10 TION 02/12 08/14 12/14 09/15 10/15 01/17 11/18 04/23 Formulation Orion Formulation Preliminary Exploration Project Critical GAO First First crewed start under cancelled under start design flight confirmation design review non-crewed committed

Constellation Constellation FORMULA review test review launch launch

program program IMPLEMEN TA TION readiness date readiness date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Johnson Space Center The Orion program is increasingly at risk of missing the November 2018 launch date for its first uncrewed exploration mission (EM-1). The program has International Partner: European Space Agency limited cost reserves and no schedule reserves for EM-1, which compromises the program’s ability to address issues as they arise. The late delivery of the Launch Location: Kennedy Space Center, European Space Agency-contributed service module (ESM) is one of the main FL reasons that the program is at risk of missing the planned launch date. Since the Launch Vehicle: Space Launch System ESM’s critical design review in June 2016, its delivery to the Orion program has been delayed by at least 8 months and may face further delays. These delays Mission Duration: Up to 21 day active mission duration capability with four will affect other Orion program efforts—like software development—that need crew to be completed before the Orion crew and service modules can be provided to Kennedy Space Center for integration and launch preparations, as well as Requirement Derived from: NASA modeling and testing efforts for the Exploration Ground Systems (EGS) program. Authorization Act of 2010 The Orion program has made progress in key areas, such as heatshield design Budget Portfolio: Exploration, Exploration and production as well as structural testing. Systems Development

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 0 $11,283.5 0% $11,283.5 MONTHS CHANGE 04/23 04/23 EM-2 EM-2 22.9% $6,768.4 $6,616.7 LAUNCH LAUNCH DATE DATE

Orion

Other major projects 11/18 11/18 EM-1 EM-1 LAUNCH LAUNCH DATE DATE

77.1% $4,515.1 $4,666.8

Operations total needed to be funded a then-year dollars in millions Development Formulation 02/12 02/12 $11,283.5 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2015 FEB. 2017 FY 2015 FEB. 2017 $5,068.2

aIncludes funding through Exploration Mission-2.

common name: ORION 63 Assessments of Major NASA Projects GAO-17-303SP ORION MULTI-PURPOSE CREW VEHICLE

Cost and Schedule Status Integration and Test The Orion program is increasingly at risk of missing the According to program officials, testing for EM-1 is November 2018 launch date for its first uncrewed exploration proceeding according to plan, but looming schedule mission (EM-1). The program has limited cost reserves and, delays could have a cascading effect on EM-2 due to the according to officials, no schedule reserves for EM-1, which program’s test strategy. The program plans to perform compromises the program’s ability to address issues as they testing for EM-2 using flight hardware from EM-1 in order arise. Until now, the program has been able to rearrange its to reduce costs, but this could pose a schedule risk. schedule when additional time was needed for a particular Program officials said that this plan places the program effort. However, prime contractor and program officials both at risk of significant delays for EM-2 should the crew stated that as the program gets closer to the EM-1 launch module suffer damage during the first mission. To mitigate readiness date, the ability to shift work decreases. NASA is this risk, program officials stated that they have set aside currently conducting a schedule analysis for the Orion, SLS, time between EM-1 and testing for EM-2 to refurbish the and EGS programs and, according to senior NASA officials, is spacecraft’s avionics and replace damaged parts with likely to delay the planned EM-1 launch date. spares. If EM-1 is delayed, the schedule risks for EM-2 would increase. Developmental Partner The late delivery of the European Space Agency (ESA)- Funding contributed service module (ESM) is one of the main reasons The Orion program’s internal EM-2 cost goal of $10.8 billion that the Orion program is at risk of missing the planned EM-1 and launch readiness date of August 2021 is not supported launch date. ESA is responsible for designing, developing, by NASA’s planned budget requests. To stay on this and providing a service module—which provides air, water, schedule, NASA is counting on receiving more appropriated power, and propulsion to Orion during in-space flight—for funds than it plans to request. EM-1 and EM-2, the first crewed mission. Since the ESM’s critical design review in June 2016, the ESM delivery date has been delayed by 8 months to September 2017 with a risk of an additional 2-month delay. According to program officials, the delays are largely due to late deliveries to the service module contractor and underestimating the effort needed to complete the service module itself. If NASA cannot mitigate these delays, it will likely miss the planned EM-1 launch date. According to program offices, the program needs 12 months to integrate and test the ESM with the crew module following delivery from ESA. NASA officials stated the complete Orion crew vehicle must be delivered to Kennedy Space Center by July 2018 to maintain the planned EM-1 launch readiness date. The ESM delivery delays have also affected Orion software development and may negatively affect EGS’s schedule, which needs Orion software and modeling data to perform integrated testing. PROJECT OFFICE COMMENTS

Design In commenting on a draft of this assessment, Orion The Orion program has made progress in a key outstanding program officials said the program continues to area of the crew vehicle design—the heatshield. The program successfully manage the complex and challenging redesigned the heatshield to improve its structural strength design, development, and testing of the EM-1 and has nearly completed production of the heatshield blocks components in a budget and schedule constrained for EM-1. The program previously switched from a single- environment. In addition, officials said the EM-2 piece, or monolithic, heatshield design to one that employs development has commenced with the initiation of blocks similar in concept to those used on the Space Shuttle. machining of the primary crew module structure and The program plans to complete testing that will verify the ESA committing to produce a second service module. blocks properly adhere and bond to representative crew The program stated that it remains on track to meet module hardware by May 2017. its schedule baseline for EM-2 of no later than

April 2023. Program officials also provide technical comments, which were incorporated as appropriate. common name: ORION 64 Assessments of Major NASA Projects GAO-17-303SP Plankton, Aerosol, Cloud, ocean Ecosystem

The Plankton, Aerosol, Cloud, ocean Ecosystem (PACE) is a polar-orbiting mission that will use advanced global remote sensing instruments to improve scientists’ understanding of ocean biology, biogeochemistry, ecology, aerosols, and cloud properties. PACE will extend climate-related observations begun under earlier NASA missions, which will enable researchers to study long-term trends on Earth’s oceans and atmosphere, and ocean-atmosphere interactions. PACE will also enable assessments of air and coastal water quality, such as the locations of harmful algae blooms.

Source: NASA. | GAO-17-303SP (Note: Ocean Color Instrument pictured above.)

TION 06/16 01/17 01/17 09/18 12/18 09/19 06/21 08/22 Formulation GAO System Preliminary Project Critical System Projected start review requirements/ design confirmation design integration launch

FORMULA mission review review review readiness

definition review IMPLEMEN TA TION date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Goddard Space Flight The PACE project entered the concept and technology development phase in Center June 2016 and is using a unique design-to-cost process to determine what set International Partner: TBD of baseline science capabilities are achievable within the mission’s $805 million cost cap. The project held its system requirements and mission definition review Launch Location: TBD in January 2017, at which point a project typically sets its top-level requirements. Launch Vehicle: TBD However, under the design-to-cost process, the project will continue to analyze mission, spacecraft, and instrument concepts through project confirmation. NASA Mission Duration: 3 years has made several key decisions about PACE’s spacecraft and payload. The Requirement Derived from: 2010 Decadal agency decided to build the spacecraft in-house, rather than buy it commercially, Survey and procure a secondary instrument, a polarimeter. The project is also exploring options to reduce its launch costs, such as arranging for a foreign partner to Budget Portfolio: Science, Earth Science contribute a launch vehicle. The President’s 2018 Budget Blueprint has proposed canceling PACE.

PROJECT BUDGET INFORMATION PRELIMINARY COST a PRELIMINARY SCHEDULE percent of current portfolio over next 5 years then-year dollars in millions

3.1% 2022 – 2023 PROJECTED LAUNCH READINESS DATE PACE

Other major projects

96.9% $805 – $850

total funded to date then-year dollars in millions 06/16 aThis estimate is preliminary, as the project is in formulation and there is uncertainty regarding the costs associated with the design options LATEST ESTIMATE $70.3 being explored. NASA uses these estimates for planning purposes. FEB. 2017

common name: PACE 65 Assessments of Major NASA Projects GAO-17-303SP PLANKTON, AEROSOL, CLOUD, OCEAN ECOSYSTEM

Cost and Schedule Status costs, such as arranging for a foreign partner to contribute The PACE project entered the concept and technology a launch vehicle or sharing a launch vehicle with another development phase in June 2016 with a cost cap of mission. The project does not expect NASA to select a $805 million for the mission. The project office has been launch vehicle until 2019. allocated $705 million for portions of the development and acquisition of the spacecraft and instruments, and launch costs. NASA headquarters was allocated $100 million for science-related activities, such as the calibration and validation of instrument data and processing of science data. The project completed its system requirements and mission definition review in January 2017 and plans to enter the preliminary design and technology completion phase in late March 2017. In March 2017, the President’s 2018 Budget Blueprint proposed canceling PACE.

Design The PACE project is using a unique design-to-cost process to determine what set of baseline capabilities is achievable within the project’s $705 million cost cap at the 65 percent confidence level. NASA headquarters selected PACE as the first project to use this process and hopes to use it for other missions directed by NASA as a way to control cost, while maximizing mission capabilities. NASA already uses cost caps for missions it initiates through competitions, such as those conducted under the Discovery program. The PACE project has defined a set of notional baseline capabilities and a preferred mission concept; however, it will continue to analyze mission, spacecraft, and instrument concepts up through project confirmation, which is later than the typical project. During that time, the project will continue to look for ways to maximize its science capabilities and minimize costs.

NASA has made several key decisions about PACE’s spacecraft and payload. In August 2016, NASA held an acquisition strategy meeting and decided to build the spacecraft in-house at the Goddard Space Flight Center, rather than buy it commercially, and add a secondary instrument, a polarimeter. PACE project officials preferred an in-house spacecraft build because they said their analysis indicated it would cost less and enable synergies with its primary instrument, the Ocean Color Instrument (OCI), which will also be built in-house with industry support. An independent review board has previously noted that the project’s schedule would be aggressive for an in- house spacecraft build.

Launch One of the PACE project’s top risks is that its launch PROJECT OFFICE COMMENTS vehicle will cost more than initially estimated, which could PACE project officials provided technical comments cause it to exceed its cost cap or have to reduce its science on a draft of this assessment, which were capabilities. The project is working with NASA’s Launch incorporated as appropriate. Services Program to explore options to reduce its launch common name: PACE 66 Assessments of Major NASA Projects GAO-17-303SP Radiation Budget Instrument

The Radiation Budget Instrument (RBI) is a scanning radiometer that will launch on the National Oceanic and Atmospheric Administration’s (NOAA) Joint Polar Satellite System 2 (JPSS-2). RBI will support global climate monitoring by continuing measurements of the Earth’s reflected sunlight and emitted thermal radiation made by NASA and NOAA satellites over the past 30 years. This data represents one of two key sets of measurements needed to determine whether the Earth is warming or cooling. RBI will also extend unique global climate measurements made by the Clouds and the Earth’s Radiant Energy System instrument since 1998.

Source: Harris Corporation. | GAO-17-303SP

TION 07/14 12/14 05/16 07/16 01/17 06/17 04/19 12/19 Formulation System Preliminary Project GAO Critical Target Committed start requirements design confirmation review design delivery delivery

FORMULA review review review date date IMPLEMEN TA TION

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Langley Research The RBI project has met key best practices for reducing product development Center risk, but it may be unable to meet its aggressive schedule goals. In July 2016, the International Partner: None project entered implementation and NASA committed to deliver RBI to NOAA’s JPSS-2 program by December 2019—8 months later than the April 2019 JPSS- Launch Location: Not applicable 2 need date. The project is working to meet the need date, but it appears to be Launch Vehicle: Not applicable; instrument unrealistic. NASA’s joint cost and schedule confidence level analysis indicated hosted on JPSS-2 spacecraft that the likelihood of the project meeting the date is low and the project’s independent review board described the schedule as optimistic when compared Mission Duration: 7 years to similar instruments. The project has taken steps to reduce development risks Requirement Derived from: 2007 Decadel by maturing its critical technologies and stabilizing its design, but it is still falling Survey behind schedule. The prime contractor has experienced delays in completing Budget Portfolio: Science, Earth Science the engineering development unit that proves out the design before the flight instrument is built, as well as continual cost overruns as it consumes more resources than planned to try to stay on schedule.

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 0 $304.8 0% $304.8 MONTHS 0.7% CHANGE 12/19 12/19 LAUNCH LAUNCH DATE DATE

RBI

Other major projects

99.3% $202.0 $202.0 Operations total needed to be funded then-year dollars in millions Development $102.8 $102.8 07/14 07/14 Formulation $304.8 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2016 FEB. 2017 FY 2016 FEB. 2017 $162.9

common name: RBI 67 Assessments of Major NASA Projects GAO-17-303SP RADIATION BUDGET INSTRUMENT

Cost and Schedule Status The RBI project entered the implementation phase in was renegotiated in March 2016. By the time the project July 2016 and formally established its cost and schedule entered implementation, the value of the Harris contract baselines. In the baseline, NASA committed to deliver had grown by $62 million, or 59 percent, from its original RBI to the JPSS-2 program in December 2019, which value, primarily due to higher than expected subcontractor is 8 months later than the April 2019 JPSS-2 need date. costs. The project added a deputy project manager and The project is working to meet the April 2019 date, but increased site visits and meetings with Harris to improve it appears to be unrealistic. NASA has calculated the subcontractor oversight. However, the contractor has joint cost and schedule confidence level, which is the continued to incur more costs than planned as it tries to likelihood a project will meet its cost and schedule estimate, meet the aggressive instrument development schedule. for that date at 42 percent, which is lower than the 50 The project currently estimates that Harris will overrun percent generally required by NASA policy. The project’s the renegotiated contract by $16 million. The project has independent review board also described the RBI schedule cost reserves to cover the projected overrun, but using as optimistic when compared against historical data for those reserves would leave it with less than the Langley similar instruments. recommended level of reserves for the remainder of the project. As a result of the continual overruns, the project RBI project officials stated that the schedule is aggressive, plans to re-evaluate its schedule, reserve levels, and risks, but still feasible. The project has evaluated actions it could and identify potential areas to descope. take to try to preserve its schedule, such as using extended or double shifts. If the RBI project cannot meet the need date, JPSS-2 can fly without the instrument because RBI is not required to meet JPSS-2’s science requirements. The project could also get some relief if the JPSS-2 project experiences delays.

Technology and Design The RBI project has met key best practices for reducing product development risk; however, it is still having difficulty maintaining its aggressive schedule. The project matured its three critical technologies to a technology readiness level 6, the level recommended by best practices, prior to its May 2016 preliminary design review. The project also released all of its expected design drawings in advance of its critical design review, which is an indicator of design stability. However, the project delayed its planned April 2017 critical design review by 2 months to June 2017 because components of its engineering development unit—a model of the system used to help validate the design and its performance—have taken longer to complete than expected. For example, the project expects the engineering development unit’s telescopes to be delivered 4 months late because the project had to repair telescope PROJECT OFFICE COMMENTS mirrors that were damaged during the process of hardening In commenting on a draft of this assessment, RBI the coating on other areas of the telescopes. The project is project officials noted that they were taking multiple assessing these delays on its overall schedule. actions to contain project costs and maintain schedule. For example, the project recently reduced Contractor the scope of the engineering development unit to The RBI project’s prime contractor Harris continues to preserve cost and schedule. However, a NASA official experience cost overruns. The project awarded a cost- noted that reducing the scope of the engineering plus award fee contract to Harris in May 2014. In May development unit may also increase technical and 2015, the prime contractor notified the project that its estimated costs had significantly increased. Langley programmatic risks to the RBI flight instrument. Research Center issued a stop work order and the contract Project officials also provided technical comments, which were incorporated as appropriate. common name: RBI 68 Assessments of Major NASA Projects GAO-17-303SP Solar Probe Plus

Solar Probe Plus (SPP) will be the first NASA mission to visit a star. Using the gravity of Venus, the spacecraft will orbit the Sun 24 times and gather information to increase knowledge about the solar wind, including its origin, acceleration, and how it is heated. SPP instruments will observe the generation and flow of solar winds from very close range and sample and take measurements of the Sun’s outer atmosphere, where solar particles are energized. To achieve its mission, parts of the spacecraft must be able to withstand temperatures exceeding 2,500 degrees Fahrenheit and endure blasts of extreme radiation.

Source: Johns Hopkins University/Applied Physics Lab. | GAO-17-303SP

TION 11/09 11/11 01/14 03/14 03/15 05/16 01/17 08/18 Formulation System Preliminary Project Critical System GAO Committed start requirements/ design confirmation design integration review launch

FORMULA mission review review review readiness

definition review IMPLEMEN TA TION date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Goddard Space Flight The SPP project remains on track to launch during its planetary launch window Center in 2018, even as it has experienced delays in its system integration and testing International Partner: None activities. The delays have not affected the overall project schedule, in part, because the project has had sufficient cost and schedule reserves to mitigate Launch Location: Kennedy Space Center, them. In July 2016, the project started system integration and testing, but delays FL in the delivery of key spacecraft components, such as a flight avionics box that Launch Vehicle: Delta IV-heavy class with serves as a communications hub, has affected its progress. The project has NASA-provided upper stage developed workarounds to try to mitigate the effect of the delays. The SPP Mission Duration: 7 years project is also working to address spacecraft computer processor issues that could degrade mission performance. The performance margins for the processor, Requirement Derived from: 2013-2022 Solar which could affect both the recording and transmission of science data, is lower and Space Physics Decadal Survey than expected. The project is incorporating software changes and considering Budget Portfolio: Science, Heliophysics other strategies for increasing the margins, and it may ultimately have to take other actions, such as making software updates after launch.

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 0 $1,553.4 0% $1,553.4 MONTHS 3.3% CHANGE 08/18 08/18 LAUNCH LAUNCH DATE DATE

SPP

Other major projects $250.6 $256.0 $1,055.7 $1,050.3 96.7%

Operations total needed to be funded then-year dollars in millions Development $247.1 $247.1 11/09 11/09 Formulation $1,553.4 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2014 FEB. 2017 FY 2014 FEB. 2017 $761.1

common name: SPP 69 Assessments of Major NASA Projects GAO-17-303SP SOLAR PROBE PLUS

Cost and Schedule Status The SPP project remains on track to launch during in another part of the project—the manufacturing of the its planetary launch window in 2018, even as it has thermal protection system. In addition to these delays with experienced delays in its system integration and testing spacecraft components, the project is tracking a number of activities. Maintaining the project’s 2018 planetary launch instrument-related risks that could cause integration and window is important because the window only opens every testing delays. 10 months and subsequent launch windows would result in a longer mission duration and require more fuel. The Design project has used cost and schedule reserves to address The SPP project demonstrated it had a stable design delivery delays for multiple spacecraft components. at its system integration review in May 2016—over one The project has low cost reserves because it consumed year after its critical design review, the point at which reserves quicker than expected, but is still holding schedule product development best practices recommend having reserves at the level recommended by the Johns Hopkins a stable design. Last year, we reported that the project University Applied Physics Laboratory guidance. held its critical design review with only 34 percent of its design drawings released, significantly lower than the best Technology practice of 90 percent. Our work on product development The SPP project is working to address spacecraft best practices shows that releasing at least 90 percent computing issues, which could affect mission performance. of engineering drawings by this review lowers the risk of The performance of its spacecraft computer processor is subsequent cost and schedule growth. lower than expected and it is currently operating below its required performance margin for this stage in the project. The processor performance affects both the recording and transmission of science data. If the available processor margin continues to erode, then the project risks losing science data during solar encounters. According to project officials, the project has incorporated software changes that have significantly increased the performance margins for the processor. However, if these efforts do not fully mitigate the problem, the project may need to re-write portions of its flight software or make software updates after launch.

Integration and Test The SPP project experienced delivery delays with multiple spacecraft components, which has affected its system- level integration and test schedule. The delays have not affected the overall project schedule, in part, because the project has had sufficient reserves to mitigate them. The redundant electronics module, a flight avionics box, was delivered for system-level integration and testing in February 2017—3 months later than planned. The module serves as a communications hub for other spacecraft components. To mitigate the effect of the delay, the project used an engineering model to practice test procedures before the flight version was delivered. The solar array cooling system was delivered for system-level integration and testing in February 2017—4 months later than planned—due to a number of issues, including a test failure on one of its components and delayed deliveries PROJECT OFFICE COMMENTS from vendors. For example, the project received two In commenting on a draft of this assessment, SPP flight platens—which transport water within the system project officials stated that they agreed with the to cool the spacecraft’s solar cells—5 months later than expected, due primarily to manufacturing difficulties. The assessment. They provided technical comments on a cooling system issues also caused a corresponding delay draft of this assessment, which were incorporated as appropriate. common name: SPP 70 Assessments of Major NASA Projects GAO-17-303SP Space Launch System

The Space Launch System (SLS) is intended to be NASA’s first human- rated heavy-lift launch vehicle since the Saturn V was developed for the Apollo program. SLS is planned to launch NASA’s Orion spacecraft and other systems on missions between the Earth and moon and eventually to Mars. NASA is designing SLS to provide an initial lift capacity of 70 metric tons to low- Earth orbit and be evolvable to 130 metric tons, enabling deep space missions. The 70-metric-ton capability will include a core stage, powered by four RS-25 engines and two five-segment boosters. The 130-metric-ton capability will use a new upper stage and advanced boosters.

Source: NASA. | GAO-17-303SP

TION 11/11 05/12 07/13 08/14 07/15 01/17 11/18 Formulation Mission/ Preliminary Project Critical GAO Committed start system design design confirmation design review launch

FORMULA review review review readiness

IMPLEMEN TA TION date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Marshall Space Flight The SLS program’s schedule is deteriorating and it is at increased risk of Center exceeding its cost baseline and missing its November 2018 launch readiness International Partner: None date. The program has delayed its internal launch readiness goal and has less cost and schedule reserves available than recommended by NASA center policy Launch Location: Kennedy Space Center, to address known and unknown development risks. The program is working FL to resolve known issues with hardware components, including the core stage Launch Vehicle: N/A and booster. Program officials stated that they have also replanned software development to accommodate more testing, among other reasons. Furthermore, Mission Duration: Varied based on destination the integration of SLS into a complete launch vehicle and the phase when the launch vehicle will be integrated with ground systems and the Orion crew Requirement Derived from: NASA capsule could reveal unforeseen challenges leading to cost growth and schedule Authorization Act of 2010 delays. Budget Portfolio: Exploration, Exploration Systems Development

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 0 $9,695.4 0% $9,695.4 MONTHS CHANGE 10.6% 11/18 11/18 LAUNCH LAUNCH DATE DATE

SLS $7,021.4 $7,021.4

Other major projects

89.4%

Operations $2,674.0 $2,674.0 total needed to be funded then-year dollars in millions Development 11/11 11/11 Formulation $9,695.4 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2014 FEB. 2017 FY 2014 FEB. 2017 $2,383.5

common name: SLS 71 Assessments of Major NASA Projects GAO-17-303SP SPACE LAUNCH SYSTEM

Cost and Schedule Status Integration and Test The SLS program’s schedule is deteriorating and it is at The SLS program must complete two distinct levels of increased risk of exceeding its cost baseline and missing its integration—integration of the SLS launch vehicle and November 2018 launch readiness date. In May 2016, the SLS cross-program integration of SLS with Orion and ground program delayed its internal goal for launch readiness from support systems—before it can achieve launch readiness July 2018 to September 2018, which means the program only in 2018. The integration period often reveals unforeseen has 2 months of schedule reserve remaining. This is lower challenges leading to cost growth and schedule delays. than the amount of schedule reserve that is standard for this stage of a program according to Marshall Space Flight Center Contractor guidance. The program was also baselined with a lower than The value of the SLS core stage development contract recommended level of cost reserves. As a result, the program increased by $962 million in February 2017. The SLS has less time and money than recommended to address program and Boeing reached agreement to modify the known and unknown risks without affecting its baseline. scope of the contract to develop, test and deliver a new exploration upper stage for the vehicle that the program The core stage development is the pacing item for the SLS plans to use during Exploration Mission-2. program and its schedule is aggressive. For example, there is no schedule reserve remaining for core stage development leading up to the “green run” test scheduled for October 2017. This test is the culmination of the development effort and includes the core stage flight article integrated with four RS- 25 engines that NASA plans to fire for about 500 seconds in a simulated flight profile. If the test does not occur as scheduled or the stage does not perform as expected, core stage delivery to Kennedy Space Center could be delayed beyond the currently scheduled date of March 2018. The program only has 20 days of reserve between the end of the test and the date the core stage is needed at Kennedy Space Center for the start of integration.

Design In July 2016, we reported that the SLS program had matured the launch vehicle’s design, but it is still working to address known hardware and software challenges.a The program’s current hardware challenges include the strength of the core stage tank welds and booster-related issues. NASA officials indicated they have identified and corrected the root cause of low strength welds in the core stage tanks and that welding of test articles has resumed. NASA is also determining the extent to which new materials within the booster can withstand the structural stress of the SLS launch environment due to effects of long-term storage prior to launch. In addition, the SLS program is assessing the potential risk of damage from booster debris impacting the engines during launch. The program has efforts underway to further analyze or mitigate these and other issues. The SLS program also has encountered delays with the development of SLS flight software due to the late delivery PROJECT OFFICE COMMENTS of core stage avionics models needed for testing. These In commenting on a draft of this assessment, the SLS development delays consumed software schedule reserves program stated that it had made progress resolving and affected the maturity of the 12th software release. The challenges, including determining the effects of program has 14 planned software releases. Program officials long-term storage on new booster materials and said that the final software release was moved from 2017 to developing software. Program officials also provided 2018 to allow for more testing, among other reasons. technical comments, which were incorporated as aGAO, NASA Human Space Exploration: Opportunity Nears to Reassess Launch Vehicle and appropriate. Ground Systems Cost and Schedule, GAO-16-612 (Washington, D.C.: July 27, 2016). common name: SLS 72 Assessments of Major NASA Projects GAO-17-303SP Space Network Ground Segment Sustainment

The Space Network Ground Segment Sustainment (SGSS) project plans to develop and deliver a new ground system for three Space Network sites, which provide essential communications and tracking services to NASA and non-NASA missions. Existing systems, based on 1980s technology, are increasingly obsolete and unsustainable. The new ground system will include updated systems, software, and equipment that will allow the Space Network to continue to provide critical communications services for the next several decades. The Space Network is managed by the Space Communication and Navigation (SCaN) program.

Source: NASA. | GAO-17-303SP

TION 09/10 11/11 09/12 04/13 06/13 06/15 01/17 TBD 09/19 TBD Formulation Mission/system Preliminary Project Critical Project GAO System Committed final Projected final start definition design confirmation design rebaseline review integration acceptance acceptance

FORMULA review review review review review review IMPLEMEN TA TION

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Goddard Space Flight The SGSS project has exceeded the new cost and schedule baseline NASA set Center for it in June 2015 and further cost and schedule growth is likely. In December International Partner: None 2016, the project estimated that it would exceed its new cost baseline by more than $53 million and its committed completion date by at least 2 months. These Launch Location: N/A cost increases and schedule delays have occurred even as the scope of the Launch Vehicle: N/A project has decreased from upgrading three Space Network sites to one site. The project attributes the new cost and schedule growth to the contractor’s Mission Duration: 25 years with periodic, incomplete understanding of its requirements, which led to poor contractor plans required upgrades to hardware and software and late design changes. This is the latest in a series of contractor performance problems on the project. The project’s oversight of the contractor has also been Requirement Derived from: March 2008 an issue and it is taking steps to fill staff shortfalls in key areas. The project has Space Network modernization concept developed a cost and schedule baseline to track the project through fiscal year study 2017 and plans to present a new cost estimate and completion date to NASA by Budget Portfolio: Space Operations, Space August 2017. and Flight Support

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions AT LEAST $493.9 81.3% $895.6 27 1.5% CHANGE MONTHS TBD COMPLETION DATE $730.2

SGSS 06/17 COMPLETION Other DATE major projects

$368.1

98.5%

Operations total needed to be funded $165.4 Development then-year dollars in millions $125.8 Formulation 09/10 09/10 $895.6 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2013 NOV. 2016 FY 2013 NOV. 2016 $309.7 Note: The SGSS project has received an additional $365.7 million from Space Network users outside of NASA.

common name: SGSS 73 Assessments of Major NASA Projects GAO-17-303SP SPACE NETWORK GROUND SEGMENT SUSTAINMENT

Cost and Schedule Status Funding The SGSS project has exceeded the new cost and schedule The SGSS project experienced a funding shortfall in fiscal baseline NASA set for it in June 2015 and further cost and year 2017, in part due to contract cost overruns, that schedule growth is likely. The 2015 rebaseline increased led the SCaN program to indefinitely defer work on two SGSS’s estimated costs by $345 million and delayed its of SGSS’s three planned sites. The delays in the SGSS estimated completion by 27 months. In December 2016, project and the deferral of work at certain sites will result in the project estimated that its development costs exceeded increased operations and maintenance costs for the Space its new baseline by more than $53 million and its final Network. acceptance review would occur at least 2 months later than NASA’s committed date. These cost increases and schedule Other Issues to Be Monitored delays have occurred even as the scope of the project has In 2016, NASA announced it was reclassifying SGSS as a decreased from upgrading nine Space Network terminals at hybrid sustainment project for the Space Network, which three sites to six terminals at one site. The project attributes has implications for project oversight. A hybrid sustainment the latest cost and schedule growth to the contractor’s effort is a sustainment effort that still includes development incomplete understanding of its requirements, which led to work. A SCaN program official told us that NASA’s inability poor contractor plans and late design changes. The project to fund the project to its 2015 rebaseline agreement drove has developed a cost and schedule baseline to track the the reclassification of the SGSS project. As a result of no project through fiscal year 2017 and plans to present a new longer being considered a major project, NASA does not cost estimate and completion date to NASA by August 2017. plan to maintain its 2015 project rebaseline, update the joint cost and schedule confidence level analysis, or report on Contractor the project’s cost and schedule in its Major Program Annual The SGSS project continues to experience contractor Report to Congress. performance problems. The contractor is not adhering to its cost and schedule to deliver the fifth, final, and most complex software increment. For example, the project estimates completion of this increment to occur in June 2017, 6 months later than planned, with nearly $20 million in contract cost growth. The cost and schedule growth is due to the contractor moving a large portion of the software’s required functionality into this increment after experiencing problems with earlier increments, the need to address a growing backlog of defects, and late design changes that were not identified in the contractor’s plan. According to a NASA official, there have recently been some positive changes in the contractor’s performance. For example, the contractor has provided an improved schedule and is providing better real time data on delays. The contractor also replaced its project manager.

The project is developing a plan to try to address contractor performance problems, although it might not have the resources needed to provide sufficient oversight. The project is working with the contractor to develop a reliable schedule to measure progress. In addition, project officials said they may change contractor performance incentives—which in the past had focused on timeliness, rather than technical quality. The project also has a new NASA project manager, an increased physical presence at the contractor facility, and more staff focused on validation and verification activities, but according to the project, there are staff shortfalls in key areas, PROJECT OFFICE COMMENTS such as business, systems engineering, and development SGSS project officials provided technical comments management, which the project is working to address. on a draft of this assessment, which were incorporated as appropriate.

common name: SGSS 74 Assessments of Major NASA Projects GAO-17-303SP Surface Water and Ocean Topography

The Surface Water and Ocean Topography (SWOT) mission will use its wide-swath radar altimetry technology to take repeated high-resolution measurements of the world’s oceans and freshwater bodies to develop a global survey. This survey will make it possible to estimate water discharge into rivers more accurately, and help improve flood prediction. It will also provide global measurements of ocean surface topography and variations in ocean currents, which will help improve weather and climate predictions. SWOT is a joint project between NASA and the French Space Agency—the Centre National d’Etudes Spatiales (CNES).

Source: Caltech/JPL (artist depiction). | GAO-17-303SP

TION 11/12 05/14 04/16 05/16 01/17 02/18 07/19 04/21 04/22 Formulation System Preliminary Project GAO Critical System Manifested Committed start requirements/ design confirmation review design integration launch launch

FORMULA mission review review review readiness readiness

definition review IMPLEMEN TA TION date date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Jet Propulsion The SWOT project took steps to reduce its development risk prior to entering Laboratory implementation in May 2016. SWOT’s cost baseline is within preliminary International Partners: Centre National estimates, but its planned launch date is 6 months later than previously expected. d’Etudes Spatiales (France), Canadian The project added the 6 months to account for potential delivery delays with a Space Agency (Canada), United Kingdom component of its main instrument, the Ka-Band Radar Interferometer (KaRIn), Space Agency (United Kingdom) and to take actions to increase SWOT’s reliability. These and other steps help to Launch Location: Vandenberg Air Force reduce project risk in implementation. For example, the project matured its four Base, CA critical technologies to the level recommended by best practices by its preliminary design review. The KaRIn instrument is the most challenging development Launch Vehicle: Falcon 9 effort and biggest schedule driver for the project. The project is tracking a Mission Duration: 3 years risk that schedule delays could occur because multiple international partners are contributing components. We have previously reported that integrating Requirement Derived from: 2007 Decadal survey contributions from multiple partners may complicate development efforts and contribute to cost and schedule growth. Budget Portfolio: Science, Earth Science

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 0 $754.9 0% $754.9 MONTHS 2.1% CHANGE 04/22 04/22 LAUNCH LAUNCH DATE DATE

SWOT $46.7 $46.7 $571.5 $571.5 Other major projects

97.9%

Operations total needed to be funded then-year dollars in millions $136.8 $136.8 Development Formulation 11/12 11/12 $754.9 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2016 FEB. 2017 FY 2016 FEB. 2017 $480.5

common name: SWOT 75 Assessments of Major NASA Projects GAO-17-303SP SURFACE WATER AND OCEAN TOPOGRAPHY Cost and Schedule Status Other Issue to Be Monitored The SWOT project entered the implementation phase in May The project has experienced challenges with one of the 2016 and formally established its cost and schedule baselines. tools—AirSWOT—that it plans to use to help understand SWOT’s cost baseline is within the preliminary estimates the data returned from the KaRIN instrument once SWOT for the project, but its planned launch date is 6 months later is in orbit. AirSWOT is an airborne sensor that collects data than previously expected. The project added the extra time to on ocean and surface water phenomena. These data can account for potential delivery delays with the Radio Frequency be used to calibrate and validate the KaRIN instrument. Unit (RFU), a key component of SWOT’s main instrument, the However, AirSWOT’s ocean measurements to date showed Ka-Band Radar Interferometer (KaRIn), and to take actions features that were not fully understood and therefore to increase SWOT’s reliability. The project received NASA degraded its utility as a calibration and validation tool. The headquarters-held reserves, which is funding that may be used project has performed analysis to explain the phenomena, to address issues outside of a project’s control, to cover the updated its algorithm for ocean data processing, and costs of the delay and reliability efforts. identified alternate ocean calibration and validation methods that could be used if needed. Technology The SWOT project met a key best practice by maturing its four critical technologies to a technology readiness level 6 by its preliminary design review. Maturing critical technologies to this level helps minimize risks for space systems entering product development.

Development Partner The KaRIn instrument, which involves multiple international partners, is the most complicated development effort and biggest schedule driver for the project. Before the project established its schedule baseline, CNES, which is contributing the KaRIN RFU, requested and received an additional 4 months to complete risk reduction activities and allow more time for parts procurement. The project added an additional 2 months of schedule reserves to account for risks, such as the need to repeat testing. Project officials said that contractor performance has been an ongoing concern for the RFU, and CNES is closely managing the responsible contractor in close coordination with NASA. The project is also tracking a risk that schedule delays could occur due to the complexity of having multiple partners contributing components of KaRIn. GAO and the NASA Office of Inspector General have previously found that receiving and integrating contributions from multiple partners may complicate development efforts and contribute to cost and schedule growth. To mitigate the risk, the project plans to mature key interfaces early and conduct a full instrument-level integration and test phase for the KaRIn engineering model, among other actions.

Design The SWOT project is taking steps to increase the reliability of its spacecraft and instruments after the failure of the Soil Moisture Active Passive (SMAP) mission. The project has not identified any areas of commonality related to SMAP’s PROJECT OFFICE COMMENTS failure. However, it has taken actions to reduce its probability SWOT project officials provided technical comments of experiencing a single-point failure, which, if it occurs, can on a draft of this assessment, which were threaten the mission. For example, the project is making design incorporated as appropriate. improvements, using higher quality parts, and conducting additional testing to ensure similar problems do not occur. common name: SWOT 76 Assessments of Major NASA Projects GAO-17-303SP Transiting Exoplanet Survey Satellite

The Transiting Exoplanet Survey Satellite (TESS) will use four identical, wide field-of-view cameras to conduct the first extensive survey of the sky from space. The mission’s goal is to discover exoplanets—or planets in other solar systems—during transit, the time when the planet’s orbit carries it in front of its star as viewed from Earth. The project plans to discover rocky and potentially habitable Earth-sized and super-Earth planets orbiting nearby bright stars for further evaluation through ground- and space-based observations by other missions, such as the James Webb Space Telescope.

Source: NASA. | GAO-17-303SP

TION 04/13 02/14 09/14 10/14 08/15 01/17 05/17 03/18 06/18 Formulation System Preliminary Project Critical GAO System Manifested Committed start requirements/ design confirmation design review integration launch launch

FORMULA mission review review review readiness readiness

definition review IMPLEMEN TA TION date date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Goddard Space Flight The TESS project still plans to launch before its committed launch date and Center within its cost baseline, despite a series of launch vehicle and instrument-related International Partner: None delays. The project delayed its launch by 7 months from August 2017 to March 2018 because SpaceX required additional time to certify its upgraded launch Launch Location: Cape Canaveral Air vehicle and the project required additional time to complete the development Force Station, FL of its instrument. Prior to moving the launch date, the project faced schedule Launch Vehicle: Falcon 9 pressures due to development delays on its data handling unit (DHU), a primary component of the TESS instrument. To mitigate DHU delays, the project used Mission Duration: 2 years cost reserves to buy an alternative DHU with a different design from a different Requirement Derived from: 2010 Decadal supplier. The project plans to select a DHU design by May 2017, in time for the Survey start of system integration and test. The project’s primary concerns leading up Budget Portfolio: Science, Astrophysics to system integration are spacecraft-related delays involving its solar arrays and one of its transmitters, among other components.

PROJECT BUDGET INFORMATION COST PERFORMANCE SCHEDULE PERFORMANCE percent of current portfolio over next 5 years then-year dollars in millions 0 $378.4 -7.1% $351.7 MONTHS 0.6% CHANGE 06/18 06/18 LAUNCH LAUNCH DATE DATE

TESS

Other major projects

$28.1 $28.2 99.4% $323.2 $296.4 Operations total needed to be funded then-year dollars in millions Development $27.1 $27.1 04/13 04/13 Formulation $351.7 BASELINE LATEST ESTIMATE BASELINE LATEST ESTIMATE FY 2015 FEB. 2017 FY 2015 FEB. 2017 $137.5

common name: TESS 77 Assessments of Major NASA Projects GAO-17-303SP TRANSITING EXOPLANET SURVEY SATELLITE

Cost and Schedule Status The TESS project delayed its launch readiness date by 7 these or other spacecraft components, it could result in months from August 2017 to March 2018 due to launch changes or delays in the project’s system-level integration vehicle and instrument-related delays, but it still plans to and test schedule. launch before its committed launch date and within its cost baseline. The project plans to use NASA headquarters-held Prior to moving its launch date, the TESS project was also reserves, which is funding that may be used to address facing schedule pressures due to development delays on issues outside of a project’s control, to cover the cost of its data handling unit (DHU)—which powers the cameras the delay. The project has made progress on test and and serves as the instrument data storage and processing integration of its spacecraft, which began in March 2016, computer. The project has completed the development of but it has delayed its system integration review twice. its cameras, but the DHU is behind schedule because of a The project plans to hold this review in May 2017—7 series of technical issues related to its complex design and months later than originally planned—after it completes the contractor performance issues. To mitigate the DHU delays, development of its science instrument. the project used cost reserves to buy an alternate DHU from a different supplier. According to the project office, Launch both DHUs are capable of meeting the mission’s top-level According to NASA officials, several launch vehicle- science requirements. The project plans to make a decision related issues led to the delay in TESS’s planned launch on which DHU to use by May 2017 after it tests engineering date. First, SpaceX required additional time to certify its development units of both versions. The project plans to upgraded Falcon 9 through NASA’s Launch Services use this engineering model to start system-level integration Program since it will be the first time that NASA will use this and test and later integrate the flight unit, to help prevent version of the vehicle. The certification process includes further schedule delays. Project officials said they would criteria, such as having six successful launches. In addition, continue to develop the other DHU as a backup and would SpaceX needed time to investigate and resolve an anomaly later make it available for use by another mission. that caused a September 2016 launch mishap. NASA has renegotiated its launch contract with SpaceX to account for these delays. SpaceX continues to upgrade the Falcon 9 and, as part of the negotiation process, NASA gained the right not to be the first launch on the planned Block 5 version of the vehicle.

Technology and Design The TESS project’s primary concerns leading up to its system integration review are spacecraft-related delays. In August 2016, the project found faulty slip rings within the TESS solar array drive assembly, which delayed the assembly’s integration with the spacecraft by 6 months— the length of time it takes to build the rings—to February 2017. The project told us that the electrical circuits in PROJECT OFFICE COMMENTS the slip rings, which allow electrical energy to pass from the rotating solar arrays into the spacecraft body, were In commenting on a draft of this assessment, TESS not properly isolated, causing interruptions in electrical project officials said they are on track to meet their conductivity during testing. The project has been unable to March 2018 launch date, and are holding a significant determine a root cause, but identified cracks in insulation amount of schedule reserves. Officials said the materials and potential over-testing as contributing completion of certain spacecraft components are the causes. To help mitigate the problem, the project made current pacing items for the project, and the prime manufacturing process improvements for the new slip rings. contractor for the Ka-band transmitter is working to According to project officials, delays in the completion of ensure an on-time delivery. Officials also noted that the spacecraft’s Ka-band transmitter, which transmits the both DHU and alternate DHU vendors delivered their mission’s science data to the ground, is also a risk. The engineering development units and are in the process transmitter will be the last spacecraft component delivered for system-level integration and test. The project expects of completing their flight boxes. Project officials also delivery in June 2017. If there are additional problems with provided technical comments, which were incorporated as appropriate. common name: TESS 78 Assessments of Major NASA Projects GAO-17-303SP Wide-Field Infrared Survey Telescope

The Wide-Field Infrared Survey Telescope (WFIRST) is an observatory designed to perform wide-field imaging and survey of the near-infrared sky to answer questions about the structure and evolution of the universe, and expand our knowledge of planets beyond our solar system. The project will utilize a telescope that was originally built and qualified by another federal agency. The project plans to launch WFIRST in the mid-2020s to an orbit about 1 million miles from the Earth. The project is also planning a guest observer program where observation time may be made available to academic and other institutions.

Source: NASA. | GAO-17-303SP

TION 02/16 01/17 06/17 04/19 05/19 01/21 09/23 09/25 Formulation GAO System Preliminary Project Critical System Projected start review requirements/ design confirmation design integration launch

FORMULA mission review review review readiness

definition review IMPLEMEN TA TION date

PROJECT INFORMATION PROJECT SUMMARY NASA Lead Center: Goddard Space Flight In February 2016, NASA approved the WFIRST project to enter the concept and Center technology development phase and established preliminary cost and schedule International Partner: TBD estimates. The current funding profile supports a launch date in the mid to late 2020s. However, if funding is accelerated starting in fiscal year 2018, the project Launch Location: TBD estimates that it could launch in 2024 at a significantly lower cost.The project is Launch Vehicle: TBD evaluating how it plans to work with international partners as it prepares for its system requirements and mission definition review in June 2017. The WFIRST Mission Duration: 6 ¼ years project also continues to mature the detector technology needed for its Wide Requirement Derived from: 2010 Field Instrument and to mitigate risks related to detector performance and Astrophysics Decadal Survey production. Design trade-offs are also being made. NASA is studying whether to incorporate design features into WFIRST to make it compatible with a starshade, Budget Portfolio: Science, Astrophysics which improves performance by blocking out starlight. The project plans to make a final decision on whether WFIRST will include these features by October 2017.

PROJECT BUDGET INFORMATION PRELIMINARY COST a PRELIMINARY SCHEDULE percent of current portfolio over next 5 years then-year dollars in millions

5.9% $3,200 – $3,800 2024 – 2026 (including launch PROJECTED vehicle) LAUNCH READINESS DATE WFIRST

Other major projects

94.1%

total funded to date then-year dollars in millions 02/16 aThis estimate is preliminary, as the project is in formulation and there is uncertainty regarding the costs associated with the design options LATEST ESTIMATE $87 being explored. NASA uses these estimates for planning purposes. FEB. 2017

common name: WFIRST 79 Assessments of Major NASA Projects GAO-17-303SP WIDE-FIELD INFRARED SURVEY TELESCOPE

Cost and Schedule Status The WFIRST project entered the concept and technology project officials, these next generation detectors will development phase and established preliminary cost provide a higher level of performance, which is needed and schedule estimates in February 2016. The project’s to meet the project’s requirement for larger sky surveys. planned launch date is still being determined and is There is a risk that the chosen detectors will not meet all dependent on the funding it receives. The current funding the project’s performance specifications, and the detector profile supports a date in the mid to late 2020s. According yield will be low due to limited production experience with to the project, WFIRST received more funding than the configuration planned for the instrument. The project anticipated in fiscal years 2014 through 2016, which the is conducting additional testing and implementing new project used to mature technology and increase the fidelity manufacturing processes to mitigate these risks. The of its design. The project is evaluating the feasibility for Coronagraph, which is designed to perform high contrast an earlier launch in 2024 if it receives significantly more imaging and spectroscopy of nearby exoplanets, also funding starting in fiscal year 2018. According to project has new technologies, but their development poses less officials, the additional funding would allow the project to of a risk since the instrument is not needed to meet the optimize the development schedule and reduce the cost of mission’s requirements. the mission. Developmental Partner Design NASA is discussing possible contributions with various The WFIRST project has selected its instruments and international partners, including the European Space defined other elements of its design. The current design Agency, Canada, Australia, and Japan, for elements of the includes a 2.4 meter telescope and two instruments—the Wide Field Instrument, Coronagraph, and ground system. Wide Field Instrument and a technology demonstration These contributions could potentially reduce the project’s instrument, the Coronagraph. The telescope was built and cost. NASA expects to determine its international partners qualified for another federal agency over 10 years ago, before the project enters the preliminary design and so the project is performing an aging assessment to see technology completion phase, which is planned for how the long storage period affected it. The project is also October 2017. evaluating which components to reuse and which to modify, refurbish, or build new. NASA has also directed WFIRST be designed so that a robotic servicing vehicle can replace various modules while it is on-orbit.

The project is also evaluating other design considerations. For example, the project is studying what design features WFIRST would need to be compatible with current starshade design concepts. A starshade is a device that is launched with or separately from an observatory and positioned between it and the star being observed to block out the starlight while allowing the light emitted by the planet through. NASA expects to make a final decision on these design features by October 2017 after assessing the potential scientific benefits, risks, and resource needs.

Technology The WFIRST project is continuing to mature its one critical technology—the Wide Field Instrument detectors. The Wide Field Instrument is WFIRST’s primary instrument and is intended to measure light from a billion galaxies and perform a survey of the inner Milky Way. The project has assessed the detector to be at a technology readiness PROJECT OFFICE COMMENTS level 5, which means that the basic components have WFIRST project officials provided technical been integrated and tested in a simulated environment. comments to a draft of this assessment, which were The project plans to use detectors evolved from those incorporated as appropriate. used on James Webb Space Telescope. According to common name: WFIRST 80 Assessments of Major NASA Projects GAO-17-303SP

We are not making any recommendations in this report. We provided a Agency Comments draft of this report to NASA for comment. In its written comments, and Our Evaluation reproduced in appendix V, NASA generally agreed with our findings. NASA also provided technical comments that were incorporated, as appropriate.

We are sending copies of the report to the NASA Administrator and interested congressional committees. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4841 or [email protected]. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report at listed in appendix VI.

Cristina T. Chaplain Director Acquisition and Sourcing Management

Page 81 GAO-17-303SP Assessments of Major NASA Projects

List of Committees

The Honorable Richard Shelby Chairman The Honorable Jeanne Shaheen Ranking Member Subcommittee on Commerce, Justice, Science, and Related Agencies Committee on Appropriations United States Senate

The Honorable Ted Cruz Chairman The Honorable Edward J. Markey Ranking Member Subcommittee on Space, Science, and Competitiveness Committee on Commerce, Science, and Transportation United States Senate

The Honorable John Culberson Chairman The Honorable José Serrano Ranking Member Subcommittee on Commerce, Justice, Science, and Related Agencies Committee on Appropriations House of Representatives

The Honorable Brian Babin Chairman The Honorable Ami Bera Ranking Member Subcommittee on Space Committee on Science, Space, and Technology House of Representatives

Page 82 GAO-17-303SP Assessments of Major NASA Projects

Appendix I: Objectives, Scope, and Methodology Appendix I: Objectives, Scope, and

Methodology

The objectives of our review were to assess (1) the cost and schedule performance of the National Aeronautics and Space Administration’s (NASA) portfolio of major projects, (2) the maturity of technologies and stability of project designs at key points in the development process, and (3) NASA’s progress in implementing initiatives to manage acquisition risk and potential challenges for project management, execution, and oversight. We also described the status and assessed the risks and challenges faced by NASA’s 21 major projects, each with life-cycle costs more than $250 million. When NASA determines that a project has an estimated life-cycle cost of over $250 million, we include that project in our annual review up through launch or completion. There are 22 major projects, but we did not develop an assessment for the Origins-Spectral Interpretation-Resource Identification-Security–Regolith Explorer (OSIRIS-REx) project because it launched in September 2016.

To respond to these objectives, we developed several standard data collection instruments (DCI). We developed multiple DCIs, which were completed by NASA’s Office of the Chief Financial Officer, to gather data on each project’s cost and schedule. We used another DCI, which was completed by each project office, to gather data on project’s technology and design maturity and development partners. The information available on individual projects depends on where a project is in its life cycle. For example, for projects in an early stage of development called formulation there are still unknowns about requirements, technology, and design. We also analyzed DCI data from prior reviews.

To assess the cost and schedule performance of NASA’s major projects, we compared current cost and schedule data provided on DCIs by NASA for the 16 projects in the implementation phase during our review to previously established cost and schedule baselines.1 The Commercial Crew Program has a tailored project life cycle and project management requirements, so it was excluded from this analysis. In addition, we assessed development cost and schedule performance for NASA’s portfolios of major projects for 2009 to 2016 to examine longer-term trends. As part of this analysis, we calculated the average age of these portfolios, by determining the length of time a project spent in the development phase and averaging that across the portfolio. We then compared that historical trend to the portfolio’s cost and schedule

1For the purpose of this review, cost performance is defined as the percentage of total development cost growth over the development cost baseline.

Page 83 GAO-17-303SP Assessments of Major NASA Projects

Appendix I: Objectives, Scope, and Methodology

performance to determine if there was a relationship. We also analyzed the development costs for seven Science Mission Directorate projects— Gravity Recovery and Climate Experiment Follow-On; Ice, Cloud, and Land Elevation Satellite-2; Interior Exploration using Seismic Investigations, Geodesy, and Heat Transport; Ionospheric Connection Explorer; OSIRIS-REx; Solar Probe Plus; and Transiting Exoplanet Survey Satellite. We chose these projects because they reported their development costs using similar cost categories and at least 6 months had passed since their cost baseline was approved. We compared the projects’ baseline development costs and current development costs across these categories. To assess the accuracy of prime operation cost baselines, we used a DCI to collect data on 20 previously launched Science Mission Directorate missions. To select these projects, we considered projects that (1) were covered in our previous annual assessments of major projects and (2) have launched and are therefore either currently in prime operations or have completed prime operations. Three projects that meet these criteria were excluded from this analysis: (1) Landsat Data Continuity Mission, since the project is operated by the U.S. Geological Survey; (2) Stratospheric Observatory for Infrared Astronomy, since it is an aircraft-installed system, rather than a launched mission; and (3) Tracking and Data Relay Satellite Replenishment, since there is no project-specific operations budget for this project. For the projects that are still in the prime operation phase, our analysis used NASA’s current estimates of each project’s operation costs at completion, which include both the project’s actual operation costs to date and the project’s estimated costs to prime mission end. Our analysis compared each project’s completed prime mission costs or currently estimated prime operations costs against the project’s prime operation cost baseline established at the project confirmation review. All cost information in this report is presented in nominal then-year dollars for consistency with budget data. Current baseline costs for all projects are adjusted to reflect the cost accounting structure in NASA’s fiscal year 2009 budget estimates. For the fiscal year 2009 budget request, NASA changed its accounting practices from full-cost accounting to reporting only direct costs at the project level.

To assess technology maturity, we asked project officials to complete a DCI that provided the technology readiness levels of each of the project’s critical and heritage technologies at various stages of project development including the preliminary design review. For the 15 projects that had held a preliminary design review and identified critical or heritage technologies, we compared those levels against our technology maturity best practice and NASA policy on technology maturity to determine the

Page 84 GAO-17-303SP Assessments of Major NASA Projects

Appendix I: Objectives, Scope, and Methodology

extent to which the portfolio was meeting the criteria. Our work has shown that reaching a technology readiness level 6—which indicates that the representative prototype of the technology has been demonstrated in a relevant environment that simulates the harsh conditions of space—by the preliminary design review is the level of maturity needed to minimize risks for space systems entering product development. Originally developed by NASA, technology readiness levels are measured on a scale of one to nine, beginning with paper studies of a technology’s feasibility and culminating with a technology fully integrated into a completed product. See appendix III for the definitions of technology readiness levels. We compared this year’s results against those in prior years to assess whether NASA was improving in this area. We did not assess technology maturity for those projects that had not yet reached the preliminary design review at the time of this assessment, or for projects that reported no critical or heritage technologies. We also excluded 2009 from our analysis since the data was only for critical technologies and did not include heritage technologies. We compared the number of critical technologies being developed per project with those in prior years to determine how the number of critical technologies developed per project had changed. We did not assess the average number of critical technologies being developed per project for projects that had not entered implementation at the time of this assessment. We also collected information on the use of heritage technologies in the projects; including what heritage technologies were being used; what effort was needed to modify the form, fit, and function of the technology for use in the new system; and whether the project considered the heritage technology as a risk to the project.

To assess design stability, we asked project officials to complete a DCI that provided the number of engineering drawings completed or projected for release by the preliminary and critical design reviews and as of our current assessment.2 We did not verify or validate project office supplied data on the number of released and expected engineering drawings. However, we collected the project offices’ rationale for cases where it appeared that only a small percentage of the expected drawings were

2In our calculation for the percentage of total number of drawings projected for release, we used the number of drawings released at the critical design review as a fraction of the total number of drawings projected, including where a growth in drawings occurred. Therefore, the denominator in the calculation may have been larger than what was projected at the critical design review. We believe that this more accurately reflected the design stability of the project.

Page 85 GAO-17-303SP Assessments of Major NASA Projects

Appendix I: Objectives, Scope, and Methodology

completed by the time of the design reviews or where the project office reported significant growth in the number of drawings released after the critical design review. In accordance with best practices, projects were assessed as having achieved design stability if at least 90 percent of projected drawings were released by the critical design review. We compared this year’s results against those in prior years to assess whether NASA was improving in this area. For this year’s assessment, 11 projects had held a critical design review and reported data on design drawings. We did not assess the design stability for those projects that had not yet reached the critical design review at the time of this assessment. To assess project technical margins, we gathered project mass and power information using a DCI and compared it against NASA requirements. We omitted the Exploration Ground Systems, Space Network Ground Segment Sustainment and Space Launch System as those projects do not contain spacecraft. We excluded the Orion program because it does not have applicable metrics. To assess completion of project validation and verification plans, we asked project officials to complete a DCI that provided data on whether a plan was completed by the critical design review. To assess how adherence to technology maturity and design stability best practices affected the outcomes for projects that we have reviewed since 2009, we used data collection instruments submitted by NASA from 2009 onward for each project that rebaselined or launched within its cost and schedule baselines to determine whether the project matured its technologies to a technology readiness level 6 at preliminary design review and released at least 90 percent of design drawings at critical design review. We also assessed each project’s number of critical technologies and calculated the percentage of drawing growth after critical design review. We used these data to compare the performance of projects that launched under cost and within schedule to projects that had rebaselined against these metrics.3 We also reviewed past GAO and NASA Inspector General reports to understand whether projects that rebaselined did so in part because of immature technologies or design.

3NASA is required to report to certain committees in the House and Senate if the development cost of a program is likely to exceed the baseline estimate by 15 percent or more, or if a milestone is likely to be delayed by 6 months or more. 51 U.S.C. § 30104(e). Further, if the development cost of a program will exceed the baseline estimate by more than 30 percent, NASA is required to prepare a new baseline if the program is to be continued. 51 U.S.C. § 30104(e),(f). NASA typically refers to the programs covered by this requirement as projects.

Page 86 GAO-17-303SP Assessments of Major NASA Projects

Appendix I: Objectives, Scope, and Methodology

To assess NASA’s progress in reducing acquisition management risk, we analyzed ongoing NASA initiatives in key areas, such as cost and schedule estimation and earned value management. We also followed up on other potential acquisition management challenges that we identified during our review, such as emerging workforce issues and project funding and budget phasing. To assess NASA’s efforts to improve cost and schedule estimation, we used a DCI to collect data on joint cost and schedule confidence level updates; reviewed documentation, such as briefings on new and updated cost and schedule tools; and interviewed officials. To assess the status of the independent assessment transition, we reviewed official documentation, such as the agency’s independent assessment white paper; interviewed officials within the Office of the Chief Financial Officer and the Office of the Chief Engineer; and evaluated the extent to which the agency’s goals were met for the reorganized independent assessment function. To assess potential workforce issues, we reviewed documentation provided regarding the transition to the new independent assessment model and information provided at the NASA Cost Symposium, and interviewed officials in the Office of the Chief Financial Officer. To assess the extent to which funding phasing issues could present problems in a constrained environment, we reviewed project documentation, interviewed project officials, and analyzed NASA’s 5-year budget in the fiscal year 2017 budget request.

Our work was performed primarily at NASA headquarters in Washington, D.C. In addition, we and related GAO engagement teams visited Goddard Space Flight Center in Greenbelt, Maryland; the Jet Propulsion Laboratory in Pasadena, California; Kennedy Space Center in Cape Canaveral, Florida; Johnson Space Center in Houston, Texas; and Marshall Space Flight Center in Huntsville, Alabama.

Project Profile Information This year, we developed project assessments for the 21 projects in the on Each Individual Project portfolio with an estimated life-cycle cost greater than $250 million. For Assessment each project assessment we included a description of each project’s objectives, information concerning the NASA center, and international partners involved in the project, if applicable, the project’s cost and schedule performance, a schedule timeline identifying key project dates, and a brief narrative describing the current status of the project. We also included budget information, including the percentage of NASA’s notional 5-year budget for all major projects in the current portfolio that the project represents. The budget information is based on NASA’s fiscal year 2017 budget. The budget covers fiscal years 2017 to 2021; however, NASA describes funding numbers beyond fiscal year 2017 as notional. For

Page 87 GAO-17-303SP Assessments of Major NASA Projects

Appendix I: Objectives, Scope, and Methodology

projects in formulation, we included the total funding that has been allocated to the project since formulation start through the end of fiscal year 2016. For projects in implementation, we included the funding needed to be allocated for project completion or launch through the project’s current life-cycle cost estimate. We also provided a detailed discussion of project challenges for selected projects as applicable.

To assess the cost and schedule changes of each project, we obtained data directly from NASA’s Office of the Chief Financial Officer through our DCI. For the Commercial Crew program, we obtained data directly from the program on the total amount of funds obligated and schedule. When applicable, we compared the level of cost and schedule reserves held by the project to the level required by center policy. To determine the funding received to date for each of the projects in formulation we calculated the total funding allocated to the project since formulation start.4 For projects in implementation, we calculated the funding needed for project launch or completion through its current life-cycle cost estimate.

The project’s timeline is based on acquisition cycle time, which is defined as the number of months between the project’s start, or formulation start, and the projected or actual launch date. Formulation start generally refers to the initiation of a project; NASA refers to a project’s start as key decision point (KDP)-A, or the beginning of the formulation phase. The preliminary design review typically occurs toward the end of the formulation phase, followed by a review at KDP-C, known as project confirmation, which allows the project to move into the implementation phase. The critical design review is generally held during the latter half of the final design and fabrication phase of implementation and demonstrates that the maturity of the design is appropriate to support continuing with the final design and fabrication phase. The manifested launch date is the launch date which the project is working toward, and when a launch vehicle is available to launch the project. This date is only a goal launch date for the project, not a commitment that they will launch on this date. The committed launch readiness date is determined through a launch readiness review that verifies that the launch system, spacecraft, and payloads are ready for launch. The implementation phase includes the operations of the mission and concludes with project disposal.

4This does not include funds for studies prior to key decision point (KDP)-A, the start of formulation.

Page 88 GAO-17-303SP Assessments of Major NASA Projects

Appendix I: Objectives, Scope, and Methodology

Project Challenges To assess the status, risk, and challenges for each project, we submitted Discussion on Each a DCI to each project office. In the DCI, we requested information on the Individual Project maturity of critical and heritage technologies, the number of releasable design drawings at project milestones, and international partnerships.5 Assessment We also held interviews with representatives from all of the projects to discuss the information on the DCI. We then reviewed project documentation—including project plans, schedules, risk assessments, and major project review documentation—to corroborate any testimonial evidence we received in the interviews. These reviews led to identification of further challenges faced by NASA projects. The second page of our project assessments highlights key challenges facing that project that have or could affect project performance. For this year’s report, we identified challenges across the projects we reviewed in the categories of launch, contractor, development partner, funding, design, technology, schedule, and integration and test. These challenges do not represent an exhaustive or exclusive list and are based on our definitions and assessments, not those of NASA.

To supplement our analysis, we relied on our work over past years examining acquisition issues across multiple agencies. These reports cover such issues as contracting, program management, acquisition best practices, and cost estimating. We also have an extensive body of work related to challenges NASA has faced with specific system acquisitions, financial management, and cost estimating. This work provided the historical context and basis for large parts of the general observations we made about the projects we reviewed.

Data Limitations NASA provided preliminary estimated life-cycle cost ranges and associated schedules for the five projects that had not yet entered implementation, which are generally established at KDP-B. NASA formally establishes cost and schedule baselines, committing itself to cost and schedule targets for a project with a specific and aligned set of planned mission objectives, at KDP-C, which follows a preliminary design review. KDP-C reflects the life-cycle point where NASA approves a project to leave the formulation phase and enter into the implementation phase. NASA explained that preliminary estimates are generated for internal planning and fiscal year budgeting purposes at KDP-B, which occurs midstream in the formulation phase, and hence, are not

5We did not collect this information for the Commercial Crew Program.

Page 89 GAO-17-303SP Assessments of Major NASA Projects

Appendix I: Objectives, Scope, and Methodology

considered a formal commitment by the agency on cost and schedule for the mission deliverables. Due to changes that occur to a project’s scope and technologies between KDP-B and KDP-C, the estimates of project cost and schedule can be significantly altered between the two KDPs.

We conducted this performance audit from April 2016 to May 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Page 90 GAO-17-303SP Assessments of Major NASA Projects

Appendix II: Major NASA Projects Reviewed in GAO’s Annual Assessments Appendix II: Major NASA Projects Reviewed

in GAO’s Annual Assessments

We have reviewed 49 major National Aeronautics and Space Administration (NASA) projects or programs since our initial review in 2009. See figure 12 below for a list of projects included in our assessments from 2009 to 2017.

Page 91 GAO-17-303SP Assessments of Major NASA Projects

Appendix II: Major NASA Projects Reviewed in GAO’s Annual Assessments

Figure 12: Major NASA Projects Reviewed in GAO’s Annual Assessments

Page 92 GAO-17-303SP Assessments of Major NASA Projects

Appendix II: Major NASA Projects Reviewed in GAO’s Annual Assessments

aIn 2014, NASA adopted Orion as the common name for Orion MPCV; the project did not change. This Orion project stems from the original Orion project that was canceled in June 2011 when the Constellation program was canceled after facing significant technical and funding issues. During the closeout process for the Constellation program, NASA identified elements of the Ares I and Orion projects that would be transitioned for use on the new Space Launch System and Orion Multi- Purpose Crew Vehicle programs. bA bid protest was filed on September 26, 2014, after NASA awarded Commercial Crew contracts. GAO issued a decision on the bid protest on January 5, 2015, which was after our review of projects had concluded; therefore, we excluded the Commercial Crew Program from the 2015 review.

Page 93 GAO-17-303SP Assessments of Major NASA Projects

Appendix III: Technology Readiness Levels

Appendix III: Technology Readiness Levels

Table 7: Characteristics of Technology Readiness Levels

Technology readiness level Description Hardware Demonstration environment 1. Basic principles observed Lowest level of technology readiness. None (paper studies and None. and reported. Scientific research begins to be translated analysis). into applied research and development. Examples might include paper studies of a technology’s basic properties.

2. Technology concept Invention begins. Once basic principles are None (paper studies and None. and/or application observed, practical applications can be analysis). formulated. invented. The application is speculative and there is no proof or detailed analysis to support the assumption. Examples are still limited to paper studies. 3. Analytical and Active research and development is initiated. Analytic studies and Lab. experimental critical This includes analytical studies and demonstration of nonscale function and/or laboratory studies to physically validate individual components (pieces characteristic proof of analytical predictions of separate elements of of subsystem). concept. the technology. Examples include components that are not yet integrated or representative. 4. Component and/or Basic technological components are Low fidelity breadboard. Lab. breadboard integrated to establish that the pieces will Validation in laboratory work together. This is relatively “low fidelity” Integration of nonscale environment. compared to the eventual system. Examples components to show pieces will include integration of ad-hoc hardware in a work together. Not fully laboratory. functional or form or fit but representative of technically feasible approach suitable for flight articles. 5. Component and/or Fidelity of breadboard technology increases High-fidelity breadboard. Lab demonstrating breadboard validation in significantly. The basic technological functionality but not relevant environment. components are integrated with reasonably Functionally equivalent but not form and fit. May realistic supporting elements so that the necessarily form and/or fit include flight technology can be tested in a simulated (size, weight, materials, etc.). demonstrating environment. Examples include high-fidelity Should be approaching breadboard in laboratory integration of components. appropriate scale. May include surrogate aircraft. integration of several Technology ready components with reasonably for detailed design realistic support studies. elements/subsystems to demonstrate functionality.

Page 94 GAO-17-303SP Assessments of Major NASA Projects

Appendix III: Technology Readiness Levels

Technology readiness level Description Hardware Demonstration environment 6. System/subsystem model Representative model or prototype system, Prototype. Should be very High-fidelity lab or prototype demonstration which is well beyond the breadboard tested close to form, fit, and function. demonstration or in a relevant environment. for technology readiness level 5, is tested in a Probably includes the limited/restricted relevant environment. Represents a major integration of many new flight demonstration step up in a technology’s demonstrated components and realistic for a relevant readiness. Examples include testing a supporting environment. prototype in a high-fidelity laboratory elements/subsystems if needed Integration of environment or in simulated realistic to demonstrate full functionality technology is well environment. of the subsystem. defined. 7. System prototype Prototype near or at planned operational Prototype. Should be form, fit, Flight demonstration demonstration in a realistic system. Represents a major step up from and function integrated with in representative environment. technology readiness level 6, requiring the other key supporting realistic demonstration of an actual system prototype elements/subsystems to environment such in a realistic environment, such as in an demonstrate full functionality of as flying test bed or aircraft, vehicle, or space. Examples include subsystem. demonstrator testing the prototype in a test bed aircraft. aircraft. Technology is well substantiated with test data. 8. Actual system completed Technology has been proven to work in its Flight qualified hardware. Developmental Test and “flight qualified” final form and under expected conditions. In and Evaluation in through test and almost all cases, this technology readiness the actual system demonstration. level represents the end of true system application. development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications. 9. Actual system “flight - Actual application of the technology in its final Actual system in final form. Technology proven” through successful form and under mission conditions, such as assessed as fully mission operations. those encountered in operational test and mature. evaluation. In almost all cases, this is the end of the last “bug fixing” aspects of true system development. Examples include using the system under operational mission conditions.

Source: GAO Analysis of National Aeronautics and Space Administration data. | GAO-17-303SP

Page 95 GAO-17-303SP Assessments of Major NASA Projects

Appendix IV: Elements of a Sound Business Case Appendix IV: Elements of a Sound Business

Case

The development and execution of a knowledge-based business case for the National Aeronautics and Space Administration’s (NASA) projects can provide early recognition of challenges, allow managers to take corrective action, and place needed and justifiable projects in a better position to succeed. Our prior work of best practice organizations shows the risks inherent in NASA’s work can be mitigated by developing a solid, executable business case before committing resources to a new product’s development.1 In its simplest form, a knowledge-based business case is evidence that (1) the customer’s needs are valid and can best be met with the chosen concept and that (2) the chosen concept can be developed and produced within existing resources—that is, proven technologies, design knowledge, adequate funding, adequate time, and adequate workforce to deliver the product when needed. A program should not be approved to go forward into product development unless a sound business case can be made. If the business case measures up, the organization commits to the development of the product, including making the financial investment. The building of knowledge consists of information that should be gathered at these three critical points over the course of a program:

• When a project begins development, the customer’s needs should match the developer’s available resources—mature technologies, time, and funding. An indication of this match is the demonstrated maturity of the technologies required to meet customer needs— referred to as critical technologies. If the project is relying on heritage—or pre-existing—technology, that technology must be in the appropriate form, fit, and function to address the customer’s needs within available resources. The project will generally enter development after completing the preliminary design review, at which time a business case should be in hand. • Then, about midway through the project’s development, its design should be stable and demonstrate it is capable of meeting performance requirements. The critical design review takes place at that point in time because it generally signifies when the program is

1GAO, Defense Acquisitions: Key Decisions to be Made on Future Combat System, GAO-07-376 (Washington, D.C.: Mar. 15, 2007); Defense Acquisitions: Improved Business Case Key for Future Combat System’s Success, GAO-06-564T (Washington, D.C.: Apr. 4, 2006); NASA: Implementing a Knowledge-Based Acquisition Framework Could Lead to Better Investment Decisions and Project Outcomes, GAO-06-218 (Washington, D.C.: Dec. 21, 2005); and NASA’s Space Vision: Business Case for Prometheus 1 Needed to Ensure Requirements Match Available Resources, GAO-05-242 (Washington, D.C.: Feb. 28, 2005).

Page 96 GAO-17-303SP Assessments of Major NASA Projects

Appendix IV: Elements of a Sound Business Case

ready to start building production-representative prototypes. If project development continues without design stability, costly redesigns to address changes to project requirements and unforeseen challenges can occur. • Finally, by the time of the production decision, the product must be shown to be producible within cost, schedule, and quality targets and have demonstrated its reliability, and the design must demonstrate that it performs as needed through realistic system-level testing. Lack of testing increases the possibility that project managers will not have information that could help avoid costly system failures in late stages of development or during system operations.

Page 97 GAO-17-303SP Assessments of Major NASA Projects

Appendix V: Comments from the National Aeronautics and Space Administration Appendix V: Comments from the National

Aeronautics and Space Administration

Page 98 GAO-17-303SP Assessments of Major NASA Projects

Appendix V: Comments from the National Aeronautics and Space Administration

Page 99 GAO-17-303SP Assessments of Major NASA Projects

Appendix VI: GAO Contact and Staff Acknowledgments Appendix VI: GAO Contact and Staff

Acknowledgments

GAO Contact Cristina Chaplain, (202) 512-4841 or [email protected].

In addition to the contact named above, Ron Schwenn, Assistant Director; Staff Andrea Bivens; Tana M. Davis; Juli Digate; Lorraine Ettaro; Lisa L. Acknowledgments Fisher; Laura Greifner; Kristine R. Hassinger; Erin Kennedy; Daniel Kuhn; Katherine Lenane; Jose A. Ramos; Carrie Rogers; Christal Ann Simanski; Daniel Singleton; Ryan Stott; Roxanna T. Sun; Kristin Van Wychen; and John S. Warren, Jr. made significant contributions to this report.

(100790) Page 100 GAO-17-303SP Assessments of Major NASA Projects

The Government Accountability Office, the audit, evaluation, and investigative GAO’s Mission arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is Obtaining Copies of through GAO’s website (http://www.gao.gov). Each weekday afternoon, GAO GAO Reports and posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to http://www.gao.gov Testimony and select “E-mail Updates.”

Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, http://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, LinkedIn, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at www.gao.gov and read The Watchblog.

To Report Fraud, Contact: Website: http://www.gao.gov/fraudnet/fraudnet.htm Waste, and Abuse in E-mail: [email protected] Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470 Katherine Siggerud, Managing Director, [email protected], (202) 512-4400, Congressional U.S. Government Accountability Office, 441 G Street NW, Room 7125, Relations Washington, DC 20548 Chuck Young, Managing Director, [email protected], (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548

James-Christian Blockwood, Managing Director, [email protected], (202) 512-4707 Strategic Planning and U.S. Government Accountability Office, 441 G Street NW, Room 7814, External Liaison Washington, DC 20548

Please Print on Recycled Paper.