United States Government Accountability Office Report to Congressional Committees GAO March 2012 NASA Assessments of Selected Large-Scale Projects GAO-12-207SP March 2012 NASA Assessments of Selected Large-Scale Projects Highlights of GAO-12-207SP, a report to congressional committees Why GAO Did This Study What GAO Found This is GAO’s fourth annual GAO assessed 21 NASA projects with a combined life-cycle cost that exceeds assessment of the National $43 billion. Of those 21 projects, 6 were in an early phase of development called Aeronautics and Space formulation, and 15 had entered the implementation phase where cost and Administration’s (NASA) large-scale schedule baselines were established. Five of the 15 projects in implementation projects. GAO’s work has shown that successfully launched in 2011, and two of them met their cost and schedule these projects, while producing baselines. NASA’s largest science project—the James Webb Space Telescope ground-breaking research and (JWST)—however, has experienced development cost growth of $3.6 billion—or advancing our understanding of the 140 percent—and a schedule delay of over 4 years. While the development cost universe, tend to cost more and take and schedule for most of the projects in implementation remained relatively longer to develop than planned, and stable, the impact of the JWST increases on the portfolio is significant. For are often approved without evidence of a sound business case. GAO has example, 14 of the 15 projects currently in implementation, excluding JWST, had designated NASA’s acquisition an average development cost growth of $79 million—or 14.6 percent—and management a high risk area. schedule growth of 8 months from their baselines. With JWST, these numbers increase dramatically to almost 47 percent and 11 months, respectively. Cost In response to congressional direction, and schedule increases within NASA’s most technologically advanced and costly GAO reviewed NASA's major projects. projects, such as JWST, can have cascading effects on the rest of NASA’s Specifically, this report provides portfolio. For example, the administration has proposed to terminate funding for observations about the performance of the joint NASA/ESA EMTGO project, and another large project in our review— NASA's major projects, assesses the Mars Science Laboratory (MSL)—experienced substantial cost overruns that knowledge attained at key junctures of led NASA to take funding from other projects. MSL and JWST account for development, identifies challenges that approximately $11.4 billion—or 51 percent—of the total life-cycle costs for the 15 can contribute to cost and schedule growth, and outlines steps NASA is projects in implementation during our review. taking to improve its acquisitions. To Most of the projects that GAO reviewed did not meet technology maturity and conduct this review, GAO assessed design stability best practices criteria, which if followed can lessen cost and data on 21 projects with an estimated schedule risks faced by the project. Specifically, 10 of the 16 projects that held a life-cycle cost of over $250 million, preliminary design review moved forward without first maturing technologies. In including data on projects’ cost, addition, 13 of the 14 projects that held a critical design review did so without first schedule, technology maturity, design achieving design stability. Some projects reported using other methods to assess stability, and contracts; analyzed design stability. Many of the projects GAO reviewed for this report also monthly project status reports; and interviewed NASA and contractor experienced challenges in the areas of launch vehicles, contractor management, officials. GAO also reviewed project parts, development partner performance, and funding. For example, nine projects cost estimates and interviewed officials we reviewed reported challenges with launch vehicles, including their increasing responsible for NASA’s cost estimation cost and availability. New launch vehicles are in development, but have not yet policy. been certified, and another vehicle has failed on its two most recent flights. What GAO Recommends The agency is continuing its implementation of initiatives to reduce acquisition management risk. One prominent effort is the Joint Cost and Schedule GAO is not making any new Confidence Level (JCL), a new cost estimation tool that involves a probabilistic recommendations in this report, but analysis of cost, schedule, and risk inputs to arrive at development cost and has made prior recommendations to schedule estimates associated with various confidence levels. Five projects GAO address transparency in project costs reviewed have completed a JCL. NASA officials stated a few projects have and the lack of consistent design excluded or not fully considered relevant cost inputs and risks, such as launch metrics; NASA concurred and is taking vehicle costs. GAO was unable to confirm that the five projects that prepared steps to address them. estimates using the JCL were budgeted at the approved confidence level. NASA has not yet launched a project that prepared a JCL; therefore, NASA officials View GAO-12-207SP. For more information, contact Cristina Chaplain at (202) 512-4841 or stated it will take several years to evaluate the impact and effectiveness of the [email protected]. JCL in improving cost and schedule estimating for its major projects. United States Government Accountability Office Contents Letter 1 Background 4 Observations on NASA’s Portfolio of Major Projects 13 Observations from Our Assessment of Knowledge Attained by Key Junctures in the Acquisition Process 18 Observations on Other Challenges That Can Affect Project Outcomes 27 Observations about NASA’s Continued Efforts to Improve Its Acquisition Management 35 Project Assessments 38 Aquarius 40 ExoMars Trace Gas Orbiter (EMTGO) 42 Global Precipitation Measurement (GPM) Mission 44 Gravity Recovery and Interior Laboratory (GRAIL) 46 Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2) 48 James Webb Space Telescope (JWST) 50 Juno 52 Landsat Data Continuity Mission (LDCM) 54 Lunar Atmosphere and Dust Environment Explorer (LADEE) 56 Magnetospheric Multiscale (MMS) 58 Mars Atmosphere and Volatile EvolutioN (MAVEN) 60 Mars Science Laboratory (MSL) 62 NPOESS Preparatory Project (NPP) 64 Orbiting Carbon Observatory 2 (OCO-2) 66 Orion Multi-Purpose Crew Vehicle (MPCV) 68 Radiation Belt Storm Probes (RBSP) 70 Soil Moisture Active and Passive (SMAP) 72 Solar Probe Plus (SPP) 74 Space Launch System (SLS) 76 Stratospheric Observatory For Infrared Astronomy (SOFIA) 78 Tracking and Data Relay Satellite (TDRS) Replenishment 80 Agency Comments and Our Evaluation 82 Appendix I Comments from the National Aeronautics and Space Administration 86 Appendix II Objectives, Scope, and Methodology 89 Page i GAO-12-207SP Assessments of Selected Large-Scale Projects Appendix III Selected Major NASA Projects Reviewed in GAO’s Annual Assessments 97 Appendix IV Technology Readiness Levels 98 Appendix V Significant Accomplishments of Projects That Have Launched 100 Appendix VI GAO Contact and Staff Acknowledgments 105 Tables Table 1: 21 Selected Major NASA Projects Reviewed in GAO’s 2012 Annual Assessment 9 Table 2: Development Cost and Schedule Growth of Selected Major NASA Projects Currently in the Implementation Phase 15 Table 3: Selected Major NASA Projects Reviewed in GAO’s Annual Assessments 97 Figures Figure 1: NASA’s Life Cycle for Flight Systems 7 Figure 2: Launch Date and Life-Cycle Cost for 15 Major NASA Projects in Implementation 11 Figure 3: JWST’s Cumulative Development Cost Growth Compared to the Total Life-Cycle Cost Estimates for Seven Selected Major NASA Projects 14 Figure 4: Percentage of Selected Major NASA Projects Meeting and Not Meeting Technology Maturity Criteria at Preliminary Design Review 20 Figure 5: Average Number of Critical Technologies Reported for Selected Major NASA Projects Reviewed from 2009 to 2012 22 Figure 6: Percentage of Engineering Drawings Releasable at CDR for Selected Major NASA Projects 24 Page ii GAO-12-207SP Assessments of Selected Large-Scale Projects Figure 7: Comparison of Design Drawings Increase for 20 Major NASA Projects That Have Held CDR prior to and since Fiscal Year 2009 26 Figure 8: Illustration of a Project’s 2-Page Summary 39 Abbreviations AFB Air Force Base AFS Air Force Station ASI Argenzia Spaciale Italiana (Italian Space Agency) ATLAS Advanced Topographic Laser Altimeter System CDDS Cavity Door Drive System CDR critical design review CONAE Comision Nacional de Actividades Espaciales (Space Agency of Argentina) CSA Canadian Space Agency DCI data collection instrument DPR dual-frequency precipitation radar DSN Deep Space Network DWSS Defense Weather Satellite System EDL entry, descent, and landing EEE electrical, electronic, and electromechanical EMTGO ExoMars Trace Gas Orbiter ESA European Space Agency EVM earned value management FAA Federal Aviation Administration FORCAST Faint Object Infrared Camera for the SOFIA Telescope FPI Fast Plasma Investigation GLAST Gamma-ray Large Area Space Telescope GMI GPM microwave imager GPM Global Precipitation Measurement (mission) GPS Global Positioning System GRAIL Gravity Recovery and Interior Laboratory GREAT German Receiver for Astronomy at Terahertz Frequencies HEPS High Efficiency Power Supply HOPE Helium-Oxygen-Proton-Electron ICESat-2 Ice, Cloud, and Land Elevation Satellite-2 IPO Integrated Program Office ISIM Integrated Science Instrument Module ISS International Space Station