GAO-07-1134SP Cost Assessment Guide: Best
Total Page:16
File Type:pdf, Size:1020Kb
United States Government Accountability Office GAO Applied Research and Methods July 2007 COST ASSESSMENT GUIDE Best Practices for Estimating and Managing Program Costs Exposure Draft GAO-07-1134SP Exposure Draft PREFACE The U.S. Government Accountability Office is responsible for, among other things, assisting the Congress in its oversight of the federal government, including agencies’ stewardship of public funds. To effectively use public funds, the government must meet the demands of today’s changing world by employing effective management practices and processes, including the measurement of government program performance. Legislators, government officials, and the public want to know whether government programs are achieving their goals and what their costs are. We developed the Cost Guide in order to establish a consistent methodology, based on best practices, to be used across the federal government for developing and managing its program cost estimates. For the purposes of this guide, a cost estimate is the summation of individual cost elements, using established methods and valid data to estimate the future costs of a program, based on what is known today. The management of a cost estimate involves continually updating the estimate with actual data as they become available, revising the estimate to reflect changes, and analyzing differences between estimated and actual costs—for example, using data from a reliable earned value management (EVM) system.1 The ability to generate reliable cost estimates is a critical function, necessary to support the Office of Management and Budget’s (OMB) capital programming process.2 Without this ability, agencies are at risk of experiencing cost overruns, missed deadlines, and performance shortfalls—all recurring problems that our program assessments too often reveal. Furthermore, cost increases often mean that the government cannot fund as many programs as intended or deliver them when promised. The methodology outlined in this guide is a compilation of best practices that federal cost estimating organizations and industry use to develop and maintain reliable cost estimates throughout the life of a government program. By default, the guide will also serve as a guiding principle for our auditors to evaluate the economy, efficiency, and effectiveness of government programs. The U.S. Government Accountability Office (GAO), the Congressional Budget Office (CBO), and others have shown through budget simulations that the nation is facing a large and growing structural deficit in the long term, primarily because the population is aging and healthcare costs are rising. As the Comptroller General has noted, “Continuing on this unsustainable path will gradually erode, if not suddenly damage, our economy, 1EVM is a project management tool that integrates the technical scope of work with schedule and cost elements for investment planning and control. As a method, it compares the value of work accomplished in a given period with the value of the work expected in that period. Differences in expectations are measured in both cost and schedule variances. The Office of Management and Budget (OMB) requires agencies to use EVM in their performance-based management systems for the parts of an investment in which development effort is required or system improvements are under way. 2Office of Management and Budget, Circular No. A-11, Preparation, Submission, and Execution of the Budget (Washington, D.C.: Executive Office of the President, June 2006); Circular No. A-130 Revised, Management of Federal Information Resources (Washington, D.C.: Executive Office of the President, Nov. 28, 2000); and Capital Programming Guide: Supplement to Circular A-11, Part 7, Preparation, Submission, and Execution of the Budget (Washington, D.C.: Executive Office of the President, June 2006). http://www.whitehouse.gov/omb/circulars/index.html. Page i GAO-07-1134SP Cost Assessment Guide Exposure Draft our standard of living and ultimately our national security.”3 New budgetary demands and demographic trends will place serious budgetary pressures on federal discretionary spending, as well as on other federal policies and programs, in the coming years. As resources become scarce, competition for them will increase. It is imperative, therefore, that government acquisition programs deliver as promised, not only because of their value to their users but because every dollar spent on one program will mean one less available dollar to fund other efforts. To get better results, programs will need higher levels of knowledge when they start and standardized monitoring metrics such as EVM so that better estimates can be made of total program costs at completion. 3GAO, 21st Century Challenges: Reexamining the Base of the Federal Government, GAO-05-325SP (Washington, D.C.: February 2005), p. 1. Page ii GAO-07-1134SP Cost Assessment Guide Exposure Draft Contents Preface i Introduction 1 Chapters 1 Characteristics of Credible Cost Estimates and a Reliable Process for Creating Them 5 2 Why Cost Estimates Are Required for Government Programs and Challenges in Developing Results 14 3 Criteria for Cost Estimating, EVM, and Data Reliability 22 4 Cost Analysis Overview 27 5 The Cost Estimate’s Purpose, Scope, and Schedule 43 6 The Cost Assessment Team 46 7 Technical Baseline Description 53 8 Work Breakdown Structure 61 9 Ground Rules and Assumptions 73 10 Data 82 11 Developing a Point Estimate 100 12 Estimating Software Costs 119 13 Sensitivity Analysis 131 14 Cost Risk and Uncertainty 137 15 Validating the Estimate 159 16 Documenting the Estimate 170 17 Presenting the Estimate to Management 175 18 Managing Program Costs: Planning 178 Page iii GAO-07-1134SP Cost Assessment Guide Exposure Draft 19 Managing Program Costs: Execution 209 20 Managing Program Costs: Updating 240 Appendixes I Auditing Agencies and Their Web Sites 264 II Case Study Backgrounds 265 III Experts Who Helped Develop This Guide 273 IV The Federal Budget Process 275 V Federal Cost Estimating and EVM Legislation, Regulations, Policies, and Guidance 277 VI Data Collection Instrument 287 VII Data Collection Instrument: Data Request Rationale 290 VIII SEI Checklist 294 IX Examples of Work Breakdown Structures 295 X Learning Curve Analysis 303 XI Technology Readiness Levels 310 XII Schedule Risk Analysis 311 XIII EVM-Related Award Fee Criteria 315 XIV GAO Contacts and Acknowledgments 319 References 320 Tables 1 GAO’s Basic Characteristics of Credible Cost Estimates 6 2 The Twelve Steps of a High-Quality Cost Estimating Process 9 3 Cost Estimating and EVM Criteria for Federal Agencies: Legislation, Regulations, Policies, and Guidance 23 Page iv GAO-07-1134SP Cost Assessment Guide Exposure Draft 4 Life-Cycle Cost Estimates, Business Case Analyses, and Other Types of Cost Estimates 30 5 Certification Standards in Business, Cost Estimating, and Financial Management in the Defense Acquisition Education, Training, and Career Development Program 50 6 Typical Technical Baseline Elements 54 7 General System Characteristics 57 8 Common Elements in Work Breakdown Structures 63 9 Cost Element Structure for a Standard DOD Automated Information System 66 10 Basic Primary and Secondary Data Sources 85 11 Three Cost Estimating Methods Compared 101 12 An Example of the Analogy Cost Estimating Method 102 13 An Example of the Engineering Build-Up Cost Estimating Method 104 14 An Example of the Parametric Cost Estimating Method 107 15 The Twelve Steps of High-Quality Cost Estimating Summarized 120 16 Sizing Metrics and Commonly Associated Issues 121 17 Common Software Risks That Affect Cost and Schedule 125 18 Best Practices Associated with Risks in Implementing ERP 129 19 Potential Sources of Program Cost Estimate Uncertainty 143 20 A Hardware Risk Scoring Matrix 148 21 A Software Risk Scoring Matrix 149 22 Eight Common Probability Distributions 151 Page v GAO-07-1134SP Cost Assessment Guide Exposure Draft 23 The Twelve Steps of High-Quality Cost Estimating Mapped to Best Practice Criteria 160 24 Questions for Checking the Accuracy of Estimating Techniques 163 25 Eight Types of Independent Cost Estimate Reviews 167 26 What Cost Estimate Documentation Includes 172 27 Key Benefits of Implementing EVM 185 28 ANSI Guidelines for EVM Systems 188 29 EVM Implementation Guides 190 30 Typical Methods for Measuring Earned Value Performance 198 31 Integrated Baseline Review Risk Categories 213 32 Contract Performance Report Data Elements: Format 1 220 33 EVM Performance Indexes 226 34 Best Predictive EAC Efficiency Factors by Program Completion Status 234 35 Basic Program Management Questions That EVM Data Help Answer 238 36 NSI Guidelines Related to Incorporating Changes in an EVM System 241 37 Elements of an Effective Surveillance Organization 244 38 Risk Factors That Warrant EVM Surveillance 245 39 A Program Surveillance Selection Matrix 246 40 A Color-Category Rating System for Summarizing Program Findings 250 41 OTB Funding Implications by Contract Type 253 42 Common Indicators of Poor Program Performance 254 Page vi GAO-07-1134SP Cost Assessment Guide Exposure Draft 43 Options for Treating Variances in Performing a Single Point Adjustment 257 44 Case Studies Drawn from GAO Reports Illustrating This Guide 265 45 Phases of the Budget Process 275 46 The Budget Process: Major Steps and Time Periods 276 47 Aircraft System Work Breakdown Structure 295 48 Electronic/Automated Software System Work Breakdown Structure 296 49 Ground Software Work Breakdown Structure 297 50 Missile System Work Breakdown Structure