GAO-09-3SP GAO Cost Estimating and Assessment Guide: Best
Total Page:16
File Type:pdf, Size:1020Kb
United States Government Accountability Office GAO Applied Research and Methods GAO COST ESTIMATING AND ASSESSMENT GUIDE Best Practices for Developing and Managing Capital Program Costs March 2009 GAO-09-3SP PREFACE The U.S. Government Accountability Office is responsible for, among other things, assisting the Congress in its oversight of the federal government, including agencies’ stewardship of public funds. To use public funds effectively, the government must meet the demands of today’s changing world by employing effective management practices and processes, including the measurement of government program performance. In addition, legislators, government officials, and the public want to know whether government programs are achieving their goals and what their costs are. To make those evaluations, reliable cost information is required and federal standards have been issued for the cost accounting that is needed to prepare that information.1 We developed the Cost Guide in order to establish a consistent methodology that is based on best practices and that can be used across the federal government for developing, managing, and evaluating capital program cost estimates. For the purposes of this guide, a cost estimate is the summation of individual cost elements, using established methods and valid data, to estimate the future costs of a program, based on what is known today. 2 The management of a cost estimate involves continually updating the estimate with actual data as they become available, revising the estimate to reflect changes, and analyzing differences between estimated and actual costs—for example, using data from a reliable earned value management (EVM) system.3 The ability to generate reliable cost estimates is a critical function, necessary to support the Office of Management and Budget’s (OMB) capital programming process.4 Without this ability, agencies are at risk of experiencing cost overruns, missed deadlines, and performance shortfalls—all recurring problems that our program assessments too often reveal. Furthermore, cost increases often mean that the government 1 Federal Accounting Standards Advisory Board, Statement of Federal Financial Accounting Standards No. 4: Managerial Cost Accounting Standards and Concepts (Washington, D.C.: July 1995). 2 In the context of the Cost Guide, a program refers to all phases in a capital asset’s life cycle—that is, concept analysis, technology definition, requirements planning, acquisition, and operations and maintenance. 3 EVM is a project management tool that integrates the technical scope of work with schedule and cost elements for investment planning and control. It compares the value of work accomplished in a given period with the value of the work expected in that period. Differences in expectations are measured in both cost and schedule variances. The Office of Management and Budget (OMB) requires agencies to use EVM in their performance-based management systems for the parts of an investment in which development effort is required or system improvements are under way. 4 Office of Management and Budget, Preparation, Submission, and Execution of the Budget, Circular No. A-11 (Washington, D.C.: Executive Office of the President, June 2006); Management of Federal Information Resources, Circular No. A-130 Revised (Washington, D.C.: Executive Office of the President, Nov. 28, 2000); and Capital Programming Guide: Supplement to Circular A-11, Part 7, Preparation, Submission, and Execution of the Budget (Washington, D.C.: Executive Office of the President, June 2006). www.whitehouse.gov/omb/circulars/index.html. GAO-09-3SP Preface i cannot fund as many programs as intended or deliver them when promised. The methodology outlined in this guide is a compilation of best practices that federal cost estimating organizations and industry use to develop and maintain reliable cost estimates throughout the life of a government acquisition program. By default, the guide will also serve as a guiding principle for our auditors to evaluate the economy, efficiency, and effectiveness of government programs. The U.S. Government Accountability Office, the Congressional Budget Office (CBO), and others have shown through budget simulations that the nation is facing a large and growing structural deficit in the long term, primarily because the population is aging and health care costs are rising. As Comptroller General David Walker noted, “Continuing on this unsustainable path will gradually erode, if not suddenly damage, our economy, our standard of living and ultimately our national security.”5 New budgetary demands and demographic trends will place serious budgetary pressures on federal discretionary spending, as well as on other federal policies and programs, in the coming years. As resources become scarce, competition for them will increase. It is imperative, therefore, that government acquisition programs deliver as promised, not only because of their value to their users but also because every dollar spent on one program will mean one less available dollar to fund other efforts. To get better results, programs will need higher levels of knowledge when they start and standardized monitoring metrics such as EVM so that better estimates can be made of total program costs at completion. 5 GAO, 21st Century Challenges: Reexamining the Base of the Federal Government, GAO-05-325SP (Washington, D.C.: February 2005), p. 1. ii Preface GAO-09-3SP CONTENTS Preface i Abbreviations xv Introduction 1 The Guide’s Case Studies 3 The Cost Guide in Relation to Established Standards 3 The Guide’s Readers 3 Acknowledgments 4 Chapter 1 The Characteristics of Credible Cost Estimates and a Reliable Process for Creating Them 5 Basic Characteristics of Credible Cost Estimates 6 A Reliable Process for Developing Credible Cost Estimates 8 Chapter 2 Why Government Programs Need Cost Estimates and the Challenges in Developing Them 15 Cost Estimating Challenges 17 Earned Value Management Challenges 23 Chapter 3 Criteria for Cost Estimating, EVM, and Data Reliability 25 Determining Data Reliability 29 Chapter 4 Cost Analysis Overview 31 Differentiating Cost Analysis and Cost Estimating 31 Main Cost Estimate Categories 31 The Overall Significance of Cost Estimates 36 Cost Estimates in Acquisition 37 The Importance of Cost Estimates in Establishing Budgets 38 Cost Estimates and Affordability 39 Evolutionary Acquisition and Cost Estimation 42 GAO-09-3SP Contents iii Chapter 5 The Cost Estimate’s Purpose, Scope, and Schedule 47 Purpose 47 Scope 47 Schedule 48 Chapter 6 The Cost Assessment Team 51 Team Composition and Organization 52 Cost Estimating Team Best Practices 53 Certification and Training for Cost Estimating and EVM Analysis 54 Chapter 7 Technical Baseline Description Definition and Purpose 57 Process 57 Schedule 58 Contents 58 Key System Characteristics and Performance Parameters 61 Chapter 8 Work Breakdown Structure 65 Best Practice: Product-Oriented WBS 65 Common WBS Elements 68 WBS Development 71 Standardized WBS 71 WBS and Scheduling 75 WBS and EVM 75 WBS and Risk Management 75 WBS Benefits 76 Chapter 9 Ground Rules and Assumptions 79 Ground Rules 79 Assumptions 79 Global and Element-Specific Ground Rules and Assumptions 80 Assumptions, Sensitivity, and Risk Analysis 82 Chapter 10 Data 89 Data Collection 89 Types of Data 91 Sources of Data 92 iv Contents GAO-09-3SP Data Applicability 95 Validating and Analyzing the Data 96 EVM Data Reliability 98 Data Normalization 100 Recurring and Nonrecurring Costs 101 Inflation Adjustments 102 Selecting the Proper Indexes 103 Data Documentation 104 Chapter 11 Developing a Point Estimate 107 Cost Estimating Methods 107 Production Rate Effects on Learning 121 Pulling the Point Estimate Together 122 Chapter 12 Estimating Software Costs 125 Unique Components of Software Estimation 126 Estimating Software Size 127 Estimating Software Development Effort 131 Scheduling Software Development 133 Software Maintenance 134 Parametric Software Estimation 135 Commercial Off-the-Shelf Software 136 Enterprise Resource Planning Software 137 Software Costs Must also Account for Information Technology Infrastructure and Services 139 Unique Components of IT Estimation 139 Chapter 13 Sensitivity Analysis 147 Sensitivity Factors 148 Steps in Performing a Sensitivity Analysis 149 Sensitivity Analysis Benefits and Limitations 150 Chapter 14 Cost Risk and Uncertainty 153 The Difference Between Risk and Uncertainty 153 Point Estimates Alone Are Insufficient for Good Decisions 154 Budgeting to a Realistic Point Estimate 155 Developing a Credible S Curve of Potential Program Costs 159 Risk Management 175 GAO-09-3SP Contents v Chapter 15 Validating the Estimate 179 The Cost Estimating Community’s Best Practices for Validating Estimates 179 Chapter 16 Documenting the Estimate 191 Elements of Cost Estimate Documentation 193 Other Considerations 195 Chapter 17 Presenting the Estimate to Management 197 Chapter 18 Managing Program Costs: Planning 199 Linking Cost Estimation as the Foundation for EVM Analysis 199 The Nature and History of EVM 202 Implementing EVM 204 Federal and Industry Guidelines for Implementing EVM 211 The Thirteen Steps in the EVM Process 215 Integrated Baseline Reviews 231 Award Fees 233 Progress and Performance-Based Payments Under Fixed-Price Contracts 234 Validating the EVM System 236 Chapter 19 Managing Program Costs: Execution 243 Validating the Performance Measurement Baseline with an IBR 243 Contract Performance Reports 249 Monthly EVM Analysis 255 Project Future Performance