Joint Agency Cost Estimating Relationship (CER) Development Handbook 9 February 2018 CER Development Handbook Preface There are many valid analytic methods for correlating potential technical or other program characteristics with cost. Analysts may find certain methods more suited to specific or niche applications than others – or have understandable preferences for particular methods. However, we recognized an overarching need within the Government cost analysis community to provide a handbook to serve as a fundamental reference for parametric Cost Estimating Relationship (CER) development. This handbook provides a variety of statistical techniques to help equip the analysts' toolboxes for attacking a range of estimating problems and data issues. Additionally, it presents tried-and-true processes to help analysts choose the appropriate method, understand the mathematics behind several of the available regression and curve-fitting techniques, validate and select the most representative CER, and properly document their efforts. Wendy P. Kunc Stephen G. Barth Director Deputy Assistant Secretary of the Army Naval Center for Cost Analysis (Cost & Economics) Charles Hunt Paul E. Titrault Agency Programmatic Analysis and Research Director, Cost Analytics and Parametric Estimating Capability Division Missile Defense Agency NASA Headquarters 2 CER Development Handbook Acknowledgements This handbook was conceptualized, developed and edited under the guidance of Mr. Duncan Thomas, Mr. John Fitch, and Mr. Bruce Parker of the Naval Center for Cost Analysis (NCCA). Principal support was provided by Mr. Nick Lanham of NCCA, and Mr. Alfred Smith of Tecolote Research. Creating this handbook has been a very rewarding experience, more so as it was developed through a joint effort with talented analysts representing multiple cost agencies across the Federal government. The work presented on the following pages is the product of dedicated individuals who committed countless hours in meticulously reviewing every paragraph, figure, and table, and providing insightful suggestions and material. This handbook is a far better product because of their generous and collaborative contributions. In particular, I wish to thank the following individuals and their organizations: Mr. Todd Andrews, NCCA Mr. Bruce Kraft, USAF Ms. Anna Bobkoskie, NAVAIR 4.2 Ms. Katherine McCormack, DASA-CE Mr. Benjamin Breaux, NCCA Mr. Justin Moul, NCCA Ms. Heather Brown, NCCA Mr. Praful Patel, NCCA Mr. Dane Cooper, NAVSEA Mr. David Proctor, NAVAIR 4.2 Mr. Mike Dorko, SPAWAR Mr. Kenneth Ragland, DASA-CE Mr. Marc Greenberg, NASA Mr. Daniel Schluckebier, NCCA Mr. David Henningsen, DASA-CE Ms. Mary Anne Scully, AFCAA Ms. Pamela Johnson, NCCA Dr. Christian Smart, MDA Mr. Jay Jordan, NRO Mr. Daniel Strickland, MDA Mr. Charles Kelley, DASA-CE Mr. Steve VanDrew, NAVAIR 4.2 Mr. Raymond Kleinberg, TACOM Ms. Corinne Wallshein, NCCA Our goal was to develop a handbook that can be easily understood and applied by junior analysts, and serve as a well regarded reference for senior analysts. We hope you find it useful. Duncan D. Thomas Technical Director Naval Center for Cost Analysis 3 CER Development Handbook TABLE OF CONTENTS TABLE OF CONTENTS ............................................................................................................. 4 TABLE OF FIGURES .................................................................................................................. 8 TABLE OF TABLES .................................................................................................................. 10 INTRODUCTION....................................................................................................................... 11 HOW TO USE THIS HANDBOOK ......................................................................................... 13 1.0 STEP 1: PURPOSE, SCOPE, COLLECT, VALIDATE, & NORMALIZE DATA ....... 14 1.1 Introduction ....................................................................................................................................... 14 1.2 Preparing to Collect Data .................................................................................................................. 14 1.3 Cost Estimate Purpose and Scope ..................................................................................................... 15 1.3.1 Cost Estimate Purpose ............................................................................................................... 15 1.3.2 Cost Estimate Scope and Work Breakdown Structure ............................................................... 15 1.3.3 Obtain Subject Matter Expert Guidance to Help Identify Potential Cost Drivers ..................... 17 1.3.4 Define Viable Hypothesis .......................................................................................................... 18 1.4 Sources of Data ................................................................................................................................. 19 1.5 Collect and Validate the Raw Data ................................................................................................. 21 1.6 Cost Data Normalization ................................................................................................................. 23 1.6.1 Content over Time ..................................................................................................................... 23 1.6.2 Accounting Changes over Time ................................................................................................. 23 1.6.3 WBS/CES mapping ................................................................................................................... 24 1.6.4 Escalation / Inflation .................................................................................................................. 25 1.6.5 Adjust for Quantity .................................................................................................................... 27 1.6.6 Cost Per Unit Characteristic ....................................................................................................... 29 1.6.7 Other Normalization Considerations ......................................................................................... 29 1.7 Linking Cost to Schedule .................................................................................................................. 30 1.8 Summary and Introducing the Electronics Example Dataset ............................................................ 30 2.0 STEP 2: ANALYZE NORMALIZED DATA .................................................................... 32 2.1 Overview ........................................................................................................................................... 32 2.2 Cost Estimating Methods .................................................................................................................. 33 2.3 Choosing Between Analogy, Straight Average or a CER ................................................................. 34 2.3.1 Assess Number of Observations ( ) .......................................................................................... 34 2.3.2 Analogy Estimate ....................................................................................................................... 35 2.3.3 Estimating with Very Small Data 풏Sets ....................................................................................... 35 2.3.4 Straight Average ........................................................................................................................ 36 2.4 Univariate Data Analysis .................................................................................................................. 38 2.4.1 Significant Digits ....................................................................................................................... 38 2.4.2 Descriptive Statistics .................................................................................................................. 39 2.4.3 Generate a Histogram ................................................................................................................ 40 2.5 Measure Correlation between Dependent and Independent Variables ............................................. 41 2.5.1 Correlation Types ...................................................................................................................... 42 2.5.2 Identify Redundant Variables and Potential Multicollinearity .................................................. 43 2.6 Scatter Plot of the Most Promising Cost Drivers .............................................................................. 44 2.7 Identify Potential Variable Sets ........................................................................................................ 46 2.8 Hypothesize Functional Form and Error Term ................................................................................. 46 2.8.1 Linear Functional Form ............................................................................................................. 50 2.8.2 Power Functional Form.............................................................................................................. 51 2.8.3 Exponential Functional Form .................................................................................................... 53 2.8.4 Logarithmic Functional Form ...................................................................................................