Estimating by the Zodiac

Estimating By The Zodiac

2017 NASA Cost and Schedule Analysis Symposium NASA Headquarters, Washington DC August 2017 Joe Hamaker, PhD Galorath Federal

• One day a senior cost estimator arrived at work early and as she was walking across the nearly empty parking lot from her car to her office building, a piece of paper blew across the pavement • Being a good citizen, she picked up the paper, planning to dispose of it properly once inside the building • To her surprise, she noticed that across the top of the piece of paper, in bold letters, it said “The Zodiac Model: World’s Best Spacecraft Cost Model” • The body of the paper contained an equation • Studying the equation a bit more, she determined that it purported to be a CER that would estimate the development cost of spacecraft based on several variables- - variables that didn’t make any engineering sense at all

• The astrological sign of the Project Manager coded as follows: • Aquarius = 1 • Pisces = 2 • Aries = 3 • Etc. • The percent of the project team who were vegetarians • For an indicator variable called Zeus, the model instructed the user to wear a tin foil hat to shield the brain from electromagnetic fields; then flip a coin: Heads assign 0, Tails assign 1 to the variable Zeus • Percent of PC vs Mac users on the team • Take Project Office cost estimate in millions, multiply by pi, and use the resulting number for Man Wearing a Tin Foil Hat the spacecraft mass in kilograms • The number of alphanumeric characters that were in the headlines of the morning edition of the New York Times on the first day of the Project PDR • The Chief Engineer’s shoe size in European units

• Where… • Zodiac = Astrological sign of PM coded as Aquarius=1, Pisces=2, Ares=3, etc. • The percent of the project team who were vegetarians • For an indicator variable called Zeus, wear a tin foil hat; then flip a coin: Heads assign 0, Tails assign 1 • Percent of PC vs Mac users on the team • Project Office cost estimate in millions multiplied by pi, for the spacecraft mass in kilograms • Number of alphanumeric characters in the headlines of the morning edition of the New York Times on the first day of the Project PDR • The Chief Engineer’s shoe size in European units

• While recognizing that was ridiculous, our estimator was working on a project for which she thought she could gather the data for the model • (Although obtaining the Zodiac sign of the PM and the shoe size of the Chief Engineer was a bit awkward) • The data was…

Variable Value Zodiac sign of the Project Manager, Aries 3 Percent of the team that are vegetarians 7% Zeus variable (from coin flip in tin foil hat) 1 Percent of PC users vs Mac users on the team 55% Mass variable (Project estimate of $370M X 3.14 = 1162 kg) 1162 Headline variable 11 Chief Engineer's shoe size in European Units 34

=1210.3 * 0.55+((1162)^0.5)*11/34 = $677M

• Our estimator noticed that our result of $677M was nearly double the Project Office Estimate of $370M • That in itself proved nothing • So with greater effort over the next several days, as time permitted, she gathered the data for a dozen historical missions and ran them through the CER • The percent errors were all very small—all within a few percentage points of the actual cost and just about evenly distributed between positive and negative • Not a single percent error was over 5% • This CER was arguably better than anything she had ever seen before for a spacecraft cost model! © 2017 Copyright Galorath Incorporated 7 The Story Continues….

• Now as it happened, our estimator was teaching a beginning cost estimating class a few days later and she showed her students the results of the Zodiac model • She told the class exactly how she came by the CER and that she had no knowledge of the data base behind it • She asked the class to ruminate on this over lunch and when they returned, she wanted to lead them in a discussion of the efficacy, or lack thereof, of this model • After lunch, about 1/3 of the students objected to the CER totally, about 1/3 weren’t sure and about 1/3 thought that since it seemed to work, it should be used • Our senior estimator prepared to voice her opinion • She paused for dramatic effect…The students all leaned forward expectantly… • Then she said…..

• As we all know, correlation does not imply causation • I submit that when using cost models, even if we don’t have firm knowledge about causation, we should not necessarily care • From https://blogs.scientificamerican.com/guest-blog/9-bizarre- and-surprising-insights-from-data-science/ • Female-named hurricanes are more deadly • Over 6 decades of data show that hurricanes with “relatively feminine” names killed almost three times the number killed by hurricanes with “relatively male” names • So if that is true, it seems to me we’d be foolish not to prepare more for female named hurricanes even though we can't explain the reason • (At that site above they make the conjecture that it is because people become passive when the storm coming is named Sue but take it more seriously if it is name Samson) • Everyday we use tools we don’t fully understand (e.g. the internet) • My assertion is, we’d be foolish NOT to use the Zodiac Model!

© 2017 Copyright Galorath Incorporated 10 Afterword… • So what Zodiac has been about is… • To have a little fun, and • Make the point that sometimes we stress too much over the statistics of our models • Tending to forget that, at the end of the day, it really matters… • That a model does a decent job predicting cost • After all, we all know that weather models include inputs regarding a butterfly flapping it’s wings in South America • So it shouldn’t bother us that the Zodiac Model