Probability and Statistics

v1.2

Mathematical underpinnings of cost estimating

“God does not play dice with the cosmos.” -Albert Einstein “Do not presume to tell God what to do.” -Niels Bohr

http://en.wikiquote.org/wiki/Quantum_mechanics

Unit III - Module 10 1

v1.2 Acknowledgments

• ICEAA is indebted to TASC, Inc., for the development and maintenance of the Cost Estimating Body of Knowledge (CEBoK®) – ICEAA is also indebted to Technomics, Inc., for the independent review and maintenance of CEBoK® • ICEAA is also indebted to the following individuals who have made significant contributions to the development, review, and maintenance of CostPROF and CEBoK ® • Module 10 Probability and Statistics – Lead authors: Megan E. Dameron, Christopher J. Leonetti, Casey D. Trail – Assistant authors: Jessica R. Summerville, Jennifer K. Murrill, Sarah E. Grinnell – Senior reviewers: Richard L. Coleman, Kevin Cincotta, Fred K. Blackburn – Reviewers: Robyn Kane, Matthew J. Pitlyk, Maureen L. Tedford – Managing editor: Peter J. Braxton

Unit III - Module 10 2

ICEAA 2016 Bristol – TRN 07 v1.2 Unit Index Unit I – Cost Estimating Unit II – Cost Analysis Techniques Unit III – Analytical Methods 6. Basic Data Analysis Principles 7. Learning Curve Analysis 8. Regression Analysis 9. Cost and Schedule Risk Analysis 10.Probability and Statistics Unit IV – Specialized Costing Unit V – Management Applications

Unit III - Module 10 3

v1.2 Prob/Stat Overview

• Key Ideas • Practical Applications – Probability and Statistics as 6 – Descriptive Statistics “Flipsides” • Mean, Median, Mode, CV – Central Tendency and Dispersion – CER Development – The Bell Curve 8 • t, F, R2, CI, PI • Normal (Gaussian) Distribution – Modeling Uncertainty and Risk and the CLT 9 • Normal, Triangular, Lognormal – Inference

• Analytical Constructs • Related Topics – Counting and Fractions – Stochastic Processes • Combinations and Permutations • Markov Chains • Pascal’s Triangle • Queueing Theory Probability – Distributions and Calculus – Simulation • pdfs and cdfs • Discrete Event • Limits • Continuous • (Maximum Likelihood) Estimators – Data Analysis – Hypothesis Testing – Regression Analysis Statistics • Test statistic, critical values, significance – Design of Experiments Unit III - Module 10 4

ICEAA 2016 Bristol – TRN 07 Prob/Stat Within The v1.2 Cost Estimating Framework Past Present Future Understanding your Developing Estimating the new historical data estimating tools system

2 3 4 5 6 7 8 9 10 11 12 2 3 4 5 6 7 8 9 10 11 12 23456789101112 Statistics Probability Probability

2 3 4 5 6 7 8 9 10 11 12 23456789101112

Observed outcomes Statistics are used as Fitted distributions are the result of the “best guesses” for are used to model “cosmic roll of the underlying population subsequent dice” parameters outcomes

Unit III - Module 10 5

v1.2 Probability

• Probability is the mathematical study of the future; the chance of an event or outcome, or the range of possible outcomes – Divided into discrete and continuous probability – Encompasses a number of models for outcomes, called distributions, such as normal (or Gaussian), triangular, and many others – There is a subset of probability called stochastic 11 processes involving models that change over time

Normal Distribution

0.4

0.3

0.2

0.1

0.0 Note: This is a layman’s definition 2 3 4 5 6 7 8 9 10 11 12 -4 -3 -2 -1 0 1 2 3 4 Unit III - Module 10 9

ICEAA 2016 Bristol – TRN 07 v1.2 Statistics

• Statistics is the mathematical study of the past; it involves describing outcomes, or inferring from outcomes what the underlying probability model might be – Divided into: Note: This is a layman’s definition • Descriptive statistics: involving the portrayal of data sets themselves and derived rates, averages, and the like • Inferential statistics: involving tests to determine if a given probabilistic model might apply, what the value of a parameter might be, or testing whether two sets of outcomes are differentiable – Divided into: • Parametric statistics: involving assumptions about the underlying model • Non-parametric statistics: involving few or no assumptions about the underlying model

Unit III - Module 10 10

v1.2 Role in Cost Estimating • Descriptive statistics are used to 6 describe and compare cost data • Statistics provide the basis for the 8 development of Cost Estimating Relationships (CERs) via regression – Inferential statistics are used to adjudge the goodness of those CERs • Probability is used to quantify the 9 uncertainty present in a cost estimate

Unit III - Module 10 11

ICEAA 2016 Bristol – TRN 07 v1.2 Definitions – Population / Sample

•A population consists of all members of a particular group, e.g., all (metaphysically possible) US Navy destroyers •A sample is a subset of the population, e.g., DDG 51 and DD 963 classes Probability uses population parameters to describe future outcomes that are subject to Population chance Sample Parameters Statistics  x, s2

Statistics are used to draw conclusions about the population from outcomes in the past

Unit III - Module 10 12

v1.2 Definitions – Random Variables •A random variable takes on values that represent outcomes in the sample space – It cannot be fully controlled or exactly predicted • Discrete vs. Continuous – A set is discrete if it consists of a finite or countably infinite number of values • e.g., number of circuits, number of test failures • e.g., {1,2,3, …} – the random variable can only have a positive integer (natural number) value – A set is continuous if it consists of an interval of real numbers (finite or infinite length) • e.g., time, height, weight • e.g., [-3,3] – the random variable can take on any value in this interval

Unit III - Module 10 13

ICEAA 2016 Bristol – TRN 07 v1.2 Definitions – pmf

• Probability Mass Function f (pmf) Probability Mass Function X – The probability that the discrete random variable will take a value equal to a is the height of the bar at a.

PX  a  f X (a)

– The pmf accounts for all Possible outcome of Random Variable possible outcomes of the distribution • The sum of heights of all bars The probability of a given is 1. outcome is the height of the Tip: Discrete distributions are the corresponding histogram bar. exception, not the rule, in cost estimating

v1.2 Definitions – pdf

• Probability Density Function (pdf) – The total area under the curve is 1 • The probability that the random variable takes on some value in the range is 1 (100%) – The probability that the continuous random variable will take a value between a and b is the area under the curve between a and b The probability that this b random variable is between Pa  X  b  p x dx a a=5 and b=10 is equal to the shaded area (40%)

Unit III - Module 10 15

ICEAA 2016 Bristol – TRN 07 v1.2 Definitions – cdf

• Cumulative Distribution CumulativeCumulative Distribution Density Function Function Function (cdf) – This curve shows the 1.00 probability that the random 0.80 variable is less than x (a 0.60 particular value of X) The probability that 0.40 this random variable – The cdf reaches/ is less than b=10 is approaches 1 0.20 0.58 (58%) • The probability that the 0.00 random variable is less than 0.0 5.0 10.0 15.0 20.0 25.0 the maximum value (may be Possible Outcomes of Random Variable infinite) is 1 (100%) The probability that this random variable is Tip: The pdf less than a=5 is 0.18 shows the shape Pa  X  b  F b  Fa (18%) of the distribution; b a b the cdf shows the  px dx  p x dx  p x dx   a percentiles Unit III - Module 10 16

v1.2 Distribution Properties • Total probability = 1 (100%)    pxi 1 px dx 1 i1  • cdf is sum of pmf / integral of pdf j x F x  P X  x  px Fx  P X  x  p t dt j j  i  i1 • pmf is delta / pdf is derivative of cdf j j1 px j  P X  x j   pxi   p xi  Fx j  F x j1  i1 i1 Fx  t Fx px  lim  F'x t0 t Unit III - Module 10 17

ICEAA 2016 Bristol – TRN 07 v1.2

Measures of Central Tendency

Where is the “center” of the distribution?

Unit III - Module 10 19

v1.2 Mean

• The expected value of a distribution (population mean), is 6 calculated as the sum (integral) of a random variable’s possible 1 values multiplied by the probability that it takes on those values E X  x p x E X  xp x dx  i  i        1   2   6   2   1  EX  2   3   7  11  12   7  36   36   36   36   36 

x 2 x 2 x 2       x 2 1 2 x   2 EX  e 2 dx   e 2 dx  e 2 dx   2   2   2  x 2   2    e 2   2  Unit III - Module 10 20

ICEAA 2016 Bristol – TRN 07 v1.2 Note that the median may not Median be in the distribution • The median of a distribution is the value that exactly 2 6 divides the distribution (pdf) into equal halves (middle value or average of two middle values); “robust” to 1 extreme values px  1 m 1  i  pxi  px dx  x m 2 x m 2  i i 2 x 2 x 2      1 1 2 2 1 2 2 px dx  e dx  e dx 1 m 2   2   2 x 2 x 2    1 2  1 2 1 e 2 dx  e 2 dx  m     2   2 2 The median of a Normal distribution – as with any symmetric distribution with finite mean – is its mean

Unit III - Module 10 21

v1.2 Mean, Median, and Skew

• The mean and the median are equal if the distribution 6 is symmetric 4 • Unequal means and medians are an indication of skewness 20

Be ta Dis tr ibution Lognormal Distribution Median = Mean 0.06 2.5

0.05 Symmetric 2

0.04 1.5

0.03 1

0.02 Normal Distribution 0.5 0.01 0 0.4 0 0 0.2 0.4 0.6 0.8 1 1.2 0 5 10 15 20 25 30 35 0.3

0.2 Median < Mean 0.1 0.0 Median > Mean Skew(ed) Right -4 -3 -2 -1 0 1 2 3 4 Skew(ed) Left

Unit III - Module 10 22

ICEAA 2016 Bristol – TRN 07 v1.2 Mode • The mode of a distribution is the most frequent 6 value in the distribution (discrete) or the value 3 with the greatest probability density (continuous) x 2 Mode = 7  1 2 px  e 2  2 x 2  x   2 p'x   e 2  3 2 2 3 4 5 6 7 8 9 10 11 12

The mode of a Normal distribution pdf is increasing to the left of the mean – as with any unimodal symmetric and decreasing to the right of the mean distribution – is its mean

Unit III - Module 10 23

v1.2

Measures of Dispersion

How “spread out” is the distribution?

Unit III - Module 10 24

ICEAA 2016 Bristol – TRN 07 v1.2 Variance / Standard Deviation

• The variance of a distribution is the measure of the “spread” of the 6 distribution about its mean – the second “moment” std dev of about 2.42 2 2 VarX  EX    xi   pxi   1   2   6   2   1  35 VarX  25  16   0  16   25    5.83  36   36   36   36   36  6 VarX  EX  2   x  2 pxdx  2 x 2 x 2 x 2      x   2  x   2 1 2 VarX  e 2 dx   e 2  2 e 2 dx   2   2 2   2  The variance of a distribution is the square of , its standard deviation

Unit III - Module 10 25

v1.2 Coefficient of Variation

• The Coefficient of Variation (CV) expresses the standard 6 deviation as a percent of the mean Tip: Low CV  indicates less CV  dispersion, i.e., a  tighter distribution 5 6 • Large CVs indicate that the mean is a poor representation of the distribution 8 – Specify distribution using complete set of parameters, if possible – Include other parameters, such as variance •CV is invariant to scaling. – e.g., CV{1,2,3}=CV{100,200,300} • Not to be confused with the CV regression statistic

Unit III - Module 10 27

ICEAA 2016 Bristol – TRN 07 v1.2 Dispersion and CV

• These two distributions have the same mean, 16 but different standard deviations

6 0.3 This distribution has a higher CV (30%) and 0.25 has more dispersion 0.2 This distribution has a Lower CV 0.15 lower CV (15%) and is Higher CV more tightly distributed 0.1

0.05

0 -30 -20 -10 0 10 20 30

Unit III - Module 10 28

v1.2

Probability Distributions •Normal • Triangular • Student’s t • Bernoulli • Lognormal • Relationships between •F Distributions

Unit III - Module 10 29

ICEAA 2016 Bristol – TRN 07 v1.2 Normal (Gaussian) • The normal distribution, or “bell-shaped curve,” is the most prevalent distribution – Many naturally-occurring phenomena have a normal distribution, such as the height of people • The normal distribution is used in many statistical tests and applications • The normal distribution is symmetric about the mean

Unit III - Module 10 30

v1.2 Normal – Parameters • The normal distribution has two parameters: – The mean of the distribution,  – The standard deviation,  • If X is a random variable with a normal  distribution, we write X ~ N(,  ) • A standard normal is a normal distribution with  = 0 and  = 1 and is denoted Z ~ N(0,1) – If X ~ N(, ), then ( X   ) ~ N(0,1)  Unit III - Module 10 31

ICEAA 2016 Bristol – TRN 07 v1.2 Normal – pdf

Normal Distribution

The point -1.4 0.4 This is a standard represents 1.4 normal distribution: standard deviations 0.3 mean  = 0 and below the mean standard deviation  = 1 0.2  = 1 0.1

0.0 x(,) -4 -3 -2 -1 0 1 2 3 4 x 2  1 2  = 0 px  e 2  2 Unit III - Module 10 10 32

v1.2 Normal – Rules of Thumb

Normal Distribution

95.5% of observations 0.4 68.3% of observations from from a normal a normal distribution fall distribution fall within 2 0.3 within 1 standard deviation standard deviations of of the mean the mean 0.2

0.1 6

0.0 -4 -3 -2 -1 0 1 2 3 4 95% of the μ = 0 observations are σ = 1 within 1.96 standard 99.7% of observations from a normal deviations of the distribution fall within 3 standard mean deviations of the mean Unit III - Module 10 33

ICEAA 2016 Bristol – TRN 07 v1.2 Central Limit Theorem (CLT)

• The sum of a large number of independent, identically distributed (iid) random variables 7 from a population with finite mean and standard deviation approaches a normal distribution – Sample size required depends on the parent distribution, but as a rule of thumb, distributions approach normal by n = 30 • Correlation: As long as the sum is not dominated by a few large, highly correlated elements, the CLT will still hold Normality of Work Breakdown Structures, M. Dameron, J. Summerville, R. Coleman, N.St. Louis, Joint ISPA/SCEA Conference, June 2001.

Unit III - Module 10 34

v1.2 CLT – Example

• The graph below shows 3 triangular distributions and the sum of the 3 triangles

Overlay Chart Sum of Triangles .153 Nor mal Di str i buti on The sum of three ( 15.43, 1.52) .115 independent Triangle 1 triangular .077 Triangle 2 distributions tests as being .038 Triangle 3 normally Sum of 3 Triangles .000 distributed 0.00 5.00 10.00 15.00 20.00

“Normality of Work Breakdown Structures,” M. Dameron, J. Summerville, R. Coleman, N. St. Louis, Joint ISPA/SCEA Conference, June 2001.

Unit III - Module 10 35

ICEAA 2016 Bristol – TRN 07 v1.2 Normal Distribution Overview

Distribution Normal Distribution Parameters and Statistics 0.4

0.3 • Mean =  0.2 • Variance = 2 0.1 x (,) 0.0 • Skewness = 0 -4 -3 -2 -1 0 1 2 3 4

2 2 th x t   •2.5percentile =  -1.96 1 2 x 1 2 px  e 2 Fx   x  e 2 dt   2  2 • 97.5th percentile =  + 1.96 Key Facts Applications (X  ) – If X N(, ), then N(0,1) – Central Limit Theorem ~  ~ (“standard” normal) • Approximation of distributions • Limiting case of t-distribution 10 – Central Limit Theorem holds for n ≥ 30 – Regression Analysis – 68.3/95.5/99.7 Rule • Assumed error term – Limiting case of t distribution – Distribution of cost – Exponential of normal is lognormal • Default distribution – Dist: NORM.DIST(x, mean, stddev, cum) – Distribution of risk • cum = TRUE for cdf and FALSE for pdf • Symmetric risks and uncertainties – Inv cdf: NORM.INV(prob, mean, stddev) Unit III - Module 10 36

v1.2 Student’s t Distribution

• The t distribution is similar to the standard normal, but is flatter with more area in the tails – Parameter is the degrees of freedom, n • In t tests, n is directly related to sample size 8 • Note that in regression, the degrees of freedom will be (n-1)-k, where n is the number of data points and k is the number of independent variables – Symmetric about the mean • As the degrees of freedom increase, the t distribution approaches normal • The t distribution is very important for confidence intervals and hypothesis testing (inferential statistics), which will be explained in later slides

Unit III - Module 10 37

ICEAA 2016 Bristol – TRN 07 v1.2 t Distribution – pdf

As n increases, the t distribution approaches standard normal

Warning: Never approximate a t distribution with a standard normal! Unit III - Module 10 38

Danger of Approximating t Distribution v1.2 with a Normal Distribution

ICEAA 2016 Bristol – TRN 07 v1.2 t Distribution Overview

Distribution Parameters and Statistics 0.4 0.35 0.3 • Degrees of freedom = n 0.25 0.2 0.15 0.1 • Mean = 0 0.05 n x (,) 0 -4-3-2-101234• Variance = n  2 [(n 1) / 2] px  • Skewness = 0 n (n / 2)[1 (x2 / n)](n1) / 2 Key Facts Applications – As n approaches infinity, the t distribution – Confidence Intervals approaches Standard Normal N(0,1) – The t is distributed as t  • Mean of Normal variates – Excel (n) / n – Regression Analysis • cdf = T.DIST(x, n, tails) • Significance of individual coefficients – Default is left-hand tail, use .2T for two tails, .RT for right-hand tail – Hypothesis Testing • Inv cdf = T.INV(prob, n)

Unit III - Module 10 40

v1.2 Lognormal Distribution

• The lognormal distribution : – Formed by raising e to the power of (“exponentiating”) a normal random variable. Y ~ N (  ,  2 )  e Y is lognormal. – Not the log of a normal. Rather, a variable is lognormal if its (natural) log is normal. Also, the (natural) log of a lognormal is normal. – The mean of the related normal distribution,  – The standard deviation of the related normal distribution, 

Lognormal Distribution Tip: If the The 0.06 distribution of X

lognormal 0.05 is lognormal, then the natural distribution is 0.04 skewed right log (ln) of X is 0.03 normally 0.02 distributed.

0.01

0 0 5 10 15 20 25 30 35 Unit III - Module 10 41

ICEAA 2016 Bristol – TRN 07 v1.2 Lognormal Distribution Overview

• Distribution Lognormal Distribution Parameters and Statistics 0.06

0.05  0.04 • Median = e 0.03

0.02 x[0,) 0.01 • Std Deviation of ln X = σ 0 2 0 5 10 15 20 25 30 35  2 1   ln(x / ) 2  • Mean = e   px  exp 2 x 2  2  10 2 2   • Variance = e2 e 1 Key Facts Applications – If X has a lognormal distribution, then – Risk Analysis ln(X) has a normal distribution – For small standard deviations, the normal approximates the lognormal distribution • For CVs < 25%, this holds 10 – Excel • Cdf = LOGNORM.DIST(x, mean, stddev) • Inv cdf = LOGNORM.INV(prob, mean, stddev) Unit III - Module 10 42

v1.2 Triangular Distribution

• The triangular distribution has three parameters: 18 minimum, mode, and maximum • Can be symmetric or skewed • Often used in risk analysis 9 – Especially useful for eliciting and quantifying expert opinions – Almost never found anywhere else

“Do Not Sum ‘Most Likely’ Costs,” Stephen A. Book, IDA/OSD CAIG Cost Symposium, May 1992.

Unit III - Module 10 44

ICEAA 2016 Bristol – TRN 07 v1.2 Triangular Distribution Overview

Distribution Parameters and Statistics 0.035 19 0.030 •Min = a 0.025 0.020 pdf •Max = b 0.015 0.010 0.005 • Mode = c (a ≤ c ≤ b) 0.000 a  b  c 0 20406080100120 • Mean = 3 2x  a /b  a c  a a  x  c px   10 a2  b2  c2  ab  ac  bc 2b  x /b  a b  c c  x  b • Variance = 18  x  a 2  a  x  c  b  a c  a Fx   Key Facts b  x 2 1 c  x  b Applications – Excel  b  a b  c – Risk Analysis • pdf and cdf calculations can be handled • SME Input as they are listed in the above formulas – A symmetrical triangle approximates a normal when

a    6 ,c  ,b    6

Unit III - Module 10 45

v1.2 Bernoulli Distribution Overview

Distribution Parameters and Statistics •Min = 0 p q •Max = 1 01 • Mean = p 0 x  0  q 1 p x  0 Fx  q 0  x 1 • Variance = pq px     p x 1 1 x 1

Key Facts Applications – Excel – Risk Analysis • pdf and cdf calculations can be • Discrete Risks handled as they are listed in the above formulas • X = Cf * Bernoulli – The sum of n Bernoullis is Binomial •p = Pf (n,p)

Unit III - Module 10 46

ICEAA 2016 Bristol – TRN 07 v1.2 Relationships Between Distributions

http://www.math.wm.edu/~leemis/chart/UDR/UDR.html X    Normal(μ, σ2) L lnX    0, 1

X e 2 Standard Normal  X Lognormal(μ, σ ) L P Beta(a,b) n   2 X r  X i Chi Square a 1,b 1 C a  r t(n) b  n  r 1

Exponential(a) Standard Uniform Rayleigh(a) F M S V V F(m,n) b  2 X1  X 2 Weibull(a,b) lnX  M S V Standard Triangular V Extreme Value(a,b) a  1,b  1,c  0 M V “Univariate Distribution Relationships,” Lawrence M. Leemis, Jacquelyn T. McQueston, The American Statistician, February 2008, Vol. 61, No. 1. Triangular(a,b,c) V Unit III - Module 10 101

v1.2

Hypothesis Testing

• One Tail vs. Two Tail • Critical Values • Statistical • P-Values Significance • Confidence Intervals • Test Statistics

Unit III - Module 10 49

ICEAA 2016 Bristol – TRN 07 v1.2 Hypothesis Test • Hypothesis tests are often used to test for differences between population groups • Two hypotheses are proposed: “innocent until proven guilty” –H0 is the null hypothesis

•H0 is presumed to be true unless the data is proven to contradict the null hypothesis “beyond a –H1 is the alternative hypothesis reasonable doubt” •H1 may only be accepted with statistical evidence contradicting the null hypothesis

Unit III - Module 10 50

v1.2 Examples of Hypotheses • Test to see if two populations have different means t test

H0:   (the means are the same) F test H1:   (the means are different) • Test to see if two populations have different standard deviations H :   (the std devs are the same) Chi Square or 0   K-S test H1:  (the std devs are different) • Test to see if two populations are identically distributed

H0: f(x) = g(x) (the distributions are the same) H1: f(x) g(x) (the distributions are different)

Unit III - Module 10 51

ICEAA 2016 Bristol – TRN 07 v1.2 One Tail vs. Two Tail • A hypothesis can be one-tailed or two- tailed • A one-tailed test makes an assumption about the direction of difference

10 – E.g., H0:   H1:   • A two-tailed test makes no assumption about the direction of difference

– E.g., H0:   We will look at this case in our main example. H1:  

Unit III - Module 10 52

v1.2 Statistical Significance

• First, we must choose a level of significance, denoted  •  is the probability of incorrectly rejecting the null hypothesis Convicting an – This is called a Type I error innocent person – Typical significant levels are  = 0.05 and  = 0.10 • The customary level of significance is  = 0.05

– Rejecting H0 at an  = 0.05 level of significance means that there is less than a 5% probability that H is true 0 Tip:  = 0.05 is the most common level of significance.  = 0.10 is occasionally used. Any other values are very rare Unit III - Module 10 53

ICEAA 2016 Bristol – TRN 07 v1.2 Test Statistic • A test statistic is a function of the sample data – Calculated under the assumption that the 11 null hypothesis is true • The decision to accept or reject the null hypothesis is based on the value of the test statistic • Different types of hypothesis tests will have different test statistics (many of these will be discussed on later slides)

Unit III - Module 10 54

v1.2 Critical Value

• A critical value c is such that the probability of getting a test statistic greater than c (in absolute value) is equal to  for a one tailed test and /2 for a two-tailed test 12 • If the test statistic is greater than c, we reject H0 Student 't' Distribution Student 't' Distribution 38 degrees of freedom 38 degrees of freedom 0.45 0.45 0.4 0.4 Fail to 0.35 0.35 Fail to 0.3 reject H0 0.3 0.25 reject H0 0.25 0.2 Reject H Reject H 0.2 0.15 0 0 0.15 0.1 Reject H 0.1 0 0.05 0.05 0 0 -4 -3 -2 -1 0 1 2 3 4 c -4 -3 -2 -1 0 1 2 3 4 -c c P(t>c) =  P(t<-c) = /2 P(t>c) = /2 Unit III - Module 10 55

ICEAA 2016 Bristol – TRN 07 v1.2 Example Problem • Suppose we have historical cost growth factors from a set of DoD programs and a set of NAVAIR programs • We wish to see if cost growth for NAVAIR programs differs from DoD-wide growth • The hypotheses are:

H0: N = D (the means are the same) H1: N D (the means are different) • This is a two-tailed test as we are making no assumptions as to whether or not NAVAIR has higher or lower growth than DoD

“NAVAIR Cost Growth Study,” R.L. Coleman, M.E. Dameron, C.L. Pullen, J.R. Summerville, D.M. Snead, 34th DoDCAS and ISPA/SCEA 2001.

Unit III - Module 10 56

v1.2 Example Problem Data

• Suppose we have the DoD NAVAIR 1.26 1.26 following cost growth factors 1.44 1.92 0.96 1.64 0.93 1.83 (CGF) for DoD and NAVAIR 1.26 1.85 0.88 1.03 programs 1.10 1.08 1.23 1.44 • Average DoD CGF = 1.19 0.76 1.60 1.75 1.24 • Average NAVAIR CGF = 1.33 1.80 1.04 1.56 1.21 1.24 1.11 But is this a “real” 1.51 1.31 0.93 1.47 difference, or did it 0.49 1.25 happen by chance? 1.29 1.11 0.76 1.15 1.21 1.11 Unit III - Module 10 1.50 0.9057

ICEAA 2016 Bristol – TRN 07 v1.2 Example Test Statistic

• For each different type of hypothesis test, there is a corresponding test statistic • In our example problem, we will be using the two- sample t-test for means 2 2 – Let Xn ~ N(x,  ) and Ym ~ N(y,  ) where X and Y are independent 2 2 – Let Sx and Sy be the two sample variances 2 – Let Sp be the pooled variance, where (n 1)S 2  (m 1)S 2 S 2  X Y p n  m  2 X Y  (   ) – Then, T  X Y 1 1 S  p n m has a t distribution with (n + m – 2) degrees of freedom Unit III - Module 10 58

Example Test Statistic v1.2 Calculation • Using our example problem data, we DoD NAVAIR 1.26 1.26 get the following: 1.44 1.92 • X = 1.19, Y = 1.33 0.96 1.64 2 2 0.93 1.83 •Sx = 0.12, Sy = 0.09 1.26 1.85 2 0.88 1.03 •Sp = (20-1)(0.12) + (20-1)(0.09) 1.10 1.08 20 + 20 – 2 1.23 1.44 0.76 1.60 = 0.11 1.75 1.24 • If H is true, then ( –  ) = 0 1.80 1.04 0 x y 1.56 1.21 • So, under H0, 1.24 1.11 1.19 - 1.33 - 0 1.51 1.31 T   1.32 0.93 1.47 1 1 0.49 1.25 0.11(  ) 1.29 1.11 20 20 0.76 1.15 1.21 1.11 Unit III - Module 10 1.50 0.9059 59 © 2002-2013 ICEAA. All rights reserved.

ICEAA 2016 Bristol – TRN 07 v1.2 Example Critical Values

Student 't' Distribution Since we do not have 38 degrees of freedom T < -2.02 or T > 2.02, The test statistic 0.45 we fail to reject the for our example is 0.4 null hypothesis. There T = –1.32 0.35 is no statistical 0.3 difference between 0.25 the two means at  = 0.2 0.05 P(t < -2.02) = 0.15 /2 = 0.025 0.1 P(t > 2.02) = /2 0.05 = 0.025 0 -4 -3 -2 -1 0 1 2 3 4

The critical value for  = 0.05 and 38 df is 2.02

Unit III - Module 10 60

Example t Test v1.2 Using Excel • In Excel, use the Data Analysis Add-In to run a t test DoD NAVAIR Tip: The Excel default Mean 1.193094 1.327704 significance level is  Variance 0.117941 0.089239 = 0.05 Observations 20 20 Pooled Variance 0.10359 Hypothesized Mean Difference 0 test statistic for df 38 our example t Stat -1.322571 P(T<=t) one-tail 0.096942 t Critical one-tail 1.685953 Critical value for a one-tailed P(T<=t) two-tail 0.193883 test. In this case, we would t Critical two-tail 2.024394 still fail to reject H0

Critical value for our two- Warning: Results of macros tailed test. Fail to reject do not update if your data the null hypothesis. change!

Unit III - Module 10 61

ICEAA 2016 Bristol – TRN 07 v1.2 Example p-Values

The p-value is the Student 't' Distribution 38 degrees of freedom probability that we The test statistic 0.45 would observe a value for our example is 0.4 as extreme as the test T = –1.32 0.35 statistic by chance. 0.3 Thus there is a 0.25 19.39% chance that 0.2 the value T=-1.32 was 0.15 by chance and not 0.1 from the means being statistically different. 0.05 0 -4 -3 -2 -1 0 1 2 3 4

Example p-Values v1.2 Using Excel • The p-value is the smallest level of significance for which H0 could have been rejected DoD NAVAIR For a two-tailed test, Mean 1.193094 1.327704 we could reject the Variance 0.117941 0.089239 null hypothesis of Observations 20 20 equal means at α = Pooled Variance 0.10359 0.194. But then there Hypothesized Mean Difference 0 would still be a 19.4% df 38 t Stat -1.322571 probability that we P(T<=t) one-tail 0.096942 incorrectly rejected t Critical one-tail 1.685953 H0. P(T<=t) two-tail 0.193883 t Critical two-tail 2.024394

Warning: Choose the level of significance ahead of time and stick with it! Unit III - Module 10 63

ICEAA 2016 Bristol – TRN 07 v1.2 Confidence Intervals

• A confidence interval (CI) suggests to us that we are (1-)·100% confident that the true parameter 13 value is contained within the calculated range* – The range is calculated using the estimated parameter value • Confidence intervals can be calculated for a variety of different parameters and distributions

/2 /2 1 - 

critical values * Note this statement provides a general sense of what a confidence interval does for us in concise language for ease of understanding. The specific statistical interpretation is that if many independent samples are taken where the levels of the predictor variableare the same as in the data set, and a (1-)*100% confidence interval is constructed for each sample, then (1-) ·100% of the intervals will contain the true value of the parameter. Unit III - Module 10 64

v1.2 Example CI Formula • Let us suppose that the Cost Growth Factors in our example problem are normally distributed • We will find a 95% confidence interval for the average DoD cost growth • The formula is  s s   y  t / 2,n1 , , y  t / 2,n1   n n 

Unit III - Module 10 65

ICEAA 2016 Bristol – TRN 07 v1.2 Example CI Calculation • Going back to the data from our example problem, we find: y = mean DoD Cost Growth Factor = 1.19 s = standard deviation of DoD data = 0.34 n = sample size = 20 • We want a 95% CI, so we have  = 0.05

• Now, we need t/2, n-1 – We can find this on a table or by using the TINV function in Excel

–t0.025, 19 = 2.093 Tip: Use α with T.INV.2T or 1- α/2 with T.INV.

Unit III - Module 10 66

v1.2 Example CI Result • So, our confidence interval is  0.34 0.34  9 1.19  2.093 , ,1.19  2.093   20 20  1.03, ,1.35

Roughly speaking, we are 95% certain that the true value of the DoD Cost Growth Factor mean is between 1.03 and 1.35

Unit III - Module 10 67

ICEAA 2016 Bristol – TRN 07 v1.2 Prob/Stat Summary • A solid understanding of probability and statistics is vital to both cost and risk analysis • Descriptive statistics are used to characterize, describe, and compare the data – Central Tendency – mean, median, mode – Dispersion – variance, standard deviation, coefficient of variation • Inferential statistics are used to draw inferences from the data – Testing multiple population groups for differences in means, variances, distributions, etc. – Confidence intervals around estimates

Unit III - Module 10 75

ICEAA 2016 Bristol – TRN 07