Analyzing data using Python Eric Marsden <[email protected]> The purpose of computing is insight, not numbers. – Richard Hamming These slides costs criteria decision-making Where does this fit into risk engineering? curve fitting data probabilistic model consequence model event probabilities event consequences risks 2 / 47 These slides Where does this fit into risk engineering? curve fitting data probabilistic model consequence model event probabilities event consequences risks costs criteria decision-making 2 / 47 Where does this fit into risk engineering? curve fitting data probabilistic model consequence model event probabilities event consequences These slides risks costs criteria decision-making 2 / 47 Descriptive statistics 3 / 47 Descriptive statistics ▷ Descriptive statistics allow you to summarize information about observations • organize and simplify data to help understand it ▷ Inferential statistics use observations (data from a sample) to make inferences about the total population • generalize from a sample to a population 4 / 47 Descriptive statistics Source: dilbert.com 5 / 47 Measures of central tendency ▷ Central tendency (the “middle” of your data) is measured either by the median or the mean ▷ The median is the point in the distribution where half the values are lower, and half higher • it’s the 0.5 quantile ▷ The (arithmetic) mean (also called the average or the mathematical expectation) is the “center of mass” of the distribution 피(푋) = ∫푏 푥푓 (푥)푑푥 • continuous case: 푎 • discrete case: 피(푋) = ∑푖 푥푖푃(푥푖) ▷ The mode is the element that occurs most frequently in your data 6 / 47 Illustration: fatigue life of aluminium sheeting Measurements of fatigue life (thousands of cycles until rupture) of > import numpy > cycles = numpy.array([370, 1016, 1235,[...] 1560, 1792]) strips of 6061-T6 aluminium sheeting, > cycles.mean() subjected to loads of 21 000 PSI. 1400.9108910891089 > numpy.mean(cycles) 1400.9108910891089 Data from Birnbaum and Saunders > numpy.median(cycles) (1958). 1416.0 Source: Birnbaum, Z. W. and Saunders, S. C. (1958), A statistical model for life-length of materials, Journal of the American Statistical Association, 53(281) 7 / 47 Aside: sensitivity to outliers Note: the mean is quite sensitive to outliers, the median much less. ▷ the median is what’s called a robust measure of central tendency > import numpy > weights = numpy.random.normal(80, 10, 1000) > numpy.mean(weights) 79.83294314806949 > numpy.median(weights) 79.69717178759265 > numpy.percentile(weights, 50) 79.69717178759265 # 50th percentile = 0.5 quantile = median > weights = numpy.append(weights, [10001, 101010]) # outliers > numpy.mean(weights) 190.4630171138418 # <-- big change > numpy.median(weights) 79.70768232050916 # <-- almost unchanged 8 / 47 Measures of central tendency If the distribution of data is symmetrical, then the mean is equal to the median. If the distribution is asymmetric (skewed), the mean is generally closer to the skew than the median. Degree of asymmetry is measured by skewness (Python: scipy.stats.skew()) Negative skew Positive skew 9 / 47 Measures of variability ▷ Variance measures the dispersion (spread) of observations around the mean • 푉푎푟(푋) = 피 [(푋 − 피[푋])2] • continuous case: 휎2 = ∫(푥 − 휇)2푓 (푥)푑푥 where 푓 (푥) is the probability density function of 푋 2 1 푛 2 • discrete case: 휎 = 푛−1 ∑푖=1(푥푖 − 휇) • note: if observations are in metres, variance is measured in 푚2 • Python: array.var() or numpy.var(array) ▷ Standard deviation is the square root of the variance • it has the same units as the mean • Python: array.std() or numpy.std(array) 10 / 47 Exercise: Simple descriptive statistics Task: Choose randomly 1000 integers from a uniform distribution between 100 and 200. Calculate the mean, min, max, variance and standard deviation of this sample. > import numpy > obs = numpy.random.randint(100, 201, 1000) > obs.mean() 149.49199999999999 > obs.min() 100 > obs.max() 200 > obs.var() 823.99793599999998 > obs.std() 28.705364237368595 11 / 47 Histograms: plots of variability Histograms are a sort of bar graph that shows the distribution of data values. The vertical axis import matplotlib.pyplot as plt displays raw counts or proportions. # our Birnbaum and Sanders failure data plt.hist(cycles) To build a histogram: plt.xlabel("Cycles until failure") 1 Subdivide the observations into several equal classes or intervals (called “bins”) 25 20 2 Count the number of observations in each interval 15 3 Plot the number of observations in each 10 interval 5 Note: the width of the bins is important to obtain a 0 500 1000 1500 2000 2500 “reasonable” histogram, but is subjective. Cycles until failure 12 / 47 Quartiles ▷ A quartile is the value that marks one of the divisions that breaks a dataset into four equal parts ▷ The first quartile, at the 25th percentile, divides the 25% of 25% of observations first ¼ of cases from the latter ¾ observations ▷ The second quartile, the median, divides the dataset in 25% of observations half 25% of observations ▷ The third quartile, the 75th percentile, divides the first ¾ of cases from the latter ¼ interquartile range ▷ The interquartile range (IQR) is the distance between the first and third quartiles • 25th percentile and the 75th percentile 13 / 47 Box and whisker plot A “box and whisker” plot or boxplot shows the spread of import matplotlib.pyplot as plt the data plt.boxplot(cycles) ▷ the median (horizontal line) plt.xlabel("Cycles until failure") ▷ lower and upper quartiles Q1 and Q3 (the box) 2500 2000 ▷ upper whisker: last datum < Q3 + 1.5×IQR 1500 ▷ the lower whisker: first datum > Q1 - 1.5×IQR 1000 Cycles until failure 500 ▷ any data beyond the whiskers are typically called 1 outliers Note that some people plot whiskers th and 95 th differently, to represent the 5 percentiles for example, or even the min and max values… 14 / 47 Violin plot Adds a kernel density estimation to a boxplot 2500 2000 1500 import seaborn as sns 1000 sns.violinplot(cycles, orient="v") plt.xlabel("Cycles until failure") 500 0 Cycles until failure 15 / 47 Bias and precision Precise Imprecise Biased Unbiased A good estimator should be unbiased, precise and consistent (converge as sample size increases). 16 / 47 Estimating values ▷ In engineering, providing a point estimate is not enough: we also need to know the associated uncertainty • especially for risk engineering! ▷ One option is to report the standard error ̂휎 ̂휎 • √푛 , where is the sample standard deviation (an estimator for the population standard deviation) and 푛 is the size of the sample • difficult to interpret without making assumptions about the distribution ofthe error (often assumed to be normal) ▷ Alternatively, we might report a confidence interval 17 / 47 Confidence intervals ▷ A two-sided confidence interval is an interval [퐿, 푈] such that C% of the time, the parameter of interest will be included in that interval • most commonly, 95% confidence intervals are used ▷ Confidence intervals are used to describe the uncertainty in apoint estimate • a wider confidence interval means greater uncertainty 18 / 47 Interpreting confidence intervals population mean m m m A 90% confidence interval means that 10% of m the time, the parameter of interest will not be m included in that interval. m Here, for a two-sided confidence interval. m m m m 19 / 47 Interpreting confidence intervals population mean m m m A 90% confidence interval means that 10% of m the time, the parameter of interest will not be m included in that interval. m Here, for a one-sided confidence interval. m m m m 20 / 47 Illustration: fatigue life of aluminium sheeting import seaborn as sns Confidence intervals can be displayed graphically on a barplot, as “error lines”. sns.barplot(cycles, ci=95, capsize=0.1) plt.xlabel("Cycles until failure (95% CI)") Note however that this graphical presentation is ambiguous, because some authors represent the standard deviation on error bars. The caption should always state what the error bars represent. 0 200 400 600 800 1000 1200 1400 Cycles until failure, with 95% confidence interval Data from Birnbaum and Saunders (1958) 21 / 47 Statistical inference population sample Statistical inference means deducing information about a population by examining only a subset of the population (the sample). We use a sample statistic to estimate a population parameter. 22 / 47 How to determine the confidence interval? ▷ If you make assumptions about the distribution of your data (eg. “the observations are normally distributed”), you can calculate the confidence interval for your estimation of the parameter of interest (eg. the mean) using analytical quantile measures ▷ If you don’t want to make too many assumptions, a technique called the bootstrap can help ▷ General idea: • I want to determine how much uncertainty is generated by the fact that I only have a limited sample of my full population • If I had a “full” sample, I could extract a large number of limited samples and examine the amount of variability in those samples (how much does my parameter of interest vary across these samples?) • I only have a limited sample, but I can look at a large number of limited samples from my own limited sample, by sampling with replacement 23 / 47 Bootstrap methods ▷ “Bootstrapping” means resampling your data with replacement ▷ Instead of fitting your model to the original 푋 and 푦, you fit your model to resampled versions of 푋 and 푦 many times ▷ Provides 푛 slightly different models from which we create a confidence interval ▷ Hypothesis: observations