Statistics with Jmp: Hypothesis Tests, Anova and Regression

STATISTICS WITH JMP: HYPOTHESIS TESTS, ANOVA AND REGRESSION

PETER GOOS AND DAVID MEINTRUP

STATISTICS WITH JMP: HYPOTHESIS TESTS, ANOVA AND REGRESSION

STATISTICS WITH JMP HYPOTHESIS TESTS, ANOVA AND REGRESSION

Peter Goos University of Leuven and University of Antwerp, Belgium

Registered office John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom

For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com.

The right of the authors to be identified as the authors of this work has been asserted in accordance withthe Copyright, Designs and Patents Act 1988.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book.

Limit of Liability/Disclaimer of Warranty: While the publisher and authors have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose.Itis sold on the understanding that the publisher is not engaged in rendering professional services and neither the publisher nor the authors shall be liable for damages arising herefrom. If professional advice or other expert assistance is required, the services of a competent professional should be sought.

Library of Congress Cataloging-in-Publication Data Names: Goos, Peter. | Meintrup, David. Title: Statistics with JMP : hypothesis tests, ANOVA, and regression / Peter Goos, David Meintrup. Description: Chichester, West Sussex : John Wiley & Sons, Inc., 2016. | Includes index. Identifiers: LCCN 2015039990 (print) | LCCN 2015047679 (ebook) | ISBN 9781119097150 (cloth)| ISBN 9781119097044 (Adobe PDF) | ISBN 9781119097167 (ePub) Subjects: LCSH: Probabilities–Data processing. | Mathematical statistics–Data processing. | Regression analysis. | JMP (Computer file) Classification: LCC QA273.19.E4 G68 2016 (print) | LCC QA273.19.E4 (ebook)| DDC 519.50285/53–dc23 LC record available at http://lccn.loc.gov/2015039990

A catalogue record for this book is available from the British Library.

Set in 10/12pt Times by Aptara Inc., New Delhi, India

1 2016 To Marijke, Bas, Loes, and Mien To Beatrice´ and Werner

Contents

Preface xv Acknowledgments xix

Part One ESTIMATORS AND TESTS 1

1 Estimating Population Parameters 3 1.1 Introduction: Estimators Versus Estimates 3 1.2 Estimating a Mean Value 4 1.2.1 The Mean of a Normally Distributed Population 4 1.2.2 The Mean of an Exponentially Distributed Population 6 1.3 Criteria for Estimators 6 1.3.1 Unbiased Estimators 8 1.3.2 The Efficiency of an Estimator 9 1.4 Methods for the Calculation of Estimators 9 1.5 The Sample Mean 10 1.5.1 The Expected Value and the Variance 10 1.5.2 The Probability Density of the Sample Mean for a Normally Distributed Population 12 1.5.3 The Probability Density of the Sample Mean for a Nonnormally Distributed Population 12 1.5.4 An Illustration of the Central Limit Theorem 14 1.6 The Sample Proportion 18 1.7 The Sample Variance 21 1.7.1 The Expected Value 22 1.7.2 The 𝜒2-Distribution 24 1.7.3 The Relation Between the Standard Normal and the 𝜒2-Distribution 25 1.7.4 The Probability Density of the Sample Variance 28 1.8 The Sample Standard Deviation 30 1.9 Applications 33

2 Interval Estimators 35 2.1 Point and Interval Estimators 35 viii Contents

2.2 Confidence Intervals for a Population Mean with Known Variance 36 2.2.1 The Percentiles of the Standard Normal Density 36 2.2.2 Computing a Confidence Interval 38 2.2.3 The Width of a Confidence Interval 39 2.2.4 The Margin of Error 41 2.3 Confidence Intervals for a Population Mean with Unknown Variance 41 2.3.1 The Student t-Distribution 42 2.3.2 The Application of the t-Distribution to Construct Confidence Intervals 45 2.4 Confidence Intervals for a Population Proportion 47 2.4.1 A First Interval Estimator Based on the Normal Distribution 47 2.4.2 A Second Interval Estimator Based on the Normal Distribution 49 2.4.3 An Interval Estimator Based on the Binomial Distribution 50 2.5 Confidence Intervals for a Population Variance 52 2.6 More Confidence Intervals in JMP 56 2.7 Determining the Sample Size 58 2.7.1 The Population Mean 60 2.7.2 The Population Proportion 63

3 Hypothesis Tests 67 3.1 Key Concepts 67 3.2 Testing Hypotheses About a Population Mean 71 3.2.1 The Right-Tailed Test 71 3.2.2 The Left-Tailed Test 77 3.2.3 The Two-Tailed Test 78 3.3 The Probability of a Type II Error and the Power 85 3.4 Determination of the Sample Size 88 3.5 JMP 90 3.6 Some Important Notes Concerning Hypothesis Testing 93 3.6.1 Fixing the Significance Level 94 3.6.2 A Note on the “Acceptance” of the Null Hypothesis 95 3.6.3 Statistical and Practical Significance 95

Part Two ONE POPULATION 97

4 Hypothesis Tests for a Population Mean, Proportion, or Variance 99 4.1 Hypothesis Tests for One Population Mean 99 4.1.1 The Right-Tailed Test 100 4.1.2 The Left-Tailed Test 102 4.1.3 The Two-Tailed Test 102 4.1.4 Nonnormal Data 107 4.1.5 The Use of JMP 108 4.2 Hypothesis Tests for a Population Proportion 110 4.2.1 Tests Based on the Normal Distribution 111 4.2.2 Tests Based on the Binomial Distribution 117 4.2.3 Testing Proportions in JMP 119 Contents ix

4.3 Hypothesis Tests for a Population Variance 122 4.3.1 The Right-Tailed Test 123 4.3.2 The Left-Tailed Test 124 4.3.3 The Two-Tailed Test 125 4.3.4 The Use of JMP 127 4.4 The Probability of a Type II Error and the Power 128 4.4.1 Tests for a Population Mean 130 4.4.2 Tests for a Population Proportion 140 4.4.3 Tests for a Population Variance and Standard Deviation 141

5 Two Hypothesis Tests for the Median of a Population 143 5.1 The Sign Test 144 5.1.1 The Starting Point of the Sign Test 145 5.1.2 Exact p-Values 147 5.1.3 Approximate p-Values Based on the Normal Distribution 148 5.2 The Wilcoxon Signed-Rank Test 151 5.2.1 The Use of Ranks 151 5.2.2 The Starting Point of the Signed-Rank Test 152 5.2.3 Exact p-Values 155 5.2.4 Exact p-Values for Ties 160 5.2.5 Approximate p-Values Based on the Normal Distribution 162 5.2.6 Approximate p-Values Based on the t-Distribution 164

6 Hypothesis Tests for the Distribution of a Population 167 6.1 Testing Probability Distributions 167 6.1.1 Known Parameters 168 6.1.2 Unknown Parameters 170 6.1.3 𝜒2-Tests for Qualitative Variables 172 6.2 Testing Probability Densities 181 6.2.1 The Normal Probability Density 182 6.2.2 Other Continuous Densities 195 6.3 Discussion 196

Part Three TWO POPULATIONS 199

7 Independent Versus Paired Samples 201

8 Hypothesis Tests for the Means, Proportions, or Variances of Two Independent Samples 205 8.1 Tests for Two Population Means for Independent Samples 205 8.1.1 The Starting Point 206 𝜎2 𝜎2 8.1.2 Known Variances 1 and 2 207 𝜎2 𝜎2 8.1.3 Unknown Variances 1 and 2 212 8.1.4 Confidence Intervals for a Difference in Population Means 222 8.2 A Hypothesis Test for Two Population Proportions 224 x Contents

8.2.1 The Starting Point 225 8.2.2 The Right-Tailed Test 226 8.2.3 The Left-Tailed Test 226 8.2.4 The Two-Tailed Test 226 8.2.5 Generalized Hypothesis Tests 227 8.2.6 The Confidence Interval for a Difference in Population Proportions 228 8.3 A Hypothesis Test for Two Population Variances 229 8.3.1 Fisher’s F-Distribution 230 8.3.2 The F-Test for the Comparison of Two Population Variances 232 8.3.3 The Confidence Interval for a Quotient of Two Population Variances 237 8.4 Hypothesis Tests for Two Independent Samples in JMP 237 8.4.1 Two Population Means 237 8.4.2 Two Population Proportions 244 8.4.3 Two Population Variances 247

9 A Nonparametric Hypothesis Test for the Medians of Two Independent Samples 251 9.1 The Hypotheses Tested 252 9.1.1 The Procedure 252 9.1.2 The Starting Point 253 9.2 Exact p-Values in the Absence of Ties 254 9.2.1 The Right-Tailed Test 256 9.2.2 The Left-Tailed Test 257 9.2.3 The Two-Tailed Test 261 9.3 Exact p-Values in the Presence of Ties 261 9.4 Approximate p-Values 265 9.4.1 The Right-Tailed Test 265 9.4.2 The Left-Tailed Test 266 9.4.3 The Two-Tailed Test 267

10 Hypothesis Tests for the Means of Two Paired Samples 273 10.1 The Hypotheses Tested 273 10.2 The Procedure 274 10.2.1 The Starting Point 274 𝜎 10.2.2 Known D 277 𝜎 10.2.3 Unknown D 278 10.3 Examples 280 10.4 The Technical Background 288 10.5 Generalized Hypothesis Tests 289 10.6 A Confidence Interval for a Difference of Two Population Means 290 𝜎 10.6.1 Known D 290 𝜎 10.6.2 Unknown D 291

11 Two Nonparametric Hypothesis Tests for Paired Samples 293 11.1 The Sign Test 294 11.1.1 The Hypotheses Tested 294 Contents xi

11.1.2 Practical Implementation 295 11.1.3 JMP 300 11.2 The Wilcoxon Signed-Rank Test 302 11.2.1 The Hypotheses Tested 302 11.2.2 Practical Implementation 303 11.2.3 Approximate p-Values 309 11.2.4 JMP 309 11.3 Contradictory Results 310

Part Four MORE THAN TWO POPULATIONS 311

12 Hypothesis Tests for More Than Two Population Means: One-Way Analysis of Variance 313 12.1 One-Way Analysis of Variance 314 12.2 The Test 320 12.2.1 Variance Within and Between Groups 320 12.2.2 The Test Statistic 324 12.2.3 The Decision Rule and the p-Value 327 12.2.4 The ANOVA Table 329 12.3 One-Way Analysis of Variance in JMP 330 12.4 Pairwise Comparisons 332 12.4.1 The Bonferroni Method 334 12.4.2 Tukey’s Method 338 12.4.3 Dunnett’s Method 341 12.5 The Relation Between a One-Way Analysis of Variance and a t-Test for Two Population Means 345 12.6 Power 345 12.6.1 The Noncentral F-Distribution 345 12.6.2 The Noncentral F-Distribution and Analysis of Variance 346 12.6.3 The Power and the Probability of a Type II Error 347 12.6.4 Determining the Sample Size and Power in JMP 350 12.7 Analysis of Variance for Nonnormal Distributions and Unequal Variances 355

13 Nonparametric Alternatives to an Analysis of Variance 357 13.1 The Kruskal–Wallis Test 358 13.1.1 Computing the Test Statistic 358 13.1.2 The Behavior of the Test Statistic 359 13.1.3 Exact p-Values 363 13.1.4 Approximate p-Values 366 13.2 The van der Waerden Test 367 13.3 The Median Test 372 13.4 JMP 377 xii Contents

14 Hypothesis Tests for More Than Two Population Variances 381 14.1 Bartlett’s Test 382 14.1.1 The Test Statistic 382 14.1.2 The Technical Background 382 14.1.3 The p-Value 384 14.2 Levene’s Test 385 14.3 The Brown–Forsythe Test 387 14.4 O’Brien’s Test 388 14.5 JMP 390 14.6 The Welch Test 392

Part Five ADDITIONAL USEFUL TESTS AND PROCEDURES 395

15 The Design of Experiments and Data Collection 397 15.1 Equal Costs for All Observations 397 15.1.1 Equal Variances 399 15.1.2 Unequal Variances 400 15.2 Unequal Costs for the Observations 402

16 Testing Equivalence 405 16.1 Shortcomings of Classical Hypothesis Tests 406 16.2 The Principles of Equivalence Tests 410 16.2.1 The Use of Two One-Sided Tests 410 16.2.2 The Use of a Confidence Interval 413 16.3 An Equivalence Test for Two Population Means 415 16.3.1 Independent Samples 415 16.3.2 Paired Samples 420

17 The Estimation and Testing of Correlation and Association 423 17.1 The Pearson Correlation Coefficient 423 17.1.1 A Test for 𝜌 = 0 424 𝜌 𝜌 ≠ 17.1.2 A Test for = 0 0 428 17.1.3 The Confidence Interval 432 17.2 Spearman’s Rank Correlation Coefficient 434 17.2.1 The Approximate Test for 𝜌(s) = 0 435 17.2.2 The Exact Test for 𝜌(s) = 0 437 𝜌(s) 𝜌(s) ≠ 17.2.3 The Approximate Test for = 0 0 441 17.2.4 The Confidence Interval 442 17.3 A Test for the Independence of Two Qualitative Variables 443 17.3.1 The Contingency Table 443 17.3.2 The Functioning of the Test 446 17.3.3 The Homogeneity Test 453

18 An Introduction to Regression Modeling 457 18.1 From a Theory to a Model 457 Contents xiii

18.2 A Statistical Model 459 18.3 Causality 461 18.4 Linear and Nonlinear Regression Models 464

19 Simple Linear Regression 467 19.1 The Simple Linear Regression Model 467 19.1.1 Examples 467 19.1.2 The Formal Description of the Model 468 19.2 Estimation of the Model 469 19.2.1 Intuition and Important Concepts 470 19.2.2 The Least Squares Method 474 19.3 The Properties of Least Squares Estimators 491 19.3.1 The Least Squares Estimators 494 𝛽̂ 𝛽̂ 19.3.2 The Expected Values of 0 and 1 496 19.3.3 A Demonstration of Unbiasedness by Means of a Simulation Study 498 𝛽̂ 𝛽̂ 19.3.4 The Variances of 0 and 1 499 19.3.5 The Gauss–Markov Theorem 501 19.4 The Estimation of 𝜎2 506 𝛽 𝛽 19.5 Statistical Inference for 0 and 1 509 19.5.1 The Normal Distribution of the Least Squares Estimators 509 19.5.2 A 𝜒2-Distribution for SSE 510 19.5.3 The t-Distribution 511 19.5.4 Confidence Intervals 512 19.5.5 Hypothesis Tests 514 19.6 The Quality of the Simple Linear Regression Model 519 19.6.1 The Coefficient of Determination 519 19.6.2 Testing the Significance of the Model 523 19.7 Predictions 527 19.7.1 The Estimation of an Expected Response Value 528 19.7.2 The Prediction of an Individual Observation of the Response Variable 532 19.8 Regression Diagnostics 536 19.8.1 Analysis of Residuals 537 19.8.2 Nonlinear Relationships 542 19.8.3 Heteroscedasticity 546 19.8.4 Nonnormally Distributed Error Terms 551 19.8.5 Statistically Dependent Responses 552

Appendix A The Binomial Distribution 559

Appendix B The Standard Normal Distribution 565

Appendix C The 𝝌2-Distribution 567

Appendix D Student’s t-Distribution 569 xiv Contents

Appendix E The Wilcoxon Signed-Rank Test 571

Appendix F The Shapiro–Wilk Test 577

Appendix G Fisher’s F-Distribution 579

Appendix H The Wilcoxon Rank-Sum Test 587

Appendix I The Studentized Range or Q-Distribution 599

Appendix J The Two-Tailed Dunnett Test 607

Appendix K The One-Tailed Dunnett Test 611

Appendix L The Kruskal–Wallis Test 615

Appendix M The Rank Correlation Test 619

Index 621 Preface

This book is the result of a thorough revision of the lecture notes for the course “Statistics for Business and Economics 2” that were developed by Peter Goos at the Faculty of Applied Economics of the University of Antwerp in Belgium. Encouraged by the success of the Dutch version of this book (entitled Verklarende Statistiek: Schatten and Toetsen, published in 2014 by Acco Leuven/Den Haag), we joined forces to create an English version. The new book builds on our first joint work, Statistics with JMP: Graphs, Descriptive Statistics and Probability (Wiley, 2015), which adopts the same philosophy, and uses the same software package, JMP. Hence, it can be regarded as a sequel, but it can also be read as a stand-alone book. In this book, we give a detailed introduction to point estimators, interval estimators, hypothesis tests, analysis of variance, and simple linear regression. Compared with other introductory textbooks on inferential statistics, we cover several additional topics. For example, considerable attention is paid to nonparametric tests, such as the sign test, the signed-rank test, and the Kruskal–Wallis test. In addition, we discuss tests and confidence intervals for the Pearson and Spearman correlation coefficients, introduce the concept of equivalence tests, and include a chapter on the principles of optimal design of experiments. For nonparametric tests, exact p-values are discussed in detail, alongside the better-known approximate p-values. Throughout the book, we discuss different versions of tests and confidence intervals, which includes the construction of confidence intervals for a proportion and approximate p-values for certain tests. The book also incorporates recent insights from the literature, such as a more detailed table with critical values for the Shapiro–Wilk test and a recent table with critical values for the Kruskal–Wallis test. As in our first book, we pay equal attention to mathematical aspects, the interpretation of all the statistical concepts that are introduced, and their practical application. In order to facilitate the understanding of the methods and to appreciate their usefulness, the book contains many examples involving real-life data. To demonstrate the broad applicability of statistics and probability, these examples have been taken from various fields of application, including business, economics, sports, engineering, and the natural sciences. Our motivation in writing this book was twofold. First, we wanted to provide students and teachers with a resource that goes beyond other textbooks of similar scope in its technical and mathematical content. It has become increasingly fashionable for authors and statistics teachers to sweep technicalities and mathematical derivations under the carpet. We decided against this, because we feel that students should be encouraged to apply their mathematical knowledge, and that doing so deepens their understanding of statistical methods. Reading xvi Preface this book obviously requires some knowledge of mathematics. In most countries, students are taught mathematics in secondary or high school and the required mathematical concepts are revisited in introductory mathematics courses at university. Therefore, we are convinced that many university students have a sufficiently strong mathematical background to appreciate and benefit from the more thorough nature of this book. In the various derivations, wehave tried to include all the intermediate steps, in order to keep the book readable. Our second motivation was to ensure that the concepts introduced in the book can be successfully put into practice. To this end, we show how to generate estimates, carry out hypothesis tests, and perform regression analyses using the statistical software package JMP (pronounced “jump”). We chose JMP as supporting software because it is powerful yet easy to use, and suitable for a wide range of statistically oriented courses (including descriptive statistics, hypothesis testing, regression, analysis of variance, design of experiments, reliability, multivariate methods, and statistical and predictive modeling). We believe that introductory courses in statistics should use such software wherever possible. Indeed, we find that, because of the way in which students can easily interact with JMP, it can actually spark enthusiasm for statistics in class. The probability that a student will use statistics in his or her future professional career is far greater if the statistics classes were more pleasurable than painful. In summary, our approach to teaching statistics combines theoretical and mathematical depth, detailed and clear explanations, numerous practical examples, and the use of a user- friendly and yet very powerful statistical package.

Software As mentioned, we use JMP as enabling software. With the purchase of a hard copy of this book, you receive a one-year license for JMP’s Student Edition. The license period starts when you activate your copy of the software using the code included with this book (see the inside front cover). To download JMP’s Student Edition, visit http://www.jmp.com/wiley. For students accessing a digital version of the book, your lecturer may contact Wiley in order to procure unique codes with which to download the free software. For more information about JMP, go to http://www.jmp.com. JMP is available for the Windows and Mac operating systems. This book is based on JMP version 12 for Windows. In our examples, we do not assume any familiarity with JMP: the step-by-step instruc- tions are detailed and accompanied by screenshots. For more explanations and descriptions, www.jmp.com offers a substantial amount of free material, including many video demonstra- tions. In addition, there is a JMP Academic User Community where you can access content, discuss questions, and collaborate with other JMP users worldwide: instructors can share teaching resources and best practices, students can ask questions, and everyone can access the latest resources provided by the JMP Academic Team. To join the community, go to http://community .jmp.com/academic.

Data Files Throughout the book, various data sets are used. We strongly encourage everyone who wants to learn statistics to actively try things out using data. JMP files containing the data sets as Preface xvii well as JMP scripts to reproduce figures, tables, and analyses can be downloaded from the publisher’s companion web site to this book:

www.wiley.com/go/goosandmeintrup/JMP

There, we also provide some additional supporting files.

Peter Goos David Meintrup

Acknowledgments

We consulted plenty of sources during the preparation of this book, and we would like to acknowledge at least the most important ones. A source on nonparametric techniques that we found extremely valuable, given its depth and comprehensive explanations, is the book Nonparametric Statistical Methods by M. Hollander, D.A. Wolfe, and E. Chicken. A more general book that we found very helpful is the Handbook of Parametric and Nonparametric Statistical Procedures by David J. Sheskin. The same applies to Biostatistical Analysis by J.H. Zar. We would like to thank numerous people who have made the publication of this book possible. The first author, Peter Goos, is very grateful to Professor Willy Gochet ofthe University of Leuven, who introduced him to the topics of statistics and probability. Professor Gochet allowed Peter to use his lecture notes as a backbone for his own course material, which later developed into this book. The authors are very grateful for the support and advice offered by several people from the JMP Division of SAS: Brady Brady, Ian Cox, Bradley Jones, Volker Kraft, John Sall, and Mia Stephens. It is Volker who brought the two authors together and encouraged them to work on a series of English books on statistics with JMP (the first book is entitled Statistics with JMP: Graphs, Descriptive Statistics and Probability). A very special word of thanks goes to Ian, whose suggestions substantially improved this book, and to Jose´ Ramirez for generously sharing an example and a data set. The authors would also like to thank Eva Angels, Kris Annaert, Stefan Becuwe, Hilde Bemelmans, Marco Castro, Filip De Baerdemaeker, Hajar Hamidouche, Jer´ emie´ Haumont, Roselinde Kessels, Ida Ruts, Bagus Sartono, Daniel Palhazi Cuervo, Evelien Stoffels, Anja Struyf, Utami Syafitri, Yahri Tillmans, Ellen Vandervieren, Katrien Van Driessen, Kristel Van Rompay, Alan Vazquez Alcocer, Diane Verbiest, Tom Vermeire, Nha Vo-Thanh, Sara Weyns, Peter Willeme,´ and Simone Willis for their detailed comments and constructive suggestions, and for their technical assistance in creating figures and tables. Finally, we thank Liz Wingett, Baljinder Kaur, Heather Kay, Audrey Koh, and Geoffrey D. Palmer at John Wiley & Sons.

Part One Estimators and Tests

Estimating Population Parameters

I don’t know how long I stand there. I don’t believe I’ve ever stood there mourning faithfully in a downpour, but statistically speaking it must have been spitting now and then, there must have been a bit of a drizzle once or twice. (from The Misfortunates, Dimitri Verhulst, pp. 125–126) A major goal in statistics is to make statements about populations or processes. Often, the interest is in specific parameters of the distributions or densities of the populations or processes under study. For instance, researchers in political science want to make statements about the proportion of a population that votes for a certain political party. Industrial engineers want to make statements about the proportion of defective smartphones produced by a production process. Bioscience engineers are interested in comparing the mean amounts of growth resulting from applying two or more different fertilizers. Economists are interested in income inequality and may want to compare the variance in income across different groups. To be able to make such statements, the proportions, means, and variances under study need to be quantified. In statistical jargon, we say that these parameters need to be estimated. It is also important to quantify how reliable each of the estimates is, in order to judge the confidence we can have in any statement we make. This chapter discusses the properties of the most important sample statistics that are used to make statements about population and process means, proportions, and variances.

1.1 Introduction: Estimators Versus Estimates In practice, population parameters such as 𝜇, 𝜎2, 𝜋, and 𝜆 (see our book Statistics with JMP: Graphs, Descriptive Statistics and Probability) are rarely known. For example, if we study the arrival times of the customers of a bank, we know that the number of arrivals per unit of time often follows a Poisson1 distribution. However, we do not know the exact value of the

1 The Poisson distribution is commonly used for random variables representing a certain number of events per unit of time, per unit of length, per unit of volume, and so on. The Poisson distribution has one parameter 𝜆, which is the average number of events per unit of time, per unit of length, per unit of volume, and so on. For more details, see Statistics with JMP: Graphs, Descriptive Statistics and Probability.

Statistics with JMP: Hypothesis Tests, ANOVA and Regression, First Edition. Peter Goos and David Meintrup. © 2016 John Wiley & Sons, Ltd. Published 2016 by John Wiley & Sons, Ltd. Companion Website: http://www.wiley.com/go/goosandmeintrup/JMP 4 Statistics with JMP: Hypothesis Tests, ANOVA and Regression distribution’s parameter 𝜆. One way or another, we therefore need to estimate this parameter. This estimate will be based on a number of measurements or observations, x1, x2, … , xn, that we perform in the bank; in other words, on the sample data we collect. 𝜆 The estimate for the unknown will be a function of the sample values x1, x2, … , xn;for example, the sample mean x. Every researcher who faces the same problem, studying the arrival pattern of customers, will obtain different sample values, and thus a different sample mean and another estimate. The reason for this is that the number of arrivals in the bank in a given time interval is a random variable. We can express this explicitly by using uppercase letters X1, X2, … , Xn for the sample observations. The fact that each researcher obtains another estimate for 𝜆 can also be made more explicit by using a capital letter to denote the sample mean: X. The sample mean is interpreted as a random variable, and then it is called an estimator instead of an estimate. In short, an estimate is always a real number, while an estimator is a random variable the value of which is not yet known. The sample mean is, of course, only one of many possible functions of the sample observations X1, X2, … , Xn, and thus only one of many possible estimators. Obviously, a researcher is not interested in an arbitrary function of the sample observations, but he wants to get a good idea of the unknown parameter. In other words, the researcher wishes to obtain an estimate that, on average, is equal to the unknown parameter, and that, ideally, is guaranteed to be close to the unknown parameter. Statisticians translate these requirements into “the estimator should be unbiased” and “the estimator should have a small variance”. These requirements will be clarified in the next section.

1.2 Estimating a Mean Value The requirements for a good estimator can best be illustrated by means of two simulation studies. The first study simulates data from a normally distributed population, whilethe second one simulates data from an exponentially distributed population.

1.2.1 The Mean of a Normally Distributed Population We first assume that a normally distributed population with mean 𝜇 = 3000 and standard deviation 𝜎 = 100 is studied by 1000 (fictitious) students. The students are unaware of the 𝜇 value and wish to estimate it. To this end, each of these students performs five measurements. A first option to estimate the unknown value 𝜇 is to calculate the sample mean. In this way, we obtain 1000 sample means, shown in the histogram in Figure 1.1, at the top left. The mean of these 1000 sample means is 2998.33, while the standard deviation is 43.38. Another possibility to estimate the unknown 𝜇 is to calculate the median. For a normally distributed population, both the median and the expected value are equal to the parameter 𝜇,so that this makes sense. Based on the samples that the students have gathered, the 1000 medians can also be calculated and displayed in a histogram. The resulting histogram is shown in Figure 1.1, at the top right2. The attentive reader will notice immediately that the second histogram is

2 Outputs as in Figures 1.1 and 1.2 can be created in JMP with the “Distribution”optioninthe“Analyze” menu. Estimating Population Parameters 5

Figure 1.1 Histograms and descriptive statistics for 1000 sample means and medians calculated based on samples of five observations from a normally distributed population with mean 3000 and standard deviation 100. 6 Statistics with JMP: Hypothesis Tests, ANOVA and Regression just a bit wider than the first. Among other things, this is reflected by the fact that thestandard deviation of the 1000 medians is 53.43. The mean of the 1000 medians is equal to 2999.08. In Figure 1.1, it can also be seen that the minimum (2841.78) and the first quartile (2962.22) of the sample medians are smaller than the minimum (2867.56) and the first quartile (2969.25) of the sample means. Also, the maximum (3161.64) and the third quartile (3033.51) of the sample medians are greater than the maximum (3140.35) and the third quartile (3027.80) of the sample means. This suggests that the sample medians are, in general, further away from the population mean 𝜇 = 3000 than the sample means. It is striking that both the mean of the 1000 sample means (2998.33) and that of the 1000 medians (2999.08) are very close to 3000. If the number of samples is raised significantly (theoretically, an infinite number of samples could be taken), the mean of the sample means and that of the sample medians will converge to the unknown 𝜇 = 3000. Therefore, both the sample mean and the sample median are called unbiased estimators of the mean of a normally distributed population. The fact that the range, the interquartile range, the standard deviation, and the variance of the 1000 sample means are smaller than those of the 1000 sample medians means that the sample mean is a more reliable estimator of the unknown population mean than the sample median. The larger variance of the medians indicates that the medians are generally further away from 𝜇 = 3000 than the sample means. In short, a researcher should have more confidence in the sample mean because it is usually closer to the unknown 𝜇. In such a case, we say that one estimator (here, the sample mean) is more efficient or precise than the other (here, the median).

1.2.2 The Mean of an Exponentially Distributed Population We now investigate an exponentially distributed population with parameter 𝜆 = 1∕100. The “unknown” population mean is therefore 𝜇 = 1∕𝜆 = 100 (see Statistics with JMP: Graphs, Descriptive Statistics and Probability). Each of the 1000 fictitious students performs five measurements. A first option to estimate the unknown value 𝜇 is again to calculate the sample mean. A histogram of the 1000 sample means is shown in Figure 1.2, at the top left. The mean of these 1000 sample means is 99.2417, while the standard deviation is 44.10. Based on the samples that the students have gathered, the 1000 medians can also be calculated and displayed in a histogram. This histogram is shown in Figure 1.2, at the top right. The mean of the 1000 medians is only 77.0114. These calculations indicate that the population mean 𝜇 = 1∕𝜆 = 100 can be approximated fairly well by using the sample means, with a mean of 99.2417. This is not the case for the medians, the mean value of which is far away from 𝜇. This remains the case if the number of samples is increased. In this example, for an exponentially distributed population, the median is not an unbiased but a biased estimator of the population mean. In addition, Figure 1.2 also shows that the standard deviation of the sample medians (46.13) is greater than that of the sample means (44.10).

1.3 Criteria for Estimators Key properties of estimators are their expected values and their variances. These statistics are related to the concepts of bias and efficiency, respectively. Estimating Population Parameters 7

Figure 1.2 Histograms and descriptive statistics for 1000 sample means and sample medians calculated based on samples of five observations from an exponentially distributed population with parameter 𝜆 = 1∕100. 8 Statistics with JMP: Hypothesis Tests, ANOVA and Regression

1.3.1 Unbiased Estimators An ideal estimator that always produces the exact value of an unknown population parameter does not exist. As illustrated in the above example, some estimators, namely unbiased estimators, are on average equal to the unknown population parameter, while others systematically under- or overestimate the parameter. The latter is an undesirable result for a researcher. For- mally, the definition of an unbiased estimator 𝜃̂ for an unknown population parameter 𝜃 is as follows:

Definition 1.3.1 An estimator 𝜃̂ of a population parameter 𝜃 is unbiased if

E(𝜃̂) = 𝜃.

The bias of an estimator is the absolute difference V(𝜃̂) = |E(𝜃̂) − 𝜃|. An unbiased estimator has a bias of zero. For an unbiased estimator, the expected value is exactly equal to the population parameter. The histograms for the sample means on the left-hand sides of Figures 1.1 and 1.2 show that, once sample data is being used, the estimate will be close to the unknown population parameter, but not exactly equal to it. So, for any particular sample, even unbiased estimators result in estimates that differ from the population parameter that is being estimated. Note that here the symbol 𝜃̂ is used to denote an estimator of the unknown population parameter 𝜃. As usual in statistics, we use Greek letters to denote unknown population parameters such as population means, population proportions, or population variances. If we want to estimate an unknown population parameter, we use an estimator, which is a synonym for an estimation method. In general in statistics, we indicate this using the symbol 𝜃̂ (pronounced “theta hat”). We will mainly focus on three specific estimators, namely the sample mean, the sample proportion, and the sample variance. For historical reasons, the symbols X, P̂ , and S2 are used for these three estimators instead of 𝜇̂, 𝜋̂, and 𝜎̂ 2. The sample mean X is always an unbiased estimator of the population∑ mean (this is proven n 𝛼 in Theorem 1.5.1). Actually,∑ this applies to all linear functions Y = i=1 iXi of sample n 𝛼 observations for which i=1 i = 1, and the sample mean is a special case of such a linear 𝛼 combination, where each i = 1∕n:

∑n 1 1 1 1 1 X = X = (X + X + ⋯ + X ) = X + X + ⋯ + X . n i n 1 2 n n 1 n 2 n n i=1 ∑ n 𝛼 It can be shown that, of all linear functions of X1, X2, … , Xn,forwhich i=1 i = 1thesample mean has the smallest variance3. In other words, the sample mean will usually provide an estimate that is closer to the population mean than any other linear function Y of X1, X2, … , Xn. In Theorem 1.7.1, we prove that the sample variance

∑n 1 S2 = (X − X)2 n − 1 i i=1

3 Therefore, the sample mean is called the “best linear unbiased estimator”, abbreviated as “BLUE”.