Level C Analysis—Height Vs

In This Section → An Introductory Example— → Example 5: Estimating the Density Obesity in America of the Earth—A Classical Study

→ The Investigatory Process → Example 6: Linear Regression at Level C Analysis—Height vs. Forearm Formulating Questions Length

Collecting Data—Types → Example 7: Comparing of Statistical Studies Mathematics Scores— An Observational Study Sample Surveys Experiments → Example 8: Observational Study— Toward Establishing Causation Observational Studies → The Role of Probability in Statistics Analyzing Data → Summary of Level C → Example 1: The Sampling Distribution of a Sample Proportion

→ Example 2: The Sampling Distribution of a Sample Mean Interpreting Results Generalizing from Samples Generalizing from Experiments

→ Example 3: A Survey of Music Preferences

→ Example 4: An Experiment on the Effects of Light on the Growth of Radish Seedlings

60 Level C evel C is designed to build on the foundation At Level C, students develop additional strategies for developed in Levels A and B. In particular, producing, interpreting, and analyzing data to help LLevels A and B introduced students to statistics answer questions of interest. In general, students as an investigatory process, the importance of using should be able to formulate questions that can be data to answer appropriately framed questions, types answered with data; devise a reasonable plan for col- of variables (categorical versus numerical), graphical lecting appropriate data through observation, sampling, displays (including bar graph, dotplot, stemplot, his- or experimentation; draw conclusions and use data to togram, boxplot, and scatterplot), tabular displays support these conclusions; and understand the role (including two-way frequency tables for categorical random variation plays in the inference process. data and both ungrouped and grouped frequency/ Speciﬁ cally, Level C recommendations include: relative frequency tables for numerical data), and numerical summaries (including counts, proportions, I. Formulate Questions mean, median, range, quartiles, interquartile range, → Students should be able to formulate questions MAD, and QCR). and determine how data can be collected and Additionally, Levels A and B covered common study analyzed to provide an answer. designs (including census, simple random sample, and II. Collect Data randomized designs for experiments), the process of drawing conclusions from data, and the role of prob- → Students should understand what constitutes ability in statistical investigations. good practice in conducting a sample survey. → Students should understand what constitutes At Level C, all of these ideas are revisited, but the good practice in conducting an experiment. types of studies emphasized are of a deeper statistical → Students should understand what constitutes nature. Statistical studies at this level require students good practice in conducting an observational to draw on basic concepts from earlier work, extend study. the concepts to cover a wider scope of investigatory issues, and develop a deeper understanding of → Students should be able to design and inferential reasoning and its connection to probability. implement a data collection plan for Students also should have increased ability to explain statistical studies, including observational statistical reasoning to others. studies, sample surveys, and simple comparative experiments.

61 III. Analyze Data An Introductory Example–Obesity → Students should be able to identify in America appropriate ways to summarize numerical or Data and the stories that surround the data must be categorical data using tables, graphical of interest to students! It is important to remember displays, and numerical summary statistics. this when teaching data analysis. It is also important → Students should understand how sampling to choose data and stories that have enough depth to distributions (developed through simulation) demonstrate the need for statistical thinking. The fol- are used to describe the sample-to-sample lowing example illustrates this. variability of sample statistics. Students are interested in issues that affect their lives, → Students should be able to recognize and issues of health often fall into that category. News association between two categorical variables. items are an excellent place to look for stories of cur- → Students should be able to recognize when rent interest, including items on health. One health- the relationship between two numerical related topic making lots of news lately is obesity. The variables is reasonably linear, know that following paragraph relates to a news story that is rich Pearson’s correlation coeffi cient is a measure of enough to provide a context for many of the statistical the strength of the linear relationship between topics to be covered at Level C. two numerical variables, and understand the least squares criterion in line fi tting. A newspaper article that appeared in 2004 begins with the following lines: “Ask anyone: Americans are get- IV. Interpret Results ting fatter and fatter. Advertising campaigns say they → Students should understand the meaning of are. So do federal offi cials and the scientists they rely statistical signifi cance and the difference on. … In 1991, 23% of Americans fell into the obese between statistical signifi cance and practical category; now 31% do, a more than 30% increase. But signifi cance. Dr. Jeffrey Friedman, an obesity researcher at Rock- → Students should understand the role of p-values efeller University, argues that contrary to popular in determining statistical signifi cance. opinion, national data do not show Americans growing uniformly fatter. Instead, he says, the statistics → Students should be able to interpret the margin demonstrate clearly that while the very fat are getting of error associated with an estimate of a fatter, thinner people have remained pretty much the population characteristic. same. …The average weight of the population has in-

62 creased by just seven to 10 pounds.” The discussion in Statistics, National Health and Nutrition the article refers to adults. Examination Survey 2002. This is a survey of approximately 5,800 residents The following are suggested questions to explore with of the United States. Although the survey students who have a Level B background in statistics, design is more complicated than a simple but are moving on to Level C. random sample, the margin of error calculated → Sketch a histogram showing what you think as if it were a simple random sample is a a distribution of weights of American adults reasonable approximation. What is an might have looked like in 1991. Adjust the approximate margin of error associated with sketch to show what the distribution of weights the 31% estimate of obesity for 2004? might have looked like in 2002, the year of the Interpret this margin of error for a newspaper reported study. Before making your sketches, reader who never studied statistics. think about the shape, center, and spread of For the curious, information about how obesity is de- your distributions. Will the distribution be ﬁ ned can be found at www.amstat.org/education/gaise/3. skewed or symmetric? Will the median be smaller than, larger than, or about the same In answering these questions, students at Level C size as the mean? Will the spread increase as should realize that a distribution of weights is going you move from the 1991 distribution to the to be skewed toward the larger values. This generally 2002 distribution? produces a situation in which the mean is larger than → Which sounds more newsworthy: “Obesity has the median. Because 8% shifted over the obesity line increased by more than 30%” or “On the aver- between 1991 and 2002, but the average weight (or age, the weight of Americans has increased by center) did not shift very much, the upper tail of the fewer than 10 pounds”? Explain your reasoning. distribution must have gotten “fatter,” indicating a → The title of the article is The Fat Epidemic: He larger spread for the 2002 data. Students will have Says It’s an Illusion. [See New York Times, June 8, a variety of interesting answers for the second and 2004, or CHANCE, Vol. 17., No. 4, Fall 2004, third questions. The role of the teacher is to help p. 3 for the complete article.] Do you think this students understand whether their answers are sup- is a fair title? Explain your reasoning. ported by the facts. The last question gets students → The data on which the percentages are based thinking about an important estimation concept come from the National Center for Health studied at Level C.

63 The Investigatory Process at Level C These two types of statistical investigations have Because Level C revisits many of the same topics ad- some common elements—each requires randomiza- dressed at Levels A and B, but at a deeper and more tion for both purposes of reducing bias and building sophisticated level, we begin by describing how the a foundation for statistical inference and each makes investigatory process looks at Level C. This general use of the common inference mechanisms of margin discussion is followed by several examples. of error in estimation and p-value in hypothesis testing (both to be explained later). But these two types of investigations have very different objectives and Formulating Questions requirements. Sample surveys are used to estimate or As stated at the beginning of Level A, data are more make decisions about characteristics (parameters) of than just numbers. Students need to understand the populations. A well-defi ned, fi xed population is the types of questions that can be answered with data. main ingredient of such a study. Experiments are used For example, the question “Is the overall health of to estimate or compare the effects of different experi- high-school students declining in this country?” is mental conditions (treatments), and require well-de- too big a question to answer with a statistical in- fi ned treatments and experimental units on which to vestigation (or even many statistical investigations). study those treatments. Certain aspects of the health of students, however, can be investigated by formulating more specifi c Estimating the proportion of residents of a city that questions, such as “What is the rate of obesity would support an increase in taxes for education re- among high-school students?”; “What is the average quires a sample survey. If the selection of residents daily caloric intake for high-school seniors?”; “Is a is random, then the results from the sample can be three-day-a-week exercise regimen enough to main- extended to represent the population from which the tain heart rate and weight within acceptable limits?” sample was selected. A measure of sampling error Question formulation, then, becomes the starting (margin of error) can be calculated to ascertain how point for a statistical investigation. far the estimate is likely to be from the true value. Testing to see if a new medication to improve breath- Collecting Data—Types of Statistical Studies ing for asthma patients produces greater lung capacity Most questions that can be answered through data than a standard medication requires an experiment in collection and interpretation require data from a which a group of patients who have consented to par- designed study, either a sample survey or an experiment. ticipate in the study are randomly assigned to either

64 the new or the standard medication. With this type of design), and how to clearly and precisely state conclu- randomized comparative design, an investigator can sions for these designed studies. Key elements of the determine, with a measured degree of uncertainty, design and implementation of data collection plans for whether the new medication caused an improvement these types of studies follow. in lung capacity. Randomized experiments are, in fact, the only type of statistical study capable of establish- Sample Surveys ing cause and effect relationships. Any generalization Students should understand that obtaining good re- extends only to the types of units used in the experi- sults from a sample survey depends on four basic fea- ment, however, as the experimental units are not usu- tures: the population, the sample, the randomization ally randomly sampled from a larger population. To process that connects the two, and the accuracy of the generalize to a larger class of experimental units, more measurements made on the sampled elements. For ex- experiments would have to be conducted. That is one ample, to investigate a question on health of students, reason why replication is a hallmark of good science. a survey might be planned for a high school. What is the population to be investigated? Is it all the students Studies that have no random selection of sampling in the school (which changes on a daily basis)? Per- units or random assignment of treatments to ex- haps the questions of interest involve only juniors and perimental units are called observational studies in this seniors. Once the population is deﬁ ned as precisely as document. A study of how many students in your possible, one must determine an appropriate sample high school have asthma and how this breaks down size and a method for randomly selecting a sample of among gender and age groups would be of this type. that size. Is there, for example, a list of students who Observational studies are not amenable to statistical can then be numbered for random selection? Once inference in the usual sense of the term, but they the sampled students are found, what questions will can provide valuable insight into the distribution of be asked? Are the questions fair and unbiased (as far measured values and the types of associations among as possible)? Can or will the students actually answer variables that might be expected. them accurately? At Level C, students should understand the key When a sample of the population is utilized, errors features of both sample surveys and experimental may occur for several reasons, including: designs, including how to set up simple versions of both types of investigations, how to analyze the data → the sampling procedure is biased appropriately (as the correct analysis is related to the → the sample was selected from the wrong population

65 → some of the units selected to be in the sample experiments with human subjects, the people involved were unable (or unwilling) to participate are often volunteers who have to sign an agreement → the questions were poorly written stating they are willing to participate in the experi- → the responses were ambiguous mental study. In experiments with agricultural crops, the experimental units are the ﬁ eld plots that happen “ When These types of errors should be considered carefully to be available. In an industrial experiment on process randomness is before the study begins so plans can be made to reduce improvement, the units may be the production lines in their chance of occurring as much as possible. One incorporated operation during a given week. way to resolve the bias in the sampling procedure is to into the incorporate randomness into the selection process. As in a sample survey, replicating an experiment will sampling produce different results. Once again, random assign- Two samples of size 50 from the same population ment of experimental units to treatments (or vice versa) procedure, of students will most likely not give the same result allows the use of probability to predict the behavior in probability on, say, the proportion of students who eat a healthy the resulting values of summary statistics from a large provides a way breakfast. This variation from sample to sample is number of replications of the experiment. Randomiza- to describe called sampling variability. When randomness is in- tion in experiments is important for another reason. the ‘long-run’ corporated into the sampling procedure, probability Suppose a researcher decides to assign treatment A provides a way to describe the “long-run” behavior of behavior only to patients over the age of 60 and treatment B this sampling variability. only to patients under the age of 50. If the treatment of sampling responses differ, it is impossible to tell whether the Experiments variability. difference is due to the treatments or the ages of the ” At Level C, students should understand that obtaining patients. (This kind of bias in experiments and other good results from an experiment depends upon four statistical studies is called confounding.) The randomiza- basic features: well-deﬁ ned treatments, appropriate tion process, if properly done, will usually balance experimental units to which these treatments can be treatment groups so this type of bias is minimized. assigned, a sound randomization process for assigning treatments to experimental units, and accurate Observational Studies measurements of the results of the experiment. Ex- At Level C, students should understand that observa- perimental units generally are not randomly selected tional studies are useful for suggesting patterns in data from a population of possible units. Rather, they are and relationships between variables, but do not provide a the ones that happen to be available for the study. In strong foundation for estimating population parameters

66 or establishing differences among treatments. Asking population mean for numerical data. The appropriate the students in one classroom whether they eat a sample statistics used to estimate these parameters are healthy breakfast is not going to help you establish the the sample proportion and the sample mean, respec- proportion of healthy breakfast-eaters in the school, tively. At Level C, the sample-to-sample variability, as the students in one particular classroom may not be as described by the sampling distribution for each of representative of the students in the school. Random these two statistics, is addressed in more depth. sampling is the only way to be conﬁ dent of a represen- Exploring how the information provided by a sam- tative sample for statistical purposes. Similarly, feed- pling distribution is used for generalizing from a ing your cats Diet A and your neighbor’s cats Diet B is sample to the larger population enables students at not going to allow you to claim that one diet is better Level C to draw more sophisticated conclusions from than the other in terms of weight control, because statistical studies. At Level C, it is recommended that there was no random assignment of experimental the sampling distributions of a sample proportion and units (cats) to treatments (diets). As a consequence, of a sample mean be developed through simulation. confounding may result. Studies of the type suggested More formal treatment of sampling distributions can above are merely observational; they may suggest pat- be left to AP Statistics and college-level introductory terns and relationships, but they are not a reliable basis statistics courses. for statistical inference. Because the sampling distribution of a sample statis- Analyzing Data tic is a topic with which many teachers may not be familiar, several examples are included here to show When analyzing data from well-designed sample sur- how simulation can be used to obtain an approximate veys, students at Level C should understand that an sampling distribution for a sample proportion and for appropriate analysis is one that can lead to justiﬁ able a sample mean. inferential statements about population parameters based on estimates from sample data. The ability to Example 1: The Sampling Distribution draw conclusions about the population using informa- of a Sample Proportion tion from a sample depends on information provided Properties of the sampling distribution for a sample by the sampling distribution of the sample statistic proportion can be illustrated by simulating the process being used to summarize the sample data. At Level of selecting a random sample from a population using C, the two most common parameters of interest are random digits as a device to model various populations. the population proportion for categorical data and the

67 For example, suppose a population is assumed to have resulted in the empirical distribution shown in Figure 60% “successes” (p = .6) and we are to take a random 25. This empirical distribution is an approximation to sample of n = 40 cases from this population. How far the true sampling distribution of the sample propor- can we expect the sample proportion of successes to tion for samples of size 40 from a population in which deviate from the true population value of .60? This the actual proportion is .60. can be answered by determining an empirical sampling distribution for the sample proportion. Sample proportions 30 One way to model a population with 60% successes (and 40% failures) is to utilize the 10 digits 0, 1,…, 9. 25 Label six of the 10 digits as “success” and the other 20 four as “failures.” To simulate selecting a sample of Count 15 size 40 from this population, randomly select 40 ran- 10 dom digits (with replacement). Record the number of successes out of the 40 digits selected and convert this 5 count to the proportion of successes in the sample. Note that: 0.4 0.5 0.6 0.7 0.8 Proportion Proportion of Successes in the Sample Figure 25: Histogram of sample proportions Number of Successes in the Sample Summarizing the above distribution based on its = Sample Size shape, center, and spread, one can state that this empirical sampling distribution has a mound shape (ap- Repeating this process a large number of times, and proximately normal). Because the mean and standard determining the proportion of successes for each deviation of the 200 sample proportions are .59 and .08, sample, illustrates the idea of the sample-to-sample respectively, the empirical distribution shown in Figure variability in the sample proportion. 25 has a mean of .59 and a standard deviation of .08. Simulating the selection of 200 random samples of By studying this empirical sampling distribution, and size 40 from a population with 60% successes and de- others that can be generated in the same way, students termining the proportion of success for each sample will see patterns emerge. For example, students will observe that, when the sample size is reasonably large

68 (and the population proportion of successes is not too 30 random digits are selected (with replacement) and near the extremes of 0 or 1), the shapes of the result- the sample mean is computed. This simulates sam- ing empirical sampling distributions are approximately pling from a population that has a uniform distribu- normal. Each of the empirical sampling distributions tion with equal numbers of 0s, 1s, 2s,…, 9s. Note that should be centered near the value of p, the population this population of numerical values has a mean, μ, of proportion of successes, and the standard deviation 4.5 and a standard deviation, σ, of 2.9. for each distribution should be close to: Sample means p(1− p) 35 n 30 Note that in Example 1, the mean of the empirical dis- 25 tribution is .59, which is close to .6, and the standard 20 deviation is .08, which is close to: 15 .6(.4) 10 ≈.0775 Frequency of Mean 40 A follow-up analysis of these empirical sampling dis- 5 tributions can show students that about 95% of the sample proportions lie within a distance of: 3.0 3.5 4.0 4.5 5.0 5.5 6.0 Mean .6(.4) 2 ≈ 0.155 40 Figure 26: Histogram of sample means from the true value of p. This distance is called the The empirical sampling distribution shown in Figure margin of error. 26 can be described as approximately normal with a mean of 4.46 (the mean of the 200 sample means from Example 2: The Sampling Distribution of a Sample Mean the simulation) and a standard deviation of 0.5 (the standard deviation of the 200 sample means). Properties of the sampling distribution for a sample mean can be illustrated in a way similar to that used By studying this empirical sampling distribution, and for proportions in Example 1. Figure 26 shows the others that can be generated in similar ways, students distribution of the sample mean when 200 samples of will see patterns emerge. For example, students will

69 observe that, when the sample size is reasonably large, about the population parameter being estimated. As the shapes of the empirical sampling distributions are described in the previous section, knowledge of the approximately normal. Each of the empirical sam- sampling distribution for a statistic, like a sample pro- pling distributions should be centered near the value portion or sample mean, leads to a margin of error of μ, the population mean, and the standard deviation that provides information about the maximum likely for each distribution should be close to: distance between a sample estimate and the popula- σ tion parameter being estimated. Another way to state this key concept of inference is that an estimator plus n or minus the margin of error produces an interval of plausible values for the population parameter. Any one Note that in Example 2, the mean of the empirical of these plausible values could have produced the ob- sampling distribution is 4.46, which is close to μ = 4.5, served sample result as a reasonably likely outcome. and the standard deviation (0.5) is close to: Generalizing from Experiments σ n ==2.9 30 0.53 Do the effects of the treatments differ? In analyzing The margin of error in estimating a population mean experimental data, this is one of the ﬁ rst questions using the sample mean from a single random sample asked. This question of difference is generally posed is approximately: in terms of differences between the centers of the data σ distributions (although it could be posed as a differ- 2 ence between the 90th percentiles or any other mea- n sure of location in a distribution). Because the mean The sample mean should be within this distance of is the most commonly used statistic for measuring the the true population mean about 95% of the time in center of a distribution, this question of differences repeated random sampling. is generally posed as a question about a difference in means. The analysis of experimental data, then, usu- Interpreting Results ally involves a comparison of means. Generalizing from Samples Unlike sample surveys, experiments do not depend on random samples from a ﬁ xed population. Instead, The key to statistical inference is the sampling distribu- they require random assignment of treatments to pre- tion of the sample statistic, which provides information selected experimental units. The key question, then,

70 is: “Could the observed difference in treatment means 2. Which of the following types of music do you like most? be due to the random assignment (chance) alone, or Select only one. can it be attributed to the treatments administered?” Country Rap/Hip Hop Rock The following examples are designed to illustrate and In order to be able to generalize to all students at the further illuminate the important concepts at Level C school, a representative sample of students from the by carefully considering the four phases of a statistical school is needed. This could be accomplished by se- analysis—question, design, analysis, interpretation— lecting a simple random sample of 50 students from in a variety of contexts. the school. The results can then be generalized to the Example 3: A Survey of Music Preferences school (but not beyond), and the Level C discussion will center on basic principles of generalization—or A survey of student music preferences was introduced statistical inference. at Level A, where the analysis consisted of making counts of student responses and displaying the data in A Level C analysis begins with a two-way table of a bar graph. At Level B, the analysis was expanded to counts that summarizes the data on two of the ques- consider relative frequencies of preferences and cross- tions: “Do you like rock music?” and “Do you like classiﬁ ed responses for two types of music displayed rap music?” The table provides a way to separately ex- in a two-way table. Suppose the survey included the amine the responses to each question and to explore following questions: possible connections (association) between the two categorical variables. Suppose the survey of 50 stu- 1. What kinds of music do you like? dents resulted in the data summarized in Table 11. Do you like country music? As demonstrated at Level B, there are a variety of ways Yes or No to interpret data summarized in a two-way table, such as Table 11. Some examples based on all 50 students in Do you like rap music? the survey include: Yes or No → 25 of the 50 students (50%) liked both rap and Do you like rock music? rock music. → 29 of the 50 students (58%) liked rap music. Yes or No → 19 of the 50 students (38%) did not like rock music.

71 Table 11: Two-Way Frequency Table The sampling distribution of proportions so generated will be similar to the one below. Like Rock Music? Sample proportions Ye s N o Row Totals Like Rap Yes25 429 Music? No 6 15 21 Column Totals 31 19 50

One type of statistical inference relates to conjectures (hypotheses) made before the data were collected. Suppose a student says “I think more than 50% of the students in the school like rap music.” Because 0.30 0.40 0.50 0.60 0.70 58% of the students in the sample liked rap music Proportion (which is more than 50%), there is evidence to sup- Movable line is at 0.58 port the student’s claim. However, because we have only a sample of 50 students, it is possible that 50% of Figure 27: Dotplot of sample proportions from a hypo- thetical population in which 50% like rap music all students like rap (in which case, the student’s claim is not correct), but the variation due to random sam- Based on this simulation, a sample proportion greater pling might produce 58% (or even more) who like rap. than or equal to the observed .58 occurred 12 times The statistical question, then, is whether the sample out of 100 just by chance variation alone when the result of 58% is reasonable from the variation we ex- actual population proportion is .50. This suggests the pect to occur when selecting a random sample from a result of .58 is not a very unusual occurrence when population with 50% successes. sampling from a population with .50 as the “true” One way to arrive at an answer is to set up a hypo- proportion of students who like rap music. So a popu- thetical population that has 50% successes (such as lation value of .50 is plausible based on what was ob- even and odd digits produced by a random number served in the sample, and the evidence in support of generator) and repeatedly take samples of size 50 from the student’s claim is not very strong. The fraction of it, each time recording the proportion of even digits. times the observed result is matched or exceeded (.12 in this investigation) is called the approximate

72 p-value. The p-value represents the chance of observ- p-value is .01, and it is not likely that a population in ing the result observed in the sample, or a result more which 40% of the students like rap music would have extreme, when the hypothesized value is in fact cor- produced a sample proportion of 58% in a random rect. A small p-value would have supported the stu- sample of size 50. This p-value provides very strong dent’s claim, because this would have indicated that if evidence in support of the student’s claim that more the population proportion was .50, it would have been than 40% of the students in the entire school like rap very unlikely that a sample proportion of .58 would music. have been observed. Another way of stating the above is that .5 is a plausible Suppose another student hypothesized that more value for the true population proportion, based on the than 40% of the students in the school like rap music. sample evidence, but .4 is not. A set of plausible values To test this student’s claim, samples of size 50 must can be found by using the margin of error introduced now be repeatedly selected from a population that has in Example 1. As explained previously, the margin of 40% successes. Figure 28 shows the results of one such error for a sample proportion is approximately: simulation. The observed result of .58 was reached p(1 − p) only one time out of 100, and no samples produced 2 n a proportion greater than .58. Thus, the approximate Sample proportions However, in this problem, the true value of p is unknown. Our sample proportion ( pˆ = .58 ) is our “best estimate” for what p might be, so the margin of error can be estimated to be: pˆ (1 − ˆp) .58(.42) 2 = 2 ≈ .14 n 50

Thus, any proportion between .58 − .14 = .44 and 0.20 0.30 0.40 0.50 0.60 .58 + .14 = .72 can be considered a plausible value Proportion for the true proportion of students at the school who Movable line is at 0.58 like rap music. Notice that .5 is well within this in- Figure 28: Dotplot of sample proportions from a hypo- terval, but .4 is not. thetical population in which 40% like rap music

73 Another type of question that could be asked about To simulate this situation, we create a population of the students’ music preferences is of the form “Do 29 1s (those who like rap) and 21 0s (those who do those who like rock music also tend to like rap mu- not like rap) and mix them together. Then, we select sic?” In other words, is there an association between 31 (representing those who like rock) at random and liking rock music and liking rap music? The same data see how many 1s (those who like rap) we get. It is this from the random sample of 50 students can be used to entry that goes into the (yes, yes) cell of the table, and answer this question. from that data the difference in proportions can be According to Table 11, a total of 31 students in the calculated. Repeating the process 100 times produces survey like rock music. Among those students, the a simulated sampling distribution for the difference proportion who also like rap music is 25/31 = .81. between the two proportions, as shown in Figure 29. Among the 19 students who do not like rock music, 4/19 = .21 is the proportion who like rap music. The large difference between these two proportions (.60) Differences between proportions suggests there may be a strong association between liking rock music and liking rap music. But could this association simply be due to chance (a consequence only of the random sampling)? If there were no association between the two groups, then the 31 students who like rock would behave as a random selection from the 50 in the sample. We would expect the proportion who like rap among these 31 students to be close to the proportion who like rap among the 19 students who don’t like rock. Essential- ly, this means that if there is no association, we expect the difference between these two proportions to be approximately 0. Because the difference in our survey -0.4 -0.2 0.0 0.2 0.4 0.6 is .6, this suggests that there is an association. Can the Difference difference, .6, be explained by the random variation Movable line is at 0.60 we expect when selecting a random sample? Figure 29: Dotplot showing simulated sampling distribution

74 The observed difference in proportions from the sample data, .6, was never reached in 100 trials, indicating that the observed difference cannot be attributed to chance alone. Thus, there is convincing evidence of a real association between liking rock music and liking rap music. Seeds Staples Example 4: An Experiment on the Effects of Light on the Growth of Radish Seedlings

What is the effect of different durations of light and Figure 30: Seed experiment dark on the growth of radish seedlings? This question was posed to a class of biology students who Although three growth chambers would be sufﬁ cient then set about designing and carrying out an experi- to examine the three treatments, this class made four ment to investigate the question. All possible relative growth chambers, with one designated for the 24 durations of light to dark cannot possibly be investi- hours of light treatment, one for the 12 hours of light gated in one experiment, so the students decided to and 12 hours of darkness treatment, and two for the focus the question on three treatments: 24 hours of 24 hours of darkness treatment. One hundred twenty light, 12 hours of light and 12 hours of darkness, and seeds were available for the study. Thirty of the seeds 24 hours of darkness. This covers the extreme cases were chosen at random and placed along the stapled and one in the middle. seam of the 24 hours of light bag. Thirty seeds were then chosen at random from the remaining 90 seeds With the help of a teacher, the class decided to use and placed in the 12 hours of light and 12 hours of plastic bags as growth chambers. The plastic bags darkness bag. Finally, 30 of the remaining 60 seeds would permit the students to observe and measure were chosen at random and placed in one of the 24 the germination of the seeds without disturbing them. hours of darkness bags. The ﬁ nal 30 seeds were placed Two layers of moist paper towel were put into a dis- in the other 24 hours of darkness bag. After three posable plastic bag, with a line stapled about 1/3 of days, the lengths of radish seedlings for the germinat- the way from the bottom of the bag (see Figure 30) to ing seeds were measured and recorded. These data are hold the paper towel in place and to provide a seam to provided in Table 12; the measurements are in milli- hold the radish seeds.

75 Table 12: Lengths of Radish Seedlings

Treatment 1 Treatment 2 Treatment 3 Treatment 1 Treatment 2 Treatment 3 24 light 12 light, 12 24 dark 24 light 12 light, 12 24 dark dark dark 2 3 52010171530 3 4 52010201530 5 5 82210201530 5 9 82410201531 510 82510201533 510 82510201535 510102510211635 710102510212035 710102514222035 711102615222035 813102915232035 815113020252036 815143021252037 915143021272038 20 40 meters. Notice that not all of the seeds in each group mm and two at 21 mm) in the Treatment 1 (24 hours germinated. of light) data. Otherwise, the distributions are fairly A good ﬁ rst step in the analyses of numerical data symmetric, which is good for statistical inference. such as these is to make graphs to look for patterns In Figure 31, Treatment 1 is 24 hours of light; treat- and any unusual departures from the patterns. Box- ment 2 is 12 hours of light and 12 of darkness; treat- plots are ideal for comparing data from more than ment 3 is 24 hours of darkness. one treatment, as you can see in Figure 31. Both the The summary statistics for these data are shown in centers and the spreads increase as the amount of Table 13. darkness increases. There are three outliers (one at 20

76 Radish seedling lengths Table 13: Treatment Summary Statistics

Treat- n Mean Median Std. Dev.

3 ment 1 28 9.64 9.5 5.03 2 28 15.82 16.0 6.76 2 3 58 21.86 20.0 9.75 “ Experiments Treatment differences large enough to rule out chance variation as a possible are designed explanation for the observed difference? to compare 1 The Treatment 2 mean is 6.2 mm larger than the treatment Treatment 1 mean. If there is no real difference be- effects, usually 0 5 10 15 20 25 30 35 40 45 tween the two treatments in terms of their effect on by comparing Length (mm) seedling growth, then the observed difference must means. be due to the random assignment of seeds to the ” Figure 31: Boxplot showing growth under different bags; that is, one bag was simply lucky enough to get conditions a preponderance of good and lively seeds. But, if a dif- Experiments are designed to compare treatment ference this large (6.2 mm) is likely to be the result of effects, usually by comparing means. The original randomization alone, then we should see differences question on the effect of different periods of light of this magnitude quite often if we repeatedly re- and dark on the growth of radish seedlings might be randomize the measurements and calculate a new dif- turned into two questions about treatment means. Is ference in observed means. This, however, is not the there evidence that the 12 hours of light and 12 hours case, as one can see from Figure 32. This dotplot was of dark (Treatment 2) group has a signiﬁ cantly higher produced by mixing the growth measurements from mean than the 24 hours of light (Treatment 1) group? Treatments 1 and 2 together, randomly splitting them Is there evidence that the 24 hours of dark (Treatment into two groups of 28 measurements, recording the 3) group has a signiﬁ cantly higher mean than the 12 difference in means for the two groups, and repeating hours of light and 12 hours of dark (Treatment 2) the process 200 times. group? Based on the boxplots and the summary statistics, it is clear that the sample means differ. Are these The observed difference of 6.2 mm was exceeded only one time in 200 trials, for an approximate p-value of

77 Differences of means Differences of means

-6 -4 -2 02468 Difference -6 -4 -2 0 2 4 6 8 Movable line is at 6.2 Difference Figure 32: Dotplot showing differences of means Movable line is at 6.0 1/200. This is very small, and gives extremely strong Figure 33: Dotplot showing differences of means evidence to support the hypothesis that there is a sta- signment of seeds to the bags. This gives us convinc- tistically signifi cant difference between the means for ing evidence of a treatment effect—the more hours Treatments 1 and 2. The observed difference of 6.2 mm of darkness, the greater the growth of the seedling, at is very unlikely to be due simply to chance variation. least for these three periods of light versus darkness. In a comparison of the means for Treatments 2 and 3, Students should be encouraged to delve more deeply the same procedure is used, except that the combined into the interpretation, relating it to what is known measurements are split into groups of 28 and 58 each about the phenomenon or issue under study. Why do time. The observed difference of 6 mm was exceeded the seedlings grow faster in the dark? Here is an ex- only one time out of 200 trials (see Figure 33), giving planation from a biology teacher. It seems to be an extremely strong evidence of a statistically signifi cant adaptation of plants to get the seedlings from the dark difference between the means for Treatments 2 and (under ground) where they germinate into the light 3. In summary, the three treatment groups show (above ground) as quickly as possible. Obviously, the statistically signifi cant differences in mean growth seedling cannot photosynthesize in the dark and is that cannot reasonably be explained by the random as- using up the energy stored in the seed to power the

78 growth. Once the seedling is exposed to light, it shifts This is a typical example of an estimation problem its energy away from growing in length to producing that occurs in science. There is no real “popula- chlorophyll and increasing the size of its leaves. These tion” of measurements that can be sampled; rather, changes allow the plant to become self-suffi cient the sample data is assumed to be a random selection and begin producing its own food. Even though the from the conceptual population of all measurements growth in length of the stem slows, the growth in di- that could have been made. At this point, there may ameter of the stem increases and the size of the leaves be some confusion between an “experiment” and a increases. Seedlings that continue to grow in the dark “sample survey” because Cavendish actually conduct- are spindly and yellow, with small yellow leaves. Seed- ed a scientifi c investigation to get his measurements. lings grown in the light are a rich, green color with The key, however, is that he conducted essentially the large, thick leaves and short stems. same investigation many times with a goal of estimating a constant, much like interviewing many people Example 5: Estimating the Density of the Earth— to estimate the proportion who favor a certain candi- A Classical Study date for offi ce. He did not randomly assign treatments to experimental units for the purpose of comparing What is the density of the Earth? This is a question treatment effects. that intrigued the great scientist Henry Cavendish, who attempted to answer the question in 1798. Cav- The famous Cavendish data set contains his 29 mea- endish estimated the density of the Earth by using the surements of the density of the Earth, in grams per crude tools available to him at the time. He did not cubic centimeter. The data are shown below [Source: literally take a random sample; he measured on dif- http://lib.stat.cmu.edu/DASL]: ferent days and at different times, as he was able. But 5.50 5.57 5.42 5.61 5.53 5.47 4.88 the density of the Earth does not change over time, 5.62 5.63 4.07 5.29 5.34 5.26 5.44 so his measurements can be thought of as a random 5.46 5.55 5.34 5.30 5.36 5.79 5.75 5.29 5.10 5.86 5.58 5.27 5.85 5.65 sample of all the measurements he could have taken 5.39 on this constant. The variation in the measurements is due to his measurement error, not to changes in the One should look at the data before proceeding with an Earth’s density. The Earth’s density is the constant analysis. The histogram in Figure 34 shows the data to that is being estimated. be roughly symmetric, with one unusually small value. If Cavendish were alive, you could ask him if he had

79 The analysis shows that any value between 5.420 Density – 0.126 and 5.420 + 0.126, or in the interval (5.294, 8 5.546), is a plausible value of the density of the Earth. 6 That is, any value in the interval is consistent with the 4 Count data obtained by Cavendish. Now, the questionable 2 low observation should be taken into account, as it will lower the mean and increase the standard devia- 4.0 4.4 4.8 5.2 5.6 6.0 tion. If that measurement is regarded as a mistake and Density removed from the data set, the mean of the 28 re- Figure 34: Histogram of Earth density measurements maining observations is 5.468 and the standard deviation is 0.222, producing a margin of error of 0.084 and Regression made a mistake (and that is certainly what you should an interval of plausible values of (5.384, 5.552). “ do for a current data set). analysis refers Students now can check on how well Cavendish did; to the study The mean of the 29 measurements is 5.42 and the standard deviation is 0.339. Recall that the margin of modern methods pretty much agree that the average of relationships error for the sample mean is: density of the Earth is about 5.515 grams per cubic between centimeter. The great 18th century scientist did well! σ variables. 2 n Example 6: Linear Regression Analysis—Height vs. ” Forearm Length where σ is the population standard deviation. In this problem, the population standard deviation is Regression analysis refers to the study of relationships not known; however, the sample standard deviation between variables. If the “cloud” of points in a scat- provides an estimate for the population standard terplot of paired numerical data has a linear shape, deviation. Consequently, the margin of error can be a straight line may be a realistic model of the rela- estimated to be: tionship between the variables under study. The least s 0.339 squares line runs through the center (in some sense) 2 =2 = 0.126 n 29 of the cloud of points. Residuals are deﬁ ned to be the deviations in the y direction between the points in the scatterplot and the least squares line; spread is now the variation around the least squares line, as

80 Table 14: Heights vs. Forearm Lengths Height vs. forearm length 190 Forearm Height (cm) Forearm Height (cm) 185 (cm) (cm) 180 45.0 180.0 41.0 163.0 175 170

44.5 173.2 39.5 155.0 Height 165 39.5 155.0 43.5 166.0 160 43.9 168.0 41.0 158.0 155 39 40 41 42 43 44 45 46 47 48 49 50 47.0 170.0 42.0 165.0 Forearm 49.1 185.2 45.5 167.0 15 0 48.0 181.1 46.0 162.0 -15 47.9 181.9 42.0 161.0 Residual 39 40 41 42 43 44 45 46 47 48 49 50 40.6 156.8 46.0 181.0 Forearm Height = 2.76Forearm + 45.8 r2 =0.64 45.5 171.0 45.6 156.0 Figure 35: Scatterplot and residual plot 46.5 175.5 43.9 172.0 43.0 158.5 44.1 167.0 The students responsible for the study sampled other measured by the standard deviation of the residuals. students on which to make forearm and height measurements. Although the details of how the sample When using a fi tted model to predict a value of y from actually was selected are not clear, we will suppose x, the associated margin of error depends on the stan- that it is representative of students at the school and dard deviation of the residuals. has the characteristics of a random sample. An impor- Relationships among various physical features, such tant consideration here is to agree on the defi nition as height versus arm span and neck size versus shoe of “forearm” before beginning to take measurements. size, can be the basis of many interesting questions The data obtained by the students (in centimeters) are for student investigation. If I were painting a picture provided in Table 14. of a person, how could I get the relative sizes of the A good fi rst step in any analysis is to plot the data, body parts correct? This question prompted students as we have done in Figure 35. The linear trend in the to carry out an investigation of one of the possible re- plot is fairly strong. The scatterplot, together with lationships, that between forearm length and height. Pearson’s correlation coeffi cient of .8, indicate that a

81 line would be a reasonable model for summarizing the Is the slope of 2.8 “real,” or simply a result of chance relationship between height and forearm length. variation from the random selection process? This The scatterplot includes a graph of the least squares line: question can be investigated using simulation. A description of this simulation is included in the Ap- Predicted Height = 45.8 + 2.76(Forearm Length). pendix to Level C. The plot below the scatterplot shows the residuals. Example 7: Comparing Mathematics Scores— There are a few large residuals but no unusual pat- An Observational Study tern in the residual plot. The slope (about 2.8) can be interpreted as an estimate of the average difference Data often are presented to us in a form that does not in heights for two persons whose forearms are 1 cm call for much analysis, but does require some insight different in length. The intercept of 45.8 centime- into statistical principles for correct interpretation. ters cannot be interpreted as the expected height Standardized test scores often fall into this category. of a person with a forearm zero centimeters long! Table 15 gives information about the state mean scores However, the regression line can reasonably be used on the National Assessment of Educational Progress to predict the height of a person for whom the fore- (NAEP) 2000 Grade 4 mathematics scores for Louisi- arm length is known, as long as the known forearm ana and Kentucky. Even though these scores are based length is in the range of the data used to develop on a sample of students, these are the scores assigned the prediction equation (39 to 50 cm for these data). to the states, and consequently, they can be considered The margin of error for this type of prediction is observational data from that point of view. approximately 2(standard deviation of the residuals). For these data, the standard deviation of the residu- Table 15: NAEP 2000 Scores in Mathematics als is 5.8 (not shown here, but provided as part of the computer output), so the margin of error is 2(5.8) Overall Mean for Mean for % White = 11.6 cm. The predicted height of someone with a Mean Whites Non- whites forearm length of 42 cm would be: Louisiana 217.96 229.51 204.94 Predicted Height = 45.8 + 2.76(42) = 161.7 cm Kentucky 220.99 224.17 87 With 95% conﬁ dence, we would predict the height of people with forearm length 42 cm to be between 150.1 To see if students understand the table, it is informa- cm and 173.3 cm (161.7 ± 11.6). tive to ask them to ﬁ ll in a few omitted entries.

82 → Fill in the two missing entries in the table Table 16: Cigarette Smoking and Lung Cancer (53% and 199.71). Lung Cancer Controls Totals More substantive questions involve the seeming con- Cases tradictions that may occur in data of this type. They Smokers 647 622 1,269 might be phrased as follows. Non- 2 27 29 → For the two states, compare the overall means. smokers Compare the means for whites. Compare the world as it is. The data collection process still can be means for nonwhites. What do you observe? designed in clever ways to obtain as much information → Explain why the reversals in direction take place as possible. once the means are separated into racial groups. Here is an example from the smoking versus lung It is genuinely surprising to students that data summa- cancer debates. A group of 649 men with lung cancer ries (means in this case) can go in one direction in the was identiﬁ ed from a certain population in England. aggregate but can go in the opposite direction for each A control group of the same size was established by subcategory when disaggregated. This phenomenon is matching these patients with other men from the called Simpson’s Paradox. same population who did not have lung cancer. The Example 8: Observational Study—Toward Establishing matching was on background variables such as eth- Causation nicity, age, and socioeconomic status. (This is called a case-control study.) The objective, then, is to compare Observational studies are the only option for situ- the rate of smoking among those with lung cancer to ations in which it is impossible or unethical to ran- the rate for those without cancer. domly assign treatments to subjects. Such situations are a common occurrence in the study of causes of First, make sure students understand the nature of diseases. A classical example from this ﬁ eld is the re- the data in Table 16. Does this show, for example, lationship between smoking and lung cancer, which that there was a very high percentage of smokers in prompted heated debates during the 1950s and 1960s. England around 1950? The rate of smoking in these Society will not condone the notion of assigning some groups was (647/649) = .997 for the cancer patients people to be smokers and others to be nonsmokers in and (622/649) = .958 for the controls. If these data an experiment to see if smoking causes lung cancer. had resulted from a random assignment or selection, So the evidence has to be gathered from observing the the difference of about 4 percentage points would be

83 statistically signiﬁ cant (by methods discussed earlier), The term “probability” is used in the same sense as which gives the researcher reason to suspect there above. If a person is randomly selected from the 1–14 is an association here that cannot be attributed to level, the chance that the person falls into the can- chance alone. Another way to look at these data is cer column is .45, and so on for the other rows. The to think about randomly selecting one person from important result is that these “probabilities” increase among the smokers and one person from among the with the level of smoking. This is evidence that an in- nonmokers. The smoker has a chance of 647/1269 crease in the disease rate is associated with an increase = .51 of being in the lung cancer column, while the in cigarette smoking. nonsmoker has only a 2/29 = .07 chance of being Even with this additional evidence, students should there. This is evidence of strong association between understand that a cause and effect relationship cannot smoking and lung cancer, but it is not conclusive be established from an observational study. The main evidence that smoking is, in fact, the cause of the reason for this is that these observational studies are lung cancer. (This is a good place to have students subject to bias in the selection of patients and controls. speculate about other possible causes that could have Another study of this type could have produced a dif- resulted in data like these.) ferent result. (As it turned out, many studies of this Another step in establishing association in observa- type produced remarkably similar results. That, cou- tional studies is to see if the increase in exposure to pled with laboratory experiments on animals that es- the risk factor produces an increase in incidence of tablished a biological link between smoking and lung the disease. This was done with the same case-control cancer, eventually settled the issue for most people.) study by looking at the level of smoking for each per- The Appendix to Level C contains more examples of son, producing Table 17. the types discussed in this section. Table 17: Level of Cigarette Smoking and Lung Cancer The Role of Probability in Statistics Cigarettes/ Lung Cancer Controls Probability Day Cases Teachers and students must understand that sta- 0 2 27 0.07 tistics and probability are not the same. Statistics uses probability, much as physics uses calculus, but 1–14 283 346 0.45 only certain aspects of probability make their way 15–24 196 190 0.51 into statistics. The concepts of probability needed 25+ 168 84 0.67 for introductory statistics (with emphasis on data

84 analysis) include relative frequency interpretations of times. This long-run relative frequency interpretation data, probability distributions as models of popula- of probability also provides the justifi cation for using tions of measurements, an introduction to the normal simulation to estimate probabilities. After observing a distribution as a model for sampling distributions, and large number of chance outcomes, the observed pro- the basic ideas of expected value and random varia- portion of occurrence for the outcome of interest can tion. Counting rules, most specialized distributions be used as an estimate of the relevant probability. and the development of theorems on the mathematics Students also need to understand the concept of in- of probability should be left to areas of discrete math- dependence. Two outcomes are independent if our ematics and/or calculus. assessment of the chance that one outcome occurs “ Probability Understanding the reasoning of statistical inference is not affected by knowledge that the other outcome is an attempt requires a basic understanding of some important has occurred. Particularly important to statistical to quantify ideas in probability. Students should be able to: inference is the notion of independence in sampling uncertainty. → Understand probability as a long-run relative settings. Random selection (with replacement) from ” frequency; a population ensures the observations in a sample are → Understand the concept of independence; and independent. For example, knowing the value of the third observation does not provide any information → Understand how probability can be used in about the value of the fi fth (or any other) observation. making decisions and drawing conclusions. Many of the methods used to draw conclusions about In addition, because so many of the standard inferential a population based on data from a sample require the procedures are based on the normal distribution, students observations in a sample to be independent. should be able to evaluate probabilities using the normal Most importantly, the concepts of probability play distribution (preferably with the aid of technology). a critical role in developing statistical methods that Probability is an attempt to quantify uncertainty. The make it possible to make inferences based on sample fact that the long-run behavior of a random process is data and to assess our confi dence in such conclusions. predictable leads to the long-run relative frequency in- To clarify the connection between data analysis and terpretation of probability. Students should be able to probability, we will return to the key ideas presented interpret the probability of an outcome as the long-run in the inference section. Suppose an opinion poll proportion of the time the outcome should occur if shows 60% of sampled voters in favor of a proposed the random experiment is repeated a large number of new law. A basic statistical question is, “How far

85 might this sample proportion be from the true As demonstrated earlier, the sampling distribution of a population proportion?” That the difference between sample mean will be approximately normal under ran- the estimate and the truth is less than the margin of dom sampling, as long as the sample size is reasonably error approximately 95% of the time is based on a large. The mean and standard deviation of this distri- probabilistic understanding of the sampling distribu- bution usually are unknown (introducing the need for tion of sample proportions. For large random samples, inference), but sometimes these parameter values can this relative frequency distribution of sample propor- be determined from basic information about the pop- tions is approximately normal. Thus, students should ulation being sampled. To compute these parameter be familiar with how to use appropriate technology to values, students will need some knowledge of expected fi nd areas under the normal curve. values, as demonstrated next. Suppose an experimenter divides subjects into two According to the March 2000 Current Population groups, with one group receiving a new treatment Survey of the U.S. Census Bureau, the distribution of for a disease and the other receiving a placebo. If the family size is as given by Table 18. (A family is defi ned treatment group does better than the placebo group, as two or more related people living together. The a basic statistical question is, “Could the difference number “7” really is the category “7 or more,” but have been a result of chance variation alone?” The very few families are larger than 7.) randomization allows us to determine the probabil- Table 18: Family Size Distribution ity of a difference being greater than that observed under the assumption of no treatment effect. In turn, Family Size, x Proportion, p(x) this probability allows us to draw a meaningful con- 2 0.437 clusion from the data. (A proposed model is rejected 3 0.223 as implausible, not primarily because the probability 4 0.201 of an observed outcome is small, but rather because 5 0.091 it is in the tail of a distribution.) An adequate answer 6 0.031 to the above question also requires knowledge of the context in which the question was asked and a sound 7 0.017 experimental design. This reliance on context and Notice fi rst the connection between data and prob- design is one of the basic differences between statis- ability: These proportions (really estimates from a tics and mathematics. very large sample survey) can be taken as approximate

86 probabilities for the next survey. In other words, if normal in shape, centering at 3.11 with a standard de- someone randomly selects a U.S. family for a new viation of 1.24/ 100 = 0.124. This would be useful survey, the probability that it will have three mem- information for the person designing the next survey. bers is about .223. In short, the relative frequency defi nition of prob- Second, note that we now can fi nd the mean and stan- ability, the normal distribution, and the concept of ex- dard deviation of a random variable (call it X), defi ned pected value are the keys to understanding sampling as the number of people in a randomly selected family. distributions and statistical inference. The mean, sometimes called the expected value of X and denoted by E(X), is found using the formula: Summary of Level C EX()= ∑ x⋅ px () Students at Level C should become adept at using all possible statistical tools as a natural part of the investigative xvalues process. Once an appropriate plan for collecting data which turns out to be 3.11 for this distribution. If the has been implemented and the resulting data are in next survey contains 100 randomly selected families, hand, the next step usually is to summarize the data then the survey is expected to produce 3.11 members using graphical displays and numerical summaries. At per family, on the average, for an estimated total of Level C, students should be able to select summary 311 people in the 100 families altogether. techniques appropriate for the type of data available, produce these summaries, and describe in context the The standard deviation of X, SD(X), is the square important characteristics of the data. Students will root of the variance of X, V(X), given by: use the graphical and numerical summaries learned at VX()= ∑ [ x−⋅ EX ()]()2 px Levels A and B, but should be able to provide a more all possible sophisticated interpretation that integrates the context xvalues and objectives of the study. For the family size data, V(X) = 1.54 and SD(X) = 1.24. At Level C, students also should be able to draw con- Third, these facts can be assembled to describe the ex- clusions from data and support these conclusions us- pected sampling distribution of the mean family size ing statistical evidence. Students should see statistics as in a random sample of 100 families yet to be taken. providing powerful tools that enable them to answer That sampling distribution will be approximately questions and to make informed decisions. Students also should understand the limitations of conclusions

87 based on data from sample surveys and experiments, and should be able to quantify uncertainty associated with these conclusions using margin of error and related properties of sampling distributions.