One-Way Correlated Samples Design Advantages and Limitations
Total Page:16
File Type:pdf, Size:1020Kb
12 - 1 Chapter 12. Experimental Design: One-Way Correlated Samples Design Advantages and Limitations Natural Pairs Matched Pairs Repeated Measures Thinking Critically About Everyday Information Comparing Two Groups Comparing t Test to ANOVA Correlated Samples t Test Correlated Samples ANOVA Comparing More Than Two Groups Case Analysis General Summary Detailed Summary Key Terms Review Questions/Exercises 12 - 2 Advantages and Limitations In the previous chapter, we included the powerful technique of random assignment in our research design to reduce systematic error (confounding variables). The assignment of participants to groups in a random fashion is one of the best ways to equate the groups on both known and unknown factors prior to administration of the independent variable. However, as we noted, there is no guarantee that they will be equated. To enhance experimental control, you may want to guarantee that one or more variables are equated among your treatment levels, and you may not want to rely on random assignment to establish that equality. Remember that any variable that varies systematically among your treatment levels and is not an independent variable is a confounder that can mask the effect of the independent variable. For example, in the previous chapter we discussed random assignment of children to two groups in a TV violence study. Random assignment, by chance, could result in having more boys in one group and more girls in the other group. If gender of the child is related to the dependent variable (aggressiveness), then we have created a confounding variable that will result in systematic error. As we will see, one advantage of the correlated samples designs discussed in this chapter is the reduction of systematic error between the treatment conditions. However, the primary advantage of the correlated samples designs is the reduction of random error due to individual differences. Recall that random error creates “noise” that makes it more difficult to detect systematic effects of the independent variable. Reducing the noise enables us to detect smaller differences (systematic variance) between treatments. In terms of statistical formulas, we will see that the denominator of the formulas for our test statistic (t or F) consists of random error and the numerator consists primarily of the treatment (systematic variance). The larger the random error, the smaller the value is for the test statistic, and the less likely we are to find a treatment effect that is statistically significant. The three techniques introduced in this chapter are all correlated samples designs. Correlated samples designs do not use random assignment of participants to conditions. Instead, they either test the same research participants under each treatment condition or match different participants on a related factor. Similar to random assignment designs, correlated samples designs can be used with two treatment conditions or more. The three types of correlated samples designs are natural pairs, matched pairs, and repeated measures. We should note that the appropriate statistical test is related to the research design that is used. For example, the t test for correlated samples design is calculated differently than that for independent samples design. 12 - 3 Table 12.1 Characteristics of research designs. Type of Number Number of Number Assignment Most Probable Research of Levels of of to Conditions Inferential Design IVs the IV DVs Statistic One-way t test or independent 1 2 or more 1 Random one-way samples ANOVA One-way Natural pairs t test or correlated 1 2 or more 1 Matched pairs one-way samples Repeated measures ANOVA Natural Pairs In a natural pairs design, the scores in the groups are paired for some natural reason; an effort is made to match the participants on some natural basis. A good example of this matching would be twin studies. Returning to our TV violence study, research suggests that there is a genetic component to some aspects of personality, including aggressiveness. That is, babies come into this world with temperaments that help shape their developing personalities. Therefore, when we observe levels of aggressive behavior in children in a day-care center, we suspect that part of the explanation for their behavior is their genetic profile. Thus, variability of scores within and between groups is partly due to different participants’ having different genetic profiles. This factor contributes to random error and makes it more difficult to detect variability due to the independent variable. One solution is to eliminate genetic differences between groups by using identical twins. If we place one of the twins in one treatment condition and the other twin in the other treatment condition, we have created a situation in which there is no genetic difference between the groups. Differences between the group means could no longer be partly explained by differences in genetic profiles. Thus, in this natural pairs design, the scores in the two groups would be paired up because they are identical twins. The primary advantage of the natural pairs design is that it uses a natural characteristic of the participants to reduce sources of error. The primary limitation of this design is often the availability of participants. The researcher must locate suitable pairs of participants (such as identical twins) and must obtain consent from both participants. Matched Pairs In a natural pairs design, scores were paired for some natural reason. In a matched pairs design, scores are paired because the experimenter decides to match them on some variable. The rationale for the matched pairs design is the same as that for the natural pairs design—to reduce error variability by controlling extraneous variables. 12 - 4 Once again, let’s return to our TV violence study. It is important for the researcher to consider possible matching variables prior to the study. As the researcher, you may decide that the gender and age of the child are critical variables that relate not only to the child’s aggressive behavior, but also to how the TV program may affect them. You suspect that 5-year-old boys may be more aggressive, in general, than 3-year-old girls and may be more affected by the violence in a TV program. In the next chapter, we will see how these variables can be included as additional independent variables in the research design. But for now our goal will be to control them. Instead of relying on simple random assignment to balance these variables (gender and age) across your groups, you begin by pairing participants in your sample. A 3-year-old girl is paired with another 3- year-old girl, a 5-year-old boy is paired with a 5-year-old boy, and so on. After all the pairs are created, you use random assignment to determine which of the participants in each pair will be in the experimental group and which one will be in the control group. Now the two groups are matched in terms of both age and gender. Differences between the group means can no longer be explained by differences in age or gender of the participants. The primary advantage of the matched pairs design is to use experimental control to reduce one or more sources of error variability. One limitation of this design can be the availability of participants. At times, there may not be a suitable match for a participant. For this reason, the researcher should not try to match the groups on too many variables. The design can quickly become too difficult to manage. Usually one or two matching variables are sufficient. But remember, the matching variable(s) must be related to scores on the dependent variable. Otherwise, error variability will not be reduced. Repeated Measures With both the natural pairs and the matched pairs designs, our objective is to better equate the groups and to reduce random error due to individual differences. However, notice that we still have different participants in the different groups. Different participants will not only have different genetic backgrounds (unless they are identical twins), they will have very different sets of life experiences (including identical twins). These different life experiences shape a person and influence how he or she will behave in any given situation. Whenever you have different participants in the different experimental conditions, there will be some error variability due to individual differences. A solution is to use a repeated measures design, in which the same group of participants experiences all the conditions; that is, each research participant is tested under each treatment condition. For our TV violence study, we would sample a group of children from day-care centers and then have them participate in both experimental conditions. On one day, the children would be observed after they had watched a TV program with violence. On another day, the same children would be observed 12 - 5 after they had watched a TV program without violence. To avoid confounding due to order effects, we would have to counterbalance the order of TV programs so that half the participants watch the violent program first and half watch the program without the violence first. Advantages of Repeated Measures Designs. The beauty of this design is that it provides a means of controlling all of the extraneous variables associated with individual differences, including genetic background, socioeconomic status, age, gender, family structure, and type of parents. We have indicated that the greatest advantage of using a repeated measures design is the marked control over individual participant variation. Because each participant receives each treatment, participants with identical characteristics necessarily receive each of the different treatment conditions. Thus, any differences in performance should result only from the treatment conditions. In fact, however, this does not happen. Even though the same participant is used across treatments, the participant may change in some systematic fashion.