Spring 2014 Math 263 Deb Hughes Hallett Class 9: Experiments and Observational Studies (Text: Section 3.1) A statistical study can tell us • What is going on (the characteristics of a population) • Effect of an intervention (taking a drug, using a new fertilizer, implementing a policy to create jobs) Studies may be of two types: • Observational study: Characteristics of a population are observed, but the population is not disturbed. • Experiment: Treatment is given to a individuals and their responses recorded. In an experiment, the treatment is generally applied to randomly selected subset of the population, and the subset compared to a randomly selected control group. This is called a RCT (randomized controlled trial). In medicine, RCTs are the gold standard for clinical trials of new drugs. Ex. To investigate the safety of the anesthetics used in surgery, records from 850,000 operations in 34 major hospitals were examined. The death rates for four common anesthetics are shown in the table.1 Anesthetic A B C D Death rate 1.7% 1.7% 3.4% 1.7% The anesthetic used is the explanatory variable and the death rate is the response variable. Was this an observational study or an experiment? What can you conclude from it? The study looked at records of previous operations, so it is an observational study. Anesthetic C looks worse, but we can’t say that it caused the death. This anesthetic may be given to more risky cases. There may be confounding/hidden variables. Ex: If you have to go to hospital, should you choose the one with the lowest mortality rates? (These rates are published.) No. The best hospitals may have the highest mortality rates. Ex. The Physicians’ Health Study (1980-1995) was a longitudinal study of the effect of aspirin and beta carotene on heart attacks.2 In the study, 21,996 male physicians were randomly divided into four equal groups and given pills or placebos of aspirin and beta carotene (real versions of both, just one, or neither). The result was that 239 of the aspirin placebo group and 139 of the aspirin group had heart attacks.3 Was this an observational study or an experiment? What can you conclude from it? This is an experiment with random assignment to the four groups. The numbers of heart attacks in the two groups were sufficiently different that it allowed doctors to conclude that aspirin helps prevent heart attacks. The experiment established the effectiveness of aspirin in reducing heart attacks. The beta carotene, which the body converts into Vitamin A and which was thought might prevent some kinds of cancer, turned out not to have significant health effects. Ex. A Sept 2006 article4, “Vitamin D halves the risk of pancreatic cancer,” reported on the work scientists at Northwestern and Harvard who examined data from two long-term health studies of 46,771 men and 75,427 women. They found that people who took vitamin D had a 43% lower risk of pancreatic cancer. Comment. This is an observational study; it maybe that people who take vitamin D have less chance of getting pancreatic can cancer in any case. It is certainly suggestive; it merits an experiment. 1 From Introduction to the Practice of Statistics, by D. Moore and G. McCabe 4th edn. 2 Physicians’ Heath Study II was launched in 1997 to study the effects of Vitamin E, C and multivitamins. 3 Moore and McCabe, Introduction to the Practice of Statistics, 4th edn. “Aspirin in the primary and secondary prevention of vascular disease” published in The Lancet on May 30, 2009 provided a more nuanced view of the effectiveness of aspirin. 4 By Halcyon Skinner et al in MedlinePlus at NIH http://www.hpnonline.com/dailyupdates/September_06.html 1 Spring 2014 Math 263 Deb Hughes Hallett Why Are Experiments/RCTs Useful? Causation Even if the sample is well chosen, an observational study cannot easily establish causation. In an observational study there may be hidden variables, which explain the effect observed, but are not among the variables studied. We say that two variables are confounded if their effects cannot be distinguished from each other. If well designed, experiments can provide evidence of causation. Ex: Children with larger shoe sizes read better. Age is hidden variable; older children read better. Shoe size is confounded with age. Ex: Countries with more TV sets per person have longer life expectancy. GDP is hidden variable; money enables people to buy TVs and to take better care of their health. Ex. Oklahoma and West Virginia adopted state-funded programs promoting marriage. These are based on studies done in Oklahoma in 1998, from which5 “state economists concluded that being single and being poor were interrelated”. The programs were aimed at poor single mothers: “The hope is that marriage will provide these women, and their children, with a route out of poverty and alleviate the burden they place on the state. The thesis is simple: If failure to marry– –or divorce––means poverty, marriage must enhance wealth.” Are the 1998 studies observational studies or experiments? Can you conclude that being single causes poverty? Observational studies. Other observers “do not accept that lack of money is a consequence of lack of marriage”. Being single and poverty are associated, but there is not necessarily a causal relationship. There may be hidden variables, or poverty may cause people to stay single. In other words, even if causation is involved, it may be in the wrong direction. (That is, reverse causation.) Ex. On Sept 3, 2002, a New York Times article, “Sorting through the Confusion over Estrogen”, reported “…..in the decades since its initial approval by the Food and Drug Administration in 1942, estrogen had acquired a reputation as an antidote to many of the illnesses and afflictions of aging. Scores of observational and case studies supported this view……Major new findings from such clinical trials have seriously challenged estrogen’s image as a preventive of chronic disease….” What might have caused the results to be so different? The article says “….there is always a chance that factors not considered [in the observational studies] could have influenced the results, especially since the women who choose to use hormones tend to have healthier habits over all and are likely to be followed more closely by their physicians.” The results of the observational studies may have been confounded with the effect of the patient’s overall health. The clinical studies suggest that with the exception of a few illnesses, estrogen not an antidote to the illnesses of aging that it was once thought to be. Multiple Regression is a primary tool for establishing causation. Also natural experiments, when available, as in the next example. Ex: The Harlem Children’s Zone is an NGO (non-governmental organization) providing social services for poor children in New York City. It runs a charter school, Promise Academy, which accepts students by lottery. A study compared students who went to the school to comparable students who went to other schools.6 Students in the school gained 1.3 or 1.4 standard deviations (usual gains are about 0.2 SD.) What kind of study is this? Because of random selection of students in the school, this is a RCT: It is a natural experiment. 5 From “Wedded to the Value of Marriage”, BBC News, January 15, 2004. 6 “The Harlem Miracle”, May 7, 2009, http://www.nytimes.com/2009/05/08/opinion/08brooks.html 2 Spring 2014 Math 263 Deb Hughes Hallett Experimental Methods: Good Design Experiments should be designed to avoid confounding and Hawthorne and experimenter effects. Confounding is made less likely by having a • Control group and • Random assignment to groups. Ex: “Learning in an online format versus and in-class format: an experimental study”7 reported that in Nova Southeastern University, FL, online students did as well as those who learned in class. What might you want to ask about the experiment before investing in online learning? Was there a control group? Yes, the in-class group. Were students assigned randomly to the two groups? No. So bias introduced by lack of randomization. Were the two groups similar? No, the online group had better scores at the start of the course. The fact they only did as well as the in-class group was worrying. Ex: A 2005 experiment8 claimed that a drink containing pig whipworm eggs reduced the symptoms of Crohn’s disease (a disease of the bowel). What might you ask? Was there a control group? No. So the result could have been a placebo effect. Was the reduction medically significant? That is, big enough it brought relief to the patients. Experimenter and Hawthorne effects The experimenter effect occurs when the experimenter influences the result of the experiment (by accident). The Hawthorne effect occurs when the subjects respond differently just because they are part of an experiment. Both can be eliminated by using a placebo. Ex: Crohn’s disease experiment: Give the control group a drink that looks the same. To make sure the experimenters’ or subjects’ behavior or beliefs do not unconsciously affect the outcome, use double blinding (neither experimenter nor subject knows who is in the control group) or single blinding (one knows; the other does not). Ex: The Physicians’ Health Study experiment Used random assignment of participants to groups, placebos, and double-blinding. Ex: In 2007, doctors in Germany showed that people who ate a small amount of dark chocolate each day had lower blood pressure than those who ate white chocolate.9 What kind of blinding was possible? This experiment was single (investigator) blinded. The subjects knew what color chocolate they were eating! Ex: Are some “natural remedies” placebos? Possibly 7 Allan Schulman and Randi Sims, T.H.E Journal June1999.