L11 gap-filling test for 1A5, 1L1, and 2R at KU (2012/01/05) prepared by Kow Kuroda The script below was taken from TED (http://www.ted.com) and modified by the tester to make it more faithful to the actual speech.

of my discipline— 4. evidence-based . And I : Battling Bad Science will talk you through all of these and demonstrate how So I’m a uh doctor, but I kind of slipped sideways into they work, exclusively using examples of people getting research, and now I’m an epidemiologist. And nobody re- stuff wrong. ally knows what epidemiology is. 1. Epidemiology is the So we’ll start with the absolute weakest form of evi- science of how we know in the real world if something dence known to man, and that is 5. authority . In science, is good for you or bad for you. And it’s best understood we don’t care how many letters you have after your name. uh through example as the science of those crazy, wacky In science, we want to know what your reasons are for be- newspaper headlines. And these are just some of the ex- lieving something. How do you know that something is amples. good for us or bad for us? But we’re also unimpressed by These are from the Daily Mail. Every country in the authority, because it’s so easy to contrive. This is some- world has a newspaper like this. It has this kind of bizarre, body called Dr. Gillian McKeith, Ph.D, or, to give her ongoing philosophical project of dividing all the inanimate full medical title, Gillian McKeith. (Laughter) Again, ev- objects in the world into the ones that either cause or pre- ery country has somebody like this. She is our TV diet vent 2. cancer . So here are some of the things they said 6. guru . She has massive kind of five series of prime- cause cancer recently: divorce, Wi-Fi, toiletries and cof- time television, giving out very lavish and exotic health fee. Here are some of the things they say prevents cancer: advice. She, ah, it turns out, has uh a non-accredited cor- crusts, red pepper, licorice and coffee. So already you can respondence course Ph.D. from somewhere in America. see there are contradictions. Coffee both causes and pre- She also boasts that she’s a certified professional member vents cancer. And as you start to read on, you can see that of the American Association of Nutritional Consultants, maybe there’s some kind of political veilance behind some which sounds very glamorous and exciting. You get a cer- of this. So for women, housework prevents breast cancer, tificate and everything. This one belongs to my dead cat but for men, shopping could make you impotent. Hetti. She was a horrible cat. You just go to the website, So we know that we need to start 3. unpicking the sci- fill out the form, give them $60, and it arrives in the post. ence behind this. And what I hope to show is that unpick- Now that’s not the only reason that we think this per- ing dodgy claims, unpicking the evidence behind dodgy son is an idiot. She also goes on ah, ah, ah and says claims, isn’t uh a kind of nasty carping activity; it’s so- things like, you should eat lots of dark green leaves, be- cially useful, but it’s also a kind of an– an extremely valu- cause they contain lots of chlorophyll, and that will really able explanatory tool. Because real science is all about oxygenate your blood. And anybody who’s done school critically appraising the evidence for somebody else’s po- biology 7. remembers that chlorophyll and chloroplasts sition. That’s what happens in academic journals. That’s only make oxygen in sunlight, uh and it’s quite dark in what happens uh at academic conferences. The Q&A ses- your bowels after you’ve eaten spinach. sion after a post-op presents data is often uh a blood bath. Next, we need proper science, proper evidence. So, uh And nobody minds that. We actively welcome it. It’s like a “Red wine can help prevent breast cancer.” This is a head- consenting intellectual S&M activity. So what I’m gonna line from in the U.K. “A glass of red show you is all of the main things, all of the main features wine a day could help prevent breast cancer.” So you go

1 and find this paper, and what you find is it is a real piece over the past decade. And this is the trial of fish oil pills. of science. It is a description of the changes in one en- And the claim was 12. fish oil pills improve school per- zyme when you drip a chemical extracted from some red formance and behavior in mainstream children. And they grape skin onto some cancer cells in a dish on a bench in a said, “We’ve done a trial. All the previous trials were pos- 8. laboratory somewhere. And that’s a really useful thing itive, and we know this one’s gonna be too.” That should to describe in a scientific paper, but on the question of your always ring alarm bells. Because if you already know the own personal risk of getting breast cancer if you drink red answer to your trial, you shouldn’t be doing one. Either wine, it tells you absolutely bugger all. Okay? Actually, it you’ve rigged it by design, or ah you’ve got enough data turns out that your risk of breast cancer actually increases so there’s no need to 13. randomize people anymore. slightly with every amount of alcohol that you drink. So this is what they were gonna do in their trial. They So, all we want is studies in real human, people. And were taking 3,000 children, they were gonna give them here’s another example. This is from uh Britain’s lead- all these huge fish oil pills, six of them a day, and then ing diet and nutritionist in the Daily Mirror, which is our a year later, they were gonna measure their school exam second biggest selling newspaper. “An Australian study performance and compare their school exam performance in 2001 found that olive oil in combination with fruits, against what they predicted their exam performance would vegetables and pulses offers 9. measurable protection have been if they hadn’t had the pills. Now can any- against skin wrinklings.” And then they give you advice: body spot a 14. flaw in this design? And no professors “If you eat olive oil and vegetables, you’ll have fewer skin of methodology are allowed to answer this wrinkles.” And they very helpfully tell you how to go and question. So there’s no control; Okay, there’s no control find the paper. So you go and find the paper, and what you group, but that sounds really techie, right? That’s a techni- find is an observational study, right? Obviously nobody cal term. The kids got the pills, and then their performance has been able to go back to 1930, get all the people born improved. What else could it possibly be if it wasn’t the in one maternity unit, and half of them eat lots of fruit and pills? veg and olive oil, and then half of them eat McDonald’s, They got older, okay? We all develop over time. And and then we see how many wrinkles you’ve got later. You of course, also there’s the effect. The placebo have to take a 10. snapshot of how people are now. And effect is one of the most fascinating things in the whole what you find is, of course, people who eat veg and olive of 15. medicine . It’s not just about taking a pill, and oil have fewer skin wrinkles. But that’s because people your performance and your pain getting better. It’s about who eat fruit and veg and olive oil, they’re freaks, okay? our beliefs and expectations. It’s about the cultural mean- They’re not normal, they’re like you; they come to events ing of a treatment. And this has been demonstrated in a like this, right? They are posh, they’re wealthy, they’re less whole raft of fascinating studies comparing one kind of uh likely to have outdoor jobs, they’re less likely to do manual placebo against another. So we know, for example, that labor, they have better social support, they’re less likely to two sugar pills a day are a more effective treatment for smoke —so for a whole host of fascinating, interlocking getting rid of gastric ulcers than one sugar pill. Two sugar social, political and cultural reasons, they are less likely to pills a day 16. beats one sugar pill a day. And that’s an have 11. skin wrinkles . That doesn’t mean that it’s the outrageous and ridiculous finding, but it’s true. We know vegetables or the olive oil. (Laughter) from three different studies on three different types of pain So ideally what you want to do is a trial. And everybody that a saltwater injection is a more effective treatment for thinks they’re very familiar with the idea of a trial. Tri- pain than taking a sugar pill, taking a dummy pill that has als are very old. The first trial was in the Bible— Daniel no medicine in it —not because the injection or the pills 1:12. It’s very straightforward— you take a bunch of peo- do anything physically to the body, but because an injec- ple, you split them in half, you treat one group one way, tion feels like a much more dramatic intervention. So we you treat the other group the other way, and a little while 17. know that our beliefs and expectations can be manip- later, you follow them up and see what happened to each of ulated, which is why we do trials where we control ag– them. So I’m gonna tell you about one trial, which is prob- against a placebo, where one half of the people get the real ably the most well-reported trial in the U.K. news media treatment and the other half get placebo.

2 But that’s not enough. What I’ve just shown you are ex- dose, but very much at the high end of normal. And so amples of the very simple and straightforward ways that you’re bound to make your new drug look better. And so journalists and food supplement pill peddlers and natur- it’s no surprise that overall, industry-funded trials are four opaths can distort evidence for their own purposes. What times more likely to give a positive result than indepen- I find really 18. fascinating is that the pharmaceutical in- dently sponsored trials. dustry uses exactly the same kinds of tricks and devices, But— and it’s a big but(t)— (Laughter) it turns out, but slightly more sophisticated versions of them, in order when you look at the methods used by industry-funded to distort the evidence that they give to doctors and pa- trials, that they’re actually 23. better than independently tients, and which we use to make vitally important deci- sponsored trials. And yet, they always manage to get the sions. result that they want. So how does this work? How can we So, firstly, trials against placebo: everybody thinks they explain this strange phenomenon? Well it turns out that know that a trial should be a comparison of your new what happens is the negative data goes missing in action; 19. drug against placebo. But actually in a lot of situ- it’s withheld from doctors and patients. And this is the ations that’s wrong. Because often we already have a very most important aspect of the whole story. It’s at the top of good treatment that is currently available, so we don’t want the 24. pyramid of evidence. We need to have all of the to know that your alternative new treatment is better than data on a particular treatment to know whether or not it re- nothing. We want to know that it’s better than the best cur- ally is effective. And there are two different ways that you rently available treatment that we have. And yet, repeat- can spot whether some data has gone missing in action. edly, you consistently see people doing trials still against You can use statistics, or you can use stories. I personally placebo. And you can get license to bring your drug to prefer statistics, so that’s what I’m gonna do first. market with only data showing that it’s better than noth- This is something called funnel plot. And a funnel plot ing, which is 20. useless for a doctor like me trying to is a very clever way of spotting if small negative trials make a decision. have disappeared, have gone missing in action. So this is a But that’s not the only way you can rig your data. You graph of all of the trials that have been done on a particular can also rig your data by making the thing you compare treatment. And as you go up towards the top of the graph, your new drug against really rubbish. You can give the what you see is each 25. dot is a trial. And as you go up, competing drug in too low a dose, so that people aren’t those are the bigger trials, so they’ve got less error in them. properly treated. You can give the competing drug in too So they’re less likely to be randomly false positives, ran- high a dose, so that people get . And this is domly false negatives. So they all cluster together. The big exactly what happened which antipsychotic medication for trials are closer to the true answer. Then as you go further schizophrenia. down at the bottom, what you can see is, over on this side, 20 years ago, a new generation of antipsychotic drugs the spurious false negatives, and over on this side, the spu- were brought in and the promise was that they would have rious false positives. If there is publication , if small fewer side effects. So people set about doing trials of these negative trials have gone missing in action, you can see it new drugs 21. against the old drugs, but they gave the old on one of these graphs. So you can see here that the small drugs in ridiculously high doses —20 milligrams a day of negative trials that should be on the bottom left have dis- haloperidol. And it’s a foregone conclusion, if you give a appeared. This is a graph demonstrating the presence of drug at that high a dose, that it will have more side effects in studies of publication 26. bias . And and that your new drug will look better. 10 years ago, his- I think that’s the funniest epidemiology joke that you will tory repeated itself, interestingly, when risperidone, which ever hear. was the first of the new-generation antipscyhotic drugs, That’s how you can prove it statistically, but what about came off copyright, so anybody could make copies. Ev- stories? Well they’re heinous, they really are. This is a erybody wanted to show that their drug was better than drug called reboxetine. This is a drug that I myself have risperidone, so you see a bunch of trials 22. comparing prescribed to patients. And I’m a very nerdy doctor. I new antipsychotic drugs against risperidone at eight mil- hope I try to go out of my way to try and read and under- ligrams a day. Again, not an insane dose, not an illegal stand all the literature. I read the trials on this. They were

3 all 27. positive . They were all well-conducted. I found force field of, of, of tediousness. And I think, with all of no flaw. Unfortunately, it turned out, that many of these the problems in science, one of the best things that we can trials were withheld. In fact, 76 percent of all of the trials do is to lift up the lid, finger around in the mechanics and that were done on this drug were withheld from doctors 33. peer in . Thank you very much. and patients. Now if you think about it, if I tossed a coin a hundred times, and I’m allowed to withhold from you the answers half , then I can convince you that I have 調査 a coin with two heads, okay? If we remove half of the data, 授業の方設計するために，以下の二つの点に関して意 we can never know 28. what the true effect size of these 見を述べてください． is. And this is not an isolated story. Around half of all of the trial data on antidepressants has been withheld, (1) 問題の量は適切でしたか？ but it goes way beyond that. The Nordic Group 1. 多過ぎた were trying to get a hold of the data on that to bring it all 2. ちょっと多かった together. The Cochrane Groups are an international non- 3. ちょうどよかった profit collaboration that produce systematic 29. reviews 4. ちょっと少なかった of all of the data that has ever been shown. And they need 5. 少な過ぎた to have access to all of the trial data. (2) 聴き取る箇所の難易度は適切でしたか? But the companies withheld that data from them, and so 1. 難しいところが多すぎた did the European Medicines Agency for three years. This 2. 難しいところが多かった is a problem that is currently 30. lacking a solution. And 3. ちょうどよかった to show how big it goes, this is a drug called Tamiflu, which 4. 簡単なところが多かった governments around the world have spent billions and bil- 5. 簡単なところが多すぎた lions of dollars on. And they spend that money on the 他に意見があれば書いてくれてよいです．今後の授業 promise that this is a drug which will reduce the rate of に生かします (成績には影響しません)． complications with flu. We already have the data showing that it reduces the du- ration of your flu by a few hours. But I don’t really care about that. Governments don’t care about 31. that . I’m very sorry if you have the flu, I know it’s horrible, but we’re not going to spend billions of dollars trying to re- duce the duration of your flu symptoms by half a day. We prescribe these drugs, we stockpile them for emergencies on the understanding that they will reduce the number of complications, which means pneumonia and which means death. The infectious diseases Cochrane Group, which are based in Italy, has been trying to get the full data in a us- able form out of the drug companies so that they can make a full decision about whether this 32. drug is effective or not, and they’ve not been able to get that information. This is undoubtedly the single biggest ethical problem facing medicine today. We cannot make decisions in the absence of all of the information. So it’s a little bit difficult from there to spin in some kind of positive conclusion. But I would say this: I think that sunlight is the best disinfectant. All of these things are happening in plain sight, and they’re all protected by a

4