Score and ignore A radio listener’s guide to ignoring health stories

Do you shout at the morning radio when a story about a medical “risk” is distorted, exaggerated, mangled out of all recognition? Does your annoyance ruin your breakfast? You are not alone. Kevin McConway and David Spiegel- halter have developed a defence strategy to save their start-of-the-day sanity. Strike back at the presenters! And make it personal…

When we wake up we are among the millions all of reporting, and applicability. In each case, a weaned] are more likely to eat fruit over the UK who turn on the Today programme on “yes” answer to the question indicates some and vegetables when they are older BBC Radio 4, presented by our favourite, veteran cause for concern about the report or the study than those given meals from jars and and generally excellent reporter John Humphrys. it is reporting on. A score of 12 “no” answers packets, researchers say.” This could be All over the US millions more tune in to simi- implies a perfect study, perfectly reported; 12 because the home-cooked food causes lar early news programmes with their own pet “yes” answers calls for apoplexy and vituperative the later eating behaviour, or it could be presenters. On many mornings there is a report letters to editors. Here goes: because some other aspect of the babies’ on the latest health risk story. At work we flip upbringing (a confounder) is associated through popular newspapers and there are more Study quality with both the weaning food and the later stories reporting the latest “research”, for exam- • Just observing people? diet. Wealth, social background, and ple on proposed links between cats and cancer Is it purely an observational study, whether both parents are working would or mobile phones and asthma. Our irritation at that is, did it just involve observing be obvious confounders here. Or there the uncritical coverage these stories generally and recording what people did when might be some reasons for a system- receive has occasionally led to loud verbal abuse left to their own devices? You cannot atic bias – for example, in asking people of the radio and of its presenters and journal- infer cause rather than correlation unless about their past use of mobile phones, ists, to the detriment of peaceful digestion of there is some controlled experimental people with a brain disease might simply toast and coffee and of the domestic harmony aspect in which, ideally, people have be more likely to recall higher usage than that should greet the new day. Our families and been randomly allocated to different people unaffected by the disease. A well- colleagues have encouraged us towards a more interventions or treatments. reported story would draw attention to creative response. • Yet another single study? issues like this. A less good effort might We have therefore constructed a simple Is it just one isolated study, which does not, but you can still use your ingenuity checklist for scoring such health stories. It is not place the new findings in the broader to think about what alternative explana- aimed at general consumers of media reports, so context of what is known? No single tions are plausible. it concentrates on issues about the news process, study is sufficient to draw a firm conclu- • Small? how the story came to be reported, and what has sion. The ideal is to pool systematically Size matters. If the number of people not been reported. Our key concern is the overall the evidence from all available studies, studied is too small (as a rule of thumb, value of the story to a listener or reader as a as is done in reviews carried out by the in the tens rather than hundreds or thou- source of reliable information relevant to his or Cochrane Collaboration. sands), then it might well be misleading. her own health. • Might there be a different explanation that It can take some technical expertise to you are not told of? be clear about what an appropriate size Is there some plausible explanation for is for a particular study, but your intui- Our checklist the results, other than the explanation tion should at least allow you to smell that you are being given? For example, a rat. It is usually the number of events Our checklist has 12 separate items, relating one press report1 began: “Babies fed that is important, not the total number to three broad aspects: study quality, standard home-cooked food [when they are of individuals studied, so there might be

© 2012 The Royal Statistical Society october2012 45 Credit: Tom Boulton

a study of a very rare disease where mil- IARC’s own press release3 on which the • Public relations puff? lions of people were followed up but only story was based. However a decision on Ask yourself: “Why am I reading this?” a handful got the disease – that makes it whether to score this item is not always Things that look superficially scientific a small study. clear-cut. Headline writers are limited to and objective may just be the product a handful of words – and they do have to of a PR process, so check who sponsored Standard of reporting try to make those words interesting. the work. It is worth asking whether the • Original information unavailable? • No independent comment? original study report was subject to peer It might be going too far to expect a link Does the story seem just to be parroting review. But beware, the most respectable to the original report in a brief printed a press release? Does it look at just one journals can err – remember that Wake- press story, but you should at least be side of the story? Are there no quotes field’s erroneous speculations on autism able to find the original source by a sim- or comments from other scientists? The and vaccination appeared first in the ple Internet search – even if, once you comments do not have to be negative. Lancet. Further, stories may be newswor- have found it, it is behind an expensive But comments from people with an obvi- thy without actually being substantively paywall. But at least you may be able ous vested interest do not count. very important; this often happens when to read the abstract and/or the press • “Higher risk”? the story mixes the mundane with the release for nothing. Does the story just tell you comparative dread (such as relating cats to cancer, • Headline exaggerated? (relative) information, like how much or relating babies’ bottles to all kinds Is it exaggerated beyond what the origi- greater the risk is if you are exposed to of scary symptoms and diseases). Or a nal study report said? For instance, in the potential hazard compared to if you story may have been made to look more reporting on the International Agency are not? Relative information does not newsworthy because it is “disease- for Research on Cancer (IARC) decision to on its own inform decisions (essentially mongering”4 – exaggerating, over-selling list the radiation from mobile phones as because twice not very much is still not or over-medicalising a condition. “possibly carcinogenic to humans”, the very much). Ideally the story should give • Half the picture? Daily Express2 used the headline “SHOCK absolute numbers – what is the chance Ask yourself: “What am I not being told?” WARNING: MOBILE PHONES CAN GIVE YOU of the disease if you are not exposed to More broadly, does the story look at all as- CANCER”. This went considerably beyond the possible risk factor, and what is the pects? Many substances that might harm the relatively measured tones of the chance if you are exposed? your health might also have benefits, and

46 october2012 ring as many alarm bells as providing adverse Box 1. Magnetic fields and asthma information). Count up the points. The nearer A report5 on a possible link between exposure everyday (babies) with the dread (radia- the score is to 12, the less notice you should to low-frequency magnetic fields in pregnancy tion), and this is probably the reason it take of the story and the more you should ignore and asthma in the children resulting from the is receiving publicity. It is in the “print it it lest it ruin your day. pregnancy was published on the Reuters web- because it will sell our paper” category, To make it easier, it is well known that a site. Our scoring is as follows. rather than “print it because it is vital mnemonic can assist greatly in making a check- the world should know”. list useful. We can reorder the items so that the J Just observing people? H Half the picture? initials give: Yes. The study was clearly observational. Yes. This is slightly tricky. Further in- O Original information unavailable? vestigation reveals that investigating J Just observing people? No. There is no link to the original study, asthma was essentially an afterthought O Original information unavailable? but it is very easy to find in an internet in a study that was originally about H Headline exaggerated? search, given that the journal and lead something else. But this was not stated N No independent comment? author are named. in the Reuters story. H Headline exaggerated? R Relevance unclear? H “Higher risk”? No. The headline was “Magnetic field No. The study is relevant to someone who U Unjustified advice? exposure linked to asthma risk”. It sum- is pregnant, or planning to become preg- M Might be explained by something else? marised the story. nant, though it is not entirely clear what P Public relations puff? N No independent comment? one might do to avoid this radiation. H Half the picture? Yes. There are appropriate remarks from Y Yet another single study? R Relevance unclear? the lead author, clarifying some of the Yes. Y Yet another single study? limits beyond which one should not push S Small? S Small? his conclusions, but nothing independent. Yes. Again, this is rather marginal. The H “Higher risk”? key number is the not the relatively large A little creative rewording of the headings No. Absolute risks are also given. number of pregnant women, but the should enable readers elsewhere to commemo- U Unjustified advice? number of children with asthma, clearly rate their own favourite, or less than favourite, No. No individual advice was given. reported in the story as 130. This is get- presenters. M Might be explained by something else? ting out of the “tens” zone, but we felt it We illustrate the use of the John Humphrys Yes. The story does not mention other was small enough to raise some doubts. scale by applying it to two example stories possibilities, although, to be fair, it does (Boxes 1 and 2). make it clear that almost anything linked The total score is thus 6/12. The story and/or to location might be a confounder. the study it reports on have several unsatisfac- P Public relations puff? tory features, but there are some good points Conclusion Yes. The story does the classic mix of the too. It should be borne in mind that what we are rating here is the usefulness of the story to a stories often do not mention these. Most in non-human animals, or perhaps it reader as a source of reliable information rel- new (and old) treatments have certain involved people who are untypical or evant to their own health. Because the scoring is undesirable effects or costs as well as simply not similar enough to you (a very inevitably somewhat subjective, because things good effects. Are they described, and different age, perhaps, or living in a are not always clear-cut, and because we are does the story make it clear how likely country where conditions are sufficiently not claiming that every item should be equally they are to occur? Also, was there a “fish- dissimilar to your own). Or perhaps it re- weighted, there is no suggestion of some fixed ing expedition” where a great number of lates to some intervention or treatment cut-off value beyond which you should simply possible outcomes or conditions were which is, for whatever reason, not avail- ignore the report entirely. The point is to raise examined – is this the only positive re- able to you. Perhaps it measures some questions and issues, not to make decisions. sult out of many looked at? Sometimes surrogate outcomes, such as hormone Finally, though the John Humphrys score it’s hard to tell – a poor report might not levels in patients with cancer rather is primarily aimed at readers of media reports mention that they investigated a lot of than actual health or survival outcomes. in relation to their own health concerns, we things and just told you about the one This is not to imply that studies with hope and believe that it may help those writ- that came out positive. surrogate outcomes are necessarily bad; ing press releases or media stories to get things often they are all that is feasible at a across more clearly and appropriately. Maybe it Applicability certain stage of research. But their direct will even be of some use to Today programme • Unjustified advice? relevance to health choices is likely to presenters. Does the story give advice on how you be unclear. should behave, or change your behav- iour, that is not justified by the results To use the checklist, score a point for every “yes” Appendix: Other checklists and criteria of the study? answer – here “yes” is bad and points mean poor • Relevance unclear? performance. Score a point also if you cannot We have taken account of previous UK checklists, Might the result simply not apply to tell what your answer should be (because not which include a piece on the NHS Choices web- people like you? Perhaps it was a study providing relevant information should generally site7, and another on the BBC’s website8.

october2012 47 Similar information sources from the USA relate to a cultural difference between the UK www.­mediadoctor.hk/ and http://­www.­ 9 include one from the Statistical Assessment and US media on the perceived news value of mediadoctor.jp/. Service (STATS). It is a rather long document different types of health story. Somebody should rather than an easily accessible checklist. The do some work on that. [Any offers? We would be Acknowledgement online Health News Review regularly rates US happy to publish it in Significance – Ed.] Related The authors gratefully acknowledge the contribu- media stories on health according to ten cri- websites exist in several other countries as part tion of Hilly Edwards in coming up with the core teria4. These excellent criteria differ somewhat of the international Media Doctor project; see joke of this paper. from ours in that they are aimed most centrally http://www.mediadoctor.org.au/, at stories about treatments and interventions, http://www.­mediadoctor.ca/, http:// References rather than health risks. This may or may not www.­mediendoktor.de/, http:// 1. Daily Mail Reporter (2011) Babies fed home-cooked food from birth are “more likely to eat fruit and vegetables later in life”. Daily Mail, Box 2. Television and length of life July 21st. Available at http://www.dailymail.

6 co.uk/health/article-2016423/Babies. The Daily Telegraph reported­ on an Australian television is given; the study outcomes html. study about the correlation between length of do not support this. 2. Reynolds, M. (2011) Shock warning: life and television viewing habits. Here is our M Might be explained by something else? mobile phones can give you cancer. Daily Express, scoring. Yes. It is true that the story deals with June 1st. Available at http://www.express. sedentary behaviour as a more plausible co.uk/posts/view/250043/Shock-Warning- J Just observing people? overall cause than watching television in Mobile-phones-can-give-you-cancer. 3. International Agency for Research Yes. The study again is clearly itself, but the main thrust is that it is on Cancer (2008) IARC classifies radiofrequency observational. the TV that is the problem; alternative electromagnetic fields as possibly carcinogenic to O Original information unavailable? explanations are obvious. humans. Press Release no. 208, IARC, Lyon. Available No. No link to the original study is given, P Public relations puff? at http://www.iarc.fr/en/media-centre/ but again it is very easy to find it in an Yes. There is no obvious PR angle, and pr/2011/pdfs/pr208_E.pdf. internet search. the original report is in a peer-reviewed 4. Health News Review (2012) Review Criteria. H Headline exaggerated? journal. But the concentration on the Available at http://www.healthnewsreview. Yes. The situation here is complicated. idea that some mundane and innocuous org/about-us/review-criteria/. The headline, “Every hour of TV watching activity is going to kill you, as an angle 5. Steenhuysen, J. (2011) Magnetic field shortens life by 22 minutes”, definitely for the press story and indeed (arguably) exposure linked to asthma risk. Reuters, August 1st. Available at sounds as if it is referring to a causal the original report, is enough to score http://www.reuters.com/ connection. Thus, on the face of it, it here. article/2011/08/01/pregnancy-asthma- idUSN1E7701IT20110801. definitely goes beyond the evidence in H Half the picture? 6. Beckford, M. (2011) Every hour the study that is being reported on. Yes. The story does not make clear why of TV watching shortens life by 22 minutes. However, again as the report makes watching television was picked out for Daily Telegraph, August 15th. Available at clear, the lead author of the study uses attention. It says little about why these http://www.telegraph.co.uk/health/ such terminology too (and, indeed, it people were chosen for study or, more healthnews/8702101/Every-hour-of-TV- appears in the abstract of the paper be- importantly, what other aspects of their watching-shortens-life-by-22-minutes. ing reported on). So does that make it behaviour were recorded (and could have html. acceptable for the newspaper report to been reported on). 7. White, A. (2009) How to read health news. NHS Knowledge Service. Available at use the same language? In our view, no. R Relevance unclear? http:// The study does not support the wording Yes, though marginal. It is a study of www.nhs.uk/news/Pages/Howtoread in the headline. If the newspaper had Australians – is that relevant to me as articlesabouthealthandhealthcare. aspx and at http://www.bazian.com/ distanced itself from the wording, by a non-Australian? – and the report does pdfs/HowToReadHealthcareNewsStories_ writing (say) “Scientists say every hour not make clear which Australians were vers05_08Mar10.pdf. of TV watching shortens life by 22 min- sampled, so it is possible they were 8. BBC (2002) Things to consider when utes”, or by using quotation marks, we untypical. However, this is verging on reading medical research. BBC h2g2 online would not have said “Yes” for this item. nitpicking. community, December 11th. Available at http:// N No independent comment? Y Yet another single study? www.bbc.co.uk/dna/h2g2/A852761. No. There is expert comment from credi- Yes. 9. Szalavist, M., Butterworth, T. and Goldin, R. (n.d.) How to evaluate health risks. Statistical ble sources (if not independent research- S Small? Assessment Service, Arlington, VA. Available ers), and, while not criticising the causal No. Numbers studied looks reasonable for at (mis)interpretation, the comments draw a study of this nature (though the report http://www.stats.org/in_depth/ evaluate_healthrisks/How_eval_health_ attention to the importance of sedentary does not make it clear how many deaths, risks.htm. behaviour. the relevant events, occurred). H “Higher risk”? No. Absolute reductions in lifetime are The total score is 8/12. The story and/or the Kevin McConway is Professor at the Department of given. study it reports on have many unsatisfactory Mathematics and Statistics, The Open University. U Unjustified advice? features; some stem from the original study ­David Spiegelhalter is Winton Professor for the Pub- Yes. Advice specifically to avoid watching rather than from the way it was reported. lic Understanding of Science at the University of ­Cambridge.

48 october2012