NEWS FEATURE NATURE|Vol 445|18 January 2007 CONVICTION BY NUMBERS Statistics have the power to trip everyone up — including judges and juries. Even when extra care is taken to get the numbers right in court, confusion often reigns. Mark Buchanan reports. n March 2003, when nurse Lucia de Berk A court of appeal later upheld de Berk’s con- rocket science, so a statistical expert was not faced trial in a Dutch court for charges of viction, but an advisory judicial committee needed.” murder and attempted murder, the statisti- has now been appointed by the central office The appeals court confirmed Clark’s origi- Ical evidence against her seemed compel- of public prosecutors to reassess the case. The nal conviction, but Dawid says that the written ling. Investigators had identified a number of committee’s decision is expected later this year, judgement showed that the jury had failed to ‘suspicious’ deaths and near deaths in hospital and it could recommend that the case be reo- grasp the relevant statistical issues. A second IMAGES.COM/CORBIS wards in which de Berk had worked from 1999 pened. But whatever the result, the case illus- appeal in 2003 freed Clark, concluding that to 2001, and records showed that she had been trates the ongoing difficulty of ensuring that the jury might have been misled in part by the present when many of those events took place. courts use statistical reasoning properly. “This statistics presented in the original trial. Without The statistical expert testifying in the case, is a serious problem,” says statistics professor a change in legal attitudes and procedures, math- Henk Elffers, reported that the chance that Philip Dawid at University College London, ematicians worry that statistical arguments will her presence was mere coincidence was only “but there is no easy solution.” continue to be misinterpreted by the courts. 1 in 342 million. In de Berk’s case, Gill and another Dutch On the basis of this number and on limited Learning from experience mathematician, Peter Grünwald of the National forensic evidence — traces of toxic substances The case also raises questions over whether Research Institute for Mathematics and Com- found in two of the exhumed bodies — the courts have learned anything from past puter Science in Amsterdam, have submitted court found de Berk guilty, and sentenced her to misuse of statistics. In a high-profile Brit- letters to the judicial committee arguing that life in prison. But some Dutch mathematicians ish case in 1999, Sally Clark was convicted the original figure of 1 in 342 million was incor- now say that the figure cited was incorrect, and of murdering her two small children based, rect — or, at best, irrelevant to the proceedings. that the case is a classic example of how statisti- at least in part, on faulty statistical reason- De Berk first became a suspect when working cal reasoning can go horribly wrong. “The mag- ing by an expert witness — the paediatrician at the Juliana Children’s Hospital in The Hague. ical power of the big number led everyone at Roy Meadow. Dawid was invited to submit But because observations from this ward were an early stage to be totally convinced of Lucia’s written evidence on the statistical arguments the source of the initial suspicion, Gill and guilt,” says mathematician Richard Gill of the during Clark’s first appeal. He was not, how- Grünwald say, those observations should not University of Leiden in the Netherlands. “Then ever, allowed to present oral testimony. “The have been used in a calculation that tested the they went to work to confirm their theories.” lawyers and the judge argued that it was not validity of the suspicion. 254 NATURE|Vol 445|18 January 2007 NEWS FEATURE Elffers, of the Netherlands Institute for same unit, in a similar period before de Berk statement said, whereas there may be unknown the Study of Crime and Law Enforcement in started working there, had seven unexplained genetic or environmental factors that predis- Leiden, combined the Juliana data with data deaths. “It seems very strange,” says Grünwald, pose families to SIDS. from another hospital in which de Berk had “that fewer people die when there is a serial killer After the Clark case, the society established previously worked to get his figure. Gill and around.” Derksen says that the statistics compar- a working group, chaired by statistician Colin Grünwald insist that the analysis was mis- ing deaths before and after de Berk started work Aitken of the University of Edinburgh, to leading. “It makes little sense to do formal at the hospital were mentioned by her defence examine how courtroom use of statistics hypothesis testing when the data themselves lawyers, but were not sufficiently emphasized to might be improved. In the immediate future, have suggested the hypothesis,” says Gill. “The have any influence on the court. the society hopes to help provide continuing only safe thing is to go out and collect new education to practising lawyers, judges and independent data.” Due process other legal practitioners, so that they can at Gill’s own calculation estimates that the prob- This neglect illustrates a difference between least recognize the potential hazards in statis- ability that the correlation arose by chance is not legal and scientific processes. Although science tical reasoning. Ultimately, Aitken suggests, 1 in 342 million, but a much smaller 1 in 48, or aims to bring together all relevant evidence, this including statistics in the core curricula of law even as low as 1 in 5 — figures that are unlikely is not necessarily true with the law. David Kaye, degrees will be more effective. “Things will to meet the ‘beyond reasonable doubt’ needed an expert in statistics and the not improve overnight,” says for a criminal conviction. But Elffers defends law at Arizona State University Aitken, “but we are in it for the his original calculations, arguing that he applied in Tempe, notes that lawyers “The magical power long haul.” a factor that corrected for his using some of the have an incentive, and even a of the big number data twice. “Everyone is aware that I applied a duty, to select the evidence led everyone at an A matter of opinion correction,” he says. that makes their case stronger. In the United States, the Federal “What the judge ends up hear- early stage to be Judicial Center, an organization Fact or fallacy? ing often comes from the two totally convinced created by Congress to improve Aside from this debate, equally important is extreme ends of the distribu- of de Berk’s guilt.” federal courts, has published a how the court interpreted the number. Philos- tion,” he says. reference manual on the use of opher of science Ton Derksen of the University Procedures to correct such — Richard Gill scientific evidence, with one of Nijmegen, who has written a book that criti- distortions are also lacking, chapter devoted to statistics. cizes de Berk’s conviction, argues that the court even after a trial has reached a verdict. In the “But education is only palliative,” says Kaye, made an elementary statistical error known as United States, written statistical arguments who helped to write the statistics chapter. “I the prosecutor’s fallacy. are often protected by court orders, and so are don’t think there is any single way to ensure The court needs to weigh up two different not available for review or correction. “The that statistics and other scientific evidence gets explanations: murder or coincidence. The data pertaining to an individual deserve some used accurately.” argument that the deaths were unlikely to have protection,” says statistical expert Joseph Although courts expect one simple answer, occurred by chance (whether 1 in 48 or 1 in Gastwirth of George Washington University statisticians know that the result depends on 342 million) is not that meaningful on its own in Washington DC, but a summary of the how questions are framed and on assumptions — for instance, the probability that ten mur- expert reports should be made publicly avail- tucked into the analysis, giving tremendous ders would occur in the same hospital might be able, he suggests. room for legal argument. Kaye recalls giving even more unlikely. What matters is the relative Independent scientific comment of this lawyers in one case what he thought was a crys- likelihood of the two explanations. However, kind occurred during the Clark case, but tal-clear explanation of a statistical argument. the court was given an estimate for only the to unknown effect. In the 1999 Clark trial, “Their response was ‘Let’s just put him on the first scenario. Without additional information, Meadow testified that the chance of two stand, he will confuse everyone’.” says Derksen, Elffer’s number is meaningless infants from the same mother dying of Sud- Indeed, the biggest practical challenge, — and could easily be misinterpreted as a very den Infant Death Syndrome (SIDS) was only some argue, lies in the unusually subtle nature small chance that de Berk is innocent. 1 in 73 million. Two years later, after the first of statistical reasoning, which research shows To raise further doubt, other important statis- appeal, the Royal Statistical Society in London, confuses experienced professionals, such as tics were neglected by the court. When de Berk condemned both this figure and its interpreta- physicians, just as easily as the general public. worked at Juliana between 1999 and 2001, there tion. The figure would be valid only if SIDS In de Berk’s case, even Elffers now suggests that were six unexplained deaths in her unit. The cases arise independently within families, the with so much uncertainty swirling around the Sally Clark was Lucia de Berk is statistics, they should play no further role in released on still waiting to the considerations.