EGU Journal Logos (RGB) Open Access Open Access Open Access Advances in Annales Nonlinear Processes Geosciences Geophysicae in Geophysics Open Access Open Access Natural Hazards Natural Hazards and Earth System and Earth System Sciences Sciences Discussions Open Access Open Access Atmospheric Atmospheric Chemistry Chemistry and Physics and Physics Discussions Open Access Open Access Atmospheric Atmospheric Measurement Measurement Techniques Techniques Discussions Open Access Open Access Biogeosciences Biogeosciences Discussions Open Access Open Access Climate Climate of the Past of the Past Discussions Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Open Access Open Access Earth Syst. Dynam. Discuss., 4, 451–505, 2013 Earth System www.earth-syst-dynam-discuss.net/4/451/2013/Earth System ESDD doi:10.5194/esdd-4-451-2013 Dynamics Dynamics Discussions © Author(s) 2013. CC Attribution 3.0 License. 4, 451–505, 2013 Open Access Geoscientific Geoscientific Open Access This discussion paper is/hasInstrumentation been under review for the journal Earth SystemInstrumentation Agnotology: learning Dynamics (ESD). Please refer to the corresponding final paper in ESD if available. Methods and Methods and from mistakes Data Systems Data Systems Discussions R. E. Benestad et al. Open Access Open Access Geoscientific Geoscientific Model Development Model Development Agnotology: learning from mistakesDiscussions Title Page Open Access Open Access Abstract Introduction R. E. Benestad1, H. O. HygenHydrology1, R. and van Dorland2, J. Cook3,Hydrology and D. Nuccitelli and 4 Earth System Earth System Conclusions References 1 The Norwegian Meteorological Institute,Sciences Norway Sciences 2 Tables Figures The Royal Netherlands Meteorological Institute, The Netherlands Discussions Open Access 3Global Change Institute, the University of Queensland,Open Access Australia 4 Tetra Tech, Inc., McClellan, California, USA Ocean Science J I Ocean Science Discussions Received: 4 April 2013 – Accepted: 22 April 2013 – Published: 3 May 2013 J I Open Access Correspondence to: R. E. Benestad ([email protected])Open Access Back Close Published by Copernicus Publications on behalf of the European GeosciencesSolid Earth Union. Solid Earth Full Screen / Esc Discussions Printer-friendly Version Open Access Open Access The Cryosphere The Cryosphere Interactive Discussion Discussions 451 Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Abstract ESDD Replication is an important part of science, and by repeating past analyses, we show that a number of papers in the scientific literature contain severe methodological flaws 4, 451–505, 2013 which can easily be identified through simple tests and demonstrations. In many cases, 5 shortcomings are related to a lack of robustness, leading to results that are not univer- Agnotology: learning sally valid but rather an artifact of a particular experimental set-up. Some examples from mistakes presented here have ignored data that do not fit the conclusions, and in several other cases, inappropriate statistical methods have been adopted or conclusions have been R. E. Benestad et al. based on misconceived physics. These papers may serve as educational case studies 10 for why certain analytical approaches sometimes are unsuitable in providing reliable answers. They also highlight the merit of replication. A lack of common replication has Title Page repercussions for the quality of the scientific literature, and may be a reason why some Abstract Introduction controversial questions remain unanswered even when ignorance could be reduced. Agnotology is the study of such ignorance. A free and open-source software is pro- Conclusions References 15 vided for demonstration purposes. Tables Figures 1 Introduction J I Bedford (2010) argued that “agnotology” (the study of how and why we do not know J I things) presents a potentially useful tool to explore topics where knowledge is or has Back Close been contested by different people. The term “agnotology” was for the first time coined 20 in Proctor and Schiebinger (2008), which provided a collection of essays addressing Full Screen / Esc the question “why we do not know what we do not know?”. Their message was that ignorance is a result of both cultural and political struggles as well as an absence of Printer-friendly Version knowledge. The counterpart to agnotology is epistemology, for which science is an important basis. In principle, the scientific way of thinking is the ideal means of resolv- Interactive Discussion 25 ing questions about causality, and science can provide valuable guidance when there are conflicting views on matters concerning physical relationships. One of the scientific 452 Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | virtues is debate and disagreement about different hypotheses, making it dynamic and providing a driving force for progress. In this process, controversial questions should ESDD be addressed with the scientific method and rigour, and in order to provide convinc- 4, 451–505, 2013 ing answers, it is important that the process is transparent, the results are replicable, 5 the hypotheses testable, and the tests objective. It is also important that critiques and debates are conveyed by the scientific literature when past findings are challenged. Agnotology: learning An agnotological study of the climate sciences can shed light on some recent contro- from mistakes versies which take place when groups with different scientific backgrounds and mind- sets dispute each other’s conclusions. Some case studies may include a number of R. E. Benestad et al. 10 recent papers that have suggested a strong influence on Earth’s climate from solar vari- ability, Jupiter, Saturn or the lunar orbit (Friis-Christensen and Lassen, 1991; Scafetta, Title Page 2010; Scafetta and West, 2008, 2007, 2006a, b, 2005; Svensmark, 1998; Svensmark and Friis-Christensen, 1997). These papers have also argued that greenhouse gases Abstract Introduction (GHG) such as CO2 play a relatively small role for Earth’s climate, and dispute the view Conclusions References 15 presented by the mainstream climate research community (National Research Council (US), 2001; Oreskes, 2004; Solomon et al., 2007). In this respect, it is important to ask Tables Figures whether these differences reflect legitimate uncertainties and gaps in our knowledge. In order to get to the bottom of such issues, one needs to follow the line from the original J I information source, via analysis, to the interpretation of the results and the final con- 20 clusions. True and universal answers should in principle be replicated independently, J I especially if they have been published in the peer reviewed scientific literature. A mes- Back Close sage from Proctor and Schiebinger (2008) is that ignorance in these issues may stem from the culture neglecting replication, not sharing methods and data, or not testing Full Screen / Esc the methods in different settings. The most persuasive arguments are the ones where 25 everybody can repeat the analyses for themselves, examine the methods, and get the Printer-friendly Version same results. Scientific truths should in principle be universal, which means that they should be generally valid and the methods objective. Interactive Discussion All the examples discussed in this paper have been cited in the public discourse to dispute the causes of climate change, where their conclusions have tended to be 453 Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | taken at face value without replication or verification. Many of the examples here have been used to back up claims in the public discourse on climate in the media (Rahm- ESDD storf, 2012). A high proportion of Americans doubt the anthropogenic cause behind 4, 451–505, 2013 the recent climate change and seem to be unaware about the level of scientific agree- 5 ment underpinning the view about anthropogenic global warming. Doran and Zimmer- man (2009) reported that 52 % of Americans think most climate scientists agree that Agnotology: learning the Earth has been warming in recent years, and 47 % think climate scientists agree from mistakes that there is a scientific consensus about human activities being a major cause of that warming. Anderegg et al. (2010), on the other hand, presented a survey that suggested R. E. Benestad et al. 10 that 97–98 % of the climate researchers most actively publishing in the field support the main conclusions by the IPCC. Cook et al. (2013) reviewed nearly 12 000 climate ab- Title Page stracts and received 1200 self-ratings from the authors of climate science publications. Using both methodologies, they found a 97 % consensus in the peer-reviewed climate Abstract Introduction science literature that humans are causing global warming. There appears to be a gap Conclusions References 15 in the understanding of the climate between experts and the lay public, and a common denominator between all the examples reported here and in the supporting material Tables Figures is that they all represent a contribution towards the agnotology associated with the climate change issue. J I In the US, the “Nongovernmental International Panel on Climate Change” (NIPCC) 20 report (Idso and Singer, 2009), the “Science & Environmental Policy Project” (SEPP) J I and the Heartland Institute have played an active role in the public discourse, pro- Back Close moting the ideas from some of these cases. In Norway, there have been campaigns led by an