Confirmation Bias in Studies of Nestmate Recognition: A Cautionary Note for Research into the Behaviour of Animals Ellen van Wilgenburg*, Mark A. Elgar Department of Zoology, University of Melbourne, Melbourne, Victoria, Australia Abstract Confirmation bias is a tendency of people to interpret information in a way that confirms their expectations. A long recognized phenomenon in human psychology, confirmation bias can distort the results of a study and thus reduce its reliability. While confirmation bias can be avoided by conducting studies blind to treatment groups, this practice is not always used. Surprisingly, this is true of research in animal behaviour, and the extent to which confirmation bias influences research outcomes in this field is rarely investigated. Here we conducted a meta-analysis, using studies on nestmate recognition in ants, to compare the outcomes of studies that were conducted blind with those that were not. Nestmate recognition studies typically perform intra- and inter colony aggression assays, with the a priori expectation that there should be little or no aggression among nestmates. Aggressive interactions between ants can include subtle behaviours such as mandible flaring and recoil, which can be hard to quantify, making these types of assays prone to confirmation bias. Our survey revealed that only 29% of our sample of 79 studies were conducted blind. These studies were more likely to report aggression among nestmates if they were conducted blind (73%) than if they were not (21%). Moreover, we found that the effect size between nestmate and non-nestmate treatment means is significantly lower in experiments conducted blind than those in which colony identity is known (1.38 versus 2.76). We discuss the implications of the impact of confirmation bias for research that attempts to obtain quantitative synthesises of data from different studies. Citation: van Wilgenburg E, Elgar MA (2013) Confirmation Bias in Studies of Nestmate Recognition: A Cautionary Note for Research into the Behaviour of Animals. PLoS ONE 8(1): e53548. doi:10.1371/journal.pone.0053548 Editor: Deborah M. Gordon, Stanford University, United States of America Received December 1, 2011; Accepted December 3, 2012; Published January 23, 2013 Copyright: ß 2013 van Wilgenburg, Elgar. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: This research was funded by the Australian Research Council. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing Interests: The authors have declared that no competing interests exist. * E-mail: [email protected] Introduction tions are more prone to bias if 1) the variable is not clearly defined, 2) the subject under observation is hard to perceive, 3) the observations require subjective assessment, and 4) the observer has ‘‘…for it is a habit of mankind to entrust to careless hope an interest in the outcome of the study. what they long for, and to use sovereign reason to thrust Confirmation bias can be avoided by designing experiments in aside what they do not desire.’’ From History of the which the observers are blind to the treatment assignment of their Peloponnesian War 431 B.C., Thucydides subjects [12–18]. For example, to test whether consumers have a taste preference for one brand of pop over another, the identity of Confirmation bias, a well-documented phenomenon in psy- the pop should be concealed because otherwise the subjects tend to chology, is the tendency of humans to seek out evidence and prefer the brand with which they are more familiar. Nowadays, interpret it in a manner that confirms their existing ideas and blind experiments are commonplace in many scientific disciplines, hypotheses [1–7]. Confirmation bias is often described as a result including pharmacology, market research, psychology, physics and of automatic processing, occurring more or less unintentionally but certain branches of biology. Indeed, in some fields of research, nevertheless potentially distorting the data collected in scientific blinding of experiments is essential for publication. research. Ideally, scientific researchers avoid confirmation bias by Such an experimental tradition appears to be less widely searching for falsifying, as well as confirming evidence [8,9]. adopted in the field of animal behaviour, where researchers collect However, in reality, scientists often have high stakes for obtaining observational data that may be subject to systematic error. For particular research outcomes [10], and the expectations for an example, a survey by Gamboa et al. [18] revealed that only 27% experiment can potentially affect which data are collected and how of 33 studies that investigated kin-recognition and were published they are interpreted and reported [1,2]. For example, two-thirds of in the journal Animal Behaviour between 1987 and 1989 mentioned recording errors in several psychological studies were biased in the blind assays. Studies of animal behaviour may be particularly direction of the observer’s hypothesis [1]. The extent to which prone to confirmation bias, especially when a certain degree of observation bias influences the outcomes of a study will depend on interpretation is required— typically when the behaviours are the kind of observations that are being made [1,7,11]. Observa- rapid, subtle or similar in appearance to other behaviours. Yet PLOS ONE | www.plosone.org 1 January 2013 | Volume 8 | Issue 1 | e53548 Confirmation Bias in Nestmate Recognition Studies remarkably few studies have investigated the extent to which specifically ask 1) what proportion of studies of nestmate confirmation bias influences research outcomes in animal behav- recognition are conducted blind, and 2) do the outcomes of blind iour [19–21]. Almost half a century ago Cordaro and Ison [21] studies differ from those of non-blind studies? conducted an experiment in which they asked students to observe the behaviour of planaria (non-parasitic Turbellaria flatworms). Methods One group of observers were told the planaria would move and turn frequently, whereas the other group of observers were told We searched for papers on nestmate recognition in ants using their planaria rarely move and turn. In reality, the planaria were ISI Web of Science (Thomson Reuters) search engine, with the search randomly allocated to the two groups. The group of students terms ‘‘nestmate recognition’’ or ‘‘nest mate recognition’’. We anticipating high-activity animals found that the planaria moved conducted the last search in July 2011. To be included in our on average 18 times and turned 49 times, while the group of analyses, studies must have conducted a nestmate recognition students anticipated low-activity animals reported the planaria experiment on ants that included both a nestmate (control) and moved on average once and tuned 10 times. Similar studies by non-nestmate aggression assay. Aggression assays had to involve Rosenthal and Fode [19] and Marsch and Hanlon [20] conducted either live, chilled or dead ants. To determine whether a study was on the behaviour of rats and salamanders respectively, also report conducted blind or not (e.g. whether the observers of the assays that a priori expectations can bias behavioural observations. were aware of the colony identity of the workers) we carefully read However, it has to be noted that the observers in all three studies through the method section of each paper. We deemed a study as were undergraduates, who generally have little or no training in blind only if this was explicitly stated, and categorised the conducting behavioural observations. More experienced research- remaining studies as non-blind. It is possible that some of the ers may make fewer observational errors and their data may studies we deemed non-blind were, in fact, conducted blind. therefore be more reliable. However, the alternative of contacting the authors of all papers to An alternative approach to addressing the question of whether ascertain whether their study was conducted blind or not confirmation bias affects research into animal behaviour is to introduces several sources of bias that we could not control. For compare the outcomes of published studies that are conducted example, authors that did not explicitly state their study was blind with those that are not. If behavioural observations are conducted blind may be less likely to recall whether the study was influenced by confirmation bias, then the outcomes of studies that done blind or more likely to remember incorrectly. have been conducted blind should have smaller effect sizes than We treated different experiments included in the same similar experiments that were not conducted blind. publication, different studies by the same author, and different In this study, we explore the evidence for confirmation bias in studies on the same species as independent because leaving them studies of animal behaviour by focussing on a single research out may lead to greater loss of information and distortion of the topic—nestmate recognition in ants. Our intention is to use studies results than those caused by their potential non-independence of nest-mate recognition as a ‘model system’ to highlight the [32]. The studies included in our sample are listed in Table 1. We potential impact of confirmation bias, which is a potential issue for do not include details of the experimental methods for each study all quantitative research, including animal behaviour. Ants, like because we see no value in drawing attention to the methods of other social insects, maintain colony cohesion by recognizing and, individual studies. The types of aggression assay and the methods if necessary, discriminating against conspecifics that are not of scoring vary between studies. For example, assays may involve members of their colony [22–26]. The mechanisms of nestmate one-on-one encounters in a petri-dish [33], or placing ants into a recognition have received considerable attention during the past nest [34] or foraging trail [35].