Cognitive Biases in Software Engineering: A Systematic Mapping and Quasi-Literature Review Rahul Mohanani, Iflaah Salman, Burak Turhan, Pilar Rodriguez, Paul Ralph Abstract—One source of software project challenges and failures is the systematic errors introduced by human cognitive biases. Although extensively explored in cognitive psychology, investigations concerning cognitive biases have only recently gained popularity in software engineering (SE) research. This paper therefore systematically maps, aggregates and synthesizes the literature on cognitive biases in software engineering to generate a comprehensive body of knowledge, understand state of the art research and provide guidelines for future research and practise. Focusing on bias antecedents, effects and mitigation techniques, we identified 67 articles, which investigated 47 cognitive biases, published between 1990 and 2016. Despite strong and increasing interest, the results reveal a scarcity of research on mitigation techniques and poor theoretical foundations in understanding and interpreting cognitive biases. Although bias-related research has generated many new insights in the software engineering community, specific bias mitigation techniques are still needed for software professionals to overcome the deleterious effects of cognitive biases on their work. Index Terms—Antecedents of cognitive bias, cognitive bias, debiasing, effects of cognitive bias, quasi-systematic literature review, systematic mapping, systematic literature review —————————— u —————————— 1 INTRODUCTION OST research on software engineering (SE) practices • Memory, e.g., remembering previous projects as M and methods focuses on better ways of developing having gone better than they did. software. That is, most studies present an ostensibly good • Perception, e.g., perceiving a unit test as designed way of doing something (e.g., a design pattern). Another to break the system when it is actually designed to way to improve software development is to identify com- confirm that the system works. mon errors (e.g., antipatterns), their causes and how to Cognitive biases help to explain many common soft- avoid them. This is where the concept of cognitive biases is ware engineering problems in diverse activities including exceptionally useful. design [3], [4], [5], testing [6], [7], requirements engineering A cognitive bias is a common, systematic error in human [8], [9] and software project management [10]. reasoning [1]. For example, confirmation bias is the ten- While there is no definitive index, over 200 cognitive bi- dency to pay undue attention to sources that confirm our ases have been identified, and few studies attempt to clas- existing beliefs while ignoring sources that challenge our sify cognitive biases [11]. Some research in the information beliefs [2]. Within software development context, cognitive systems community (e.g. [2], [12], [13]) notwithstanding, biases may affect: no previous comprehensive literature review of research on • Behavior, e.g., deciding to use a relational data- cognitive biases in SE exists. This hinders development of an base because it seems like a default option. actionable, cumulative body of knowledge. This literature • Belief, e.g., strongly believing that well-docu- review tries to systematically uncover all the cognitive bi- mented requirements are crucial for success with- ases investigated in SE research, not limited to any specific out any good supporting evidence. knowledge area of SE by explicitly focusing on the anteced- • Estimation, e.g., (over/under)-estimating the time ents and effects of biases, and any debiasing techniques used and effort needed to develop a system. to mitigate the effects of cognitive biases. The aim of this study is therefore to document the state of the art of human ———————————————— cognitive bias related research in SE, addressing the follow- • R. Mohanani, I. Salman, P. Rodriguez are with M3S Group, University of Oulu, Oulu, Finland. ing research question. E-mail: [email protected], [email protected], pilar.rodri- Research question: What is the current state of research on [email protected] cognitive biases in software engineering? • B. Turhan is with the Department of Computer Science, Brunel University London, London, UK. To address this question, we adopt a quasi-systematic E-mail: [email protected] literature review and systematic mapping approach. This • P. Ralph is with the Department of Computer Science, University of Auck- combination facilitates classifying and aggregating find- land, Auckland, New Zealand. ings into a comprehensive and structured body of E-mail: [email protected] knowledge. By presenting gaps in available research and discussing implications to research and practise, this study provides grounded avenues for future bias-informed in- 4. Psychological phenomena, for example, the Polly- vestigations with a view to establishing foundational in- anna Principle - an individual's “tendency to be sights in application of cognitive psychology in SE. overly positive in perception, memory and judg- This paper is organized as follows: the next section pro- ment” may reinforce optimism bias [12]. vides a brief primer on cognitive biases, focusing on their 5. Human cognitive limitations, for example, diffi- causes, effects and consequences. Section 3 describes our culty in quickly processing complex information research approach. We then present the findings of our sys- [P24]. tematic mapping (Section 4) and literature review (Section 6. Motivational factors such as rewards and incen- 5). Section 6 discusses the implications and limitations of tives, responsibility / empowerment, and stress, our findingsand offers avenues for future research. Section risk or lack of communication channels [22]. 7 summarizes the study’s contributions. 7. Adaptation to natural environments [33]. While the causes of cognitive biases are of great interest in psychology, SE researchers focus more on their effects 2 COGNITIVE BIASES: A BRIEF PRIMER [21]. While Section 5.2 provides a more comprehensive list, Human behavior and cognition constitutes a critical re- some examples include: search area in software engineering (SE) [14], [15], [16], • Confirmation bias (defined earlier) is implicated in [17]. Although agile methodologies and empirical research the common antipattern where unit tests attempt highlight the importance of people and teams involved in to confirm that the code works rather than to software development, SE research and practice continue break it [34]. to focus more on technologies, techniques and processes • Optimism bias, the tendency to produce unrealisti- than common patterns of human-related errors and their cally optimistic estimates [35], contributes to the manifestation in project failures. tendency for all kinds of projects (not only soft- Moreover, many studies on human decision-making in ware projects) to exceed their schedules and budg- software engineering adopt theories and concepts from ets [36]. psychology to address issues in, for example, decision sup- • The framing effect can lower design performance. port systems [2], [18], software project management [19] The framing effect refers to the tendency to give and software engineering and development paradigms different responses to problems that have surface [20], [12], [21]. Numerous systematic literature reviews dissimilarities but are formally identical [37]. have explored psychological and sociological aspects of SE Framing desiderata more formally and defini- including motivation [22], personality [23], organizational tively reduces design quality by impeding creativ- culture [24] and behavioral software engineering [25]. ity [3], [38]. These studies collectively demonstrate the importance of SE researchers are also interested inhibiting cognitive human cognition and behavior for software engineering biases or mitigating their deleterious effects [21], that is, de- success. biasing [39], [40]. As Fischoff [39], explains, debiasing inter- Since Tversky and Kahneman [26] introduced the term ventions can be divided into four levels: in the early 1970s, a vast body of research has investigated To eliminate an unwanted behavior, one might use an escalation de- cognitive biases in diverse disciplines including psychol- sign, with steps reflecting increasing pessimism about the ease of per- ogy [27], medicine [28] and management [13]. Kahneman fecting human performance: (A) warning about the possibility of bias [29] went on to differentiate fast, “system one” thinking without specifying its nature… (B) describing the direction (and per- from slow, “system two” thinking. System one thinking is haps extent) of the bias that is typically observed; (C) providing a dose unconscious, effortless, intuitive, and leads to systematic of feedback, personalizing the implications of the warning; (D) offer- errors (i.e. biases). System two thinking, contrastingly, is ing an extended program of training with feedback, coaching, and conscious, effortful, more insightful, more rational, more whatever else it takes to afford the respondent cognitive mastery of statistical and less prone to systematic errors [29]. the task. (p. 426) However, system-one thinking is not the only source of biased reasoning. More generally, cognitive biases can be The problem with this approach is that options A, B and caused by: C are ineffective [41], [42], and option D can be very expen- 1. Heuristics, that is, simple and efficient, but not nec- sive. Fischoff therefore