G.I. Joe Phenomena: Understanding the Limits of Metacognitive Awareness on Debiasing Ariella S. Kristal Laurie R. Santos Working Paper 21-084 G.I. Joe Phenomena: Understanding the Limits of Metacognitive Awareness on Debiasing Ariella S. Kristal Harvard Business School Laurie R. Santos Yale University Working Paper 21-084 Copyright © 2021 by Ariella S. Kristal and Laurie R. Santos. Working papers are in draft form. This working paper is distributed for purposes of comment and discussion only. It may not be reproduced without permission of the copyright holder. Copies of working papers are available from the author. Funding for this research was provided in part by Harvard Business School. 1 RUNNING HEAD: G.I. JOE PHENOMENA G.I. Joe Phenomena: Understanding the limits of metacognitive awareness on debiasing Ariella S. Kristal1* and Laurie R. Santos2 *Corresponding author: [email protected] 1 Harvard Business School 2 Yale University 2 RUNNING HEAD: G.I. JOE PHENOMENA Abstract Knowing about one’s biases does not always allow one to overcome those biases— a phenomenon referred to as the G. I. Joe fallacy. We explore why knowing about a bias doesn’t necessarily change biased behavior. We argue that seemingly disparate G. I. Joe phenomenon biases fall into two main categories based on their cognitive architecture. Some biases are encapsulated— knowledge cannot affect a bias because the representations or emotions that give rise to the bias are informationally encapsulated. Encapsulated biases are the hardest to overcome since they are cognitively impenetrable by their very nature. In contrast, attentional biases are cognitively penetrable enough to be overcome through awareness, yet people still fall prey to such biases under conditions of distraction or limited attention during the moment of decision-making. We conclude by discussing the important implications these two discrete categories have for overcoming these biased and for debiasing efforts generally. Keywords: biases; debiasing; illusions; judgment and decision-making; nudge 3 RUNNING HEAD: G.I. JOE PHENOMENA “It's not a case of: ‘Read this book and you'll think differently.’ I've written this book, and I don't think differently”- Daniel Kahneman on Thinking Fast and Slow (2011) 1. Introduction Many of our most common self-regulatory failures have a shared feature— that is, we often fail to behave in a certain way even though we know the positive consequences of that behavior. Many of us know that we should limit our sugar and fat intake, yet we still stop at the donut shop too often. Many of us know we need to save for retirement, yet we fail to put enough money away each week. Many of us know we should exercise to reduce our disease risk, but we still hit the snooze button rather than heading for the gym. This disconnect between our knowledge and action does not just occur in the domain of self-regulation. There are examples throughout the field of cognitive science and decision-making where our behavior systematically deviates from rationality. These examples— which we refer to as a “biases'' throughout this paper— include cases where people deviate from a standard that is objectively and normatively defined, as well as more subjective cases in which people deviate from a standard behavior they themselves would reflectively express wanting to follow (either by failing to take an action they would want their rational self to take or by performing an action that they reflectively regret). The term “G. I. Joe fallacy” was coined in 2014 to deal with these sorts of cases— situations where our behaviors do not match up with our knowledge and beliefs. The G. I. Joe fallacy refers to the misguided notion that knowing about a bias is enough to overcome it (Santos & Gendler, 2014). The name of this fallacy derives from the 1980s television series G. I. Joe, which ended each cartoon episode with a public service announcement and closing tagline, “Now you know. And knowing is half the battle.” Santos and Gendler (2014) argued that for many cognitive and social biases, knowing is much less than half of the battle. They further 4 RUNNING HEAD: G.I. JOE PHENOMENA argued that many people believe this is not the case, that knowledge is in fact enough to behave better, hence the G. I. Joe fallacy. But the existence of the G.I. Joe fallacy raises an important question: why is knowing not half the battle? That is, why does consciously accessing information relevant to a situation not alter our biased behaviors? Why are there so many G.I. Joe phenomena— a term we will use in this paper to refer to empirical cases in cognitive science in which knowing is not enough to change the experience of a bias? And do all of these G.I. Joe phenomena work in the same way, via the same mechanistic processes? By understanding the different processes that give rise to a disconnect between reflective awareness and action, we hope to approach the problem of debiasing— changing people’s behavior so that they behave more rationally— in ways that are more effective than simply learning about the existence of the phenomena themselves. 1.2 Overview of this paper The goal of this paper is to review the puzzle of these G.I. Joe phenomena in more detail and explore why knowing is not always enough from a cognitive perspective. We argue that different G.I. Joe phenomena emerge for importantly different mechanistic reasons. In doing so, we develop a novel categorization of cognitive biases, arguing for the first time that there are at least two different kinds of cognitive biases from a mechanistic perspective. First, we discuss cases where informational encapsulation prevents knowledge from affecting our perceptions and emotional states— a set of biases we refer to as encapsulated biases. Encapsulated biases occur because our reflective awareness of our own biases is unable to cognitively penetrate the information processing that leads to our biased perceptions or behaviors. As we review below, biases can be encapsulated representationally— because reflective knowledge states can’t penetrate the specific representations that give rise to a bias— or emotionally— because conscious reflection is unable to penetrate a strong emotional state, making it nearly impossible for us to successfully apply that knowledge. We next contrast encapsulated biases with a second class of errors, one that we refer to as attentional biases. As we explain below, attentional biases 5 RUNNING HEAD: G.I. JOE PHENOMENA involve cases in which knowing information could make us less likely to commit mistakes in theory, but in practice we often fail to attend to the relevant information in order to make better decisions in the moment. We then discuss the implications of these two novel categories for debiasing efforts, arguing that the mechanisms underlying these biases necessitate that one category of biases will be much harder to overcome than the other . The recommendations that stem from our bias categories channel Soll, Milkman and Payne (2015)’s “User’s Guide to Debiasing,” which builds off of Fischhoff (1981)’s classic approach to debiasing by suggesting that one can attempt to modify the decision-maker or modify the environment. We will show that for encapsulated biases modifying the decision-maker will be difficult at best, and futile at worst. We will end our review by explaining why a better approach will involve educating the decision-maker to modify their environment differently depending on which class of biases they are tackling. The heart of the argument we propose in this paper is that the biases that plague human decision-making and are immune to conscious situational knowledge do not fall into a single category. Although scholars have attempted to classify biases into categories previously (Arkes, 1991; Fischhoff, 1981; Stanovich, 2009; Wilson & Brekke, 1994), the current review provides a novel conceptualization based on the mechanisms that drive these biases, arguing that seemingly different biases fall into two relatively distinct natural kinds based on their underlying cognitive architectures. This new framework is important as it allows for some important implications, both for how to understand the way these biases operate mechanistically as well as how to design effective interventions to overcome them. 2. Encapsulated biases One of the key drivers of G.I. Joe phenomena is the cognitively impenetrable nature of certain biases: information from other cognitive domains simply cannot interact with the representations and emotional states that lead to our biased behaviors. To use Fodor’s (1990) term, many biases are informationally encapsulated from conscious reflection. Encapsulated 6 RUNNING HEAD: G.I. JOE PHENOMENA biases have two distinct flavors, which are based on the content that cannot be accessed by conscious reflection. Some biases occur because the cognitive representations that give rise to a bias can’t be penetrated by knowledge from another domain. As we review below, a classic case of representational encapsulated biases are visual illusions— even in cases where we know how an illusion works, our knowledge cannot cognitively penetrate the representations that lead us to perceive that illusion (Fodor, 1990). Other biases occur because cognitive reflection is unable to penetrate the affective states that give rise to that bias. These emotionally encapsulated biases occur because the emotional states that cause biased behavior are cognitively impenetrable, at least during the heat of the moment. 2.1 Representationally encapsulated biases We begin by discussing representationally encapsulated biases— cases where we behave in a biased way because the representations that cause our biases are cognitively impenetrable to information from other domains. 2.1.1 Visual illusions. Visual illusions epitomize the phenomena of informational encapsulation because learning how they work rarely affects the percepts we experience when viewing them.