A Framework for Studying Biases in Visualization Research
Total Page：16
File Type：pdf, Size：1020Kb
A Framework for Studying Biases in Visualization Research Andre´ Calero Valdez* Martina Ziefle† Michael Sedlmair‡ Human-Computer Interaction Center Human-Computer Interaction Center Visualization & Data Analysis Group RWTH Aachen University RWTH Aachen University University of Vienna ABSTRACT such decisions, visual inspection and communication of the under- lying data and models is often an essential component. Ideally, the In this position paper, we propose and discuss a lightweight frame- data and models are mapped to visual presentations that are easy to work to help organize research questions that arise around biases in process and easy to convert into mental representations. The visual visualization and visual analysis. We contrast our framework against presentations should be as accurate as possible. Or as as Edward cognitive bias codex by Buster Benson. The framework is inspired Tufte put it: Keep the lie factor between 0:95 and 1:05 [31]. So in by Norman’s Human Action Cycle [23] and classifies biases into theory, accurately presenting information with respect to the visual three levels: perceptual biases, action biases, and social biases. For system should yield accurate decisions. each of the levels of cognitive processing, we discuss examples of biases from the cognitive science literature, and speculate how they Still, our saber-tooth-fearing minds interfere. Not only is the might also be important to the area of visualization. In addition, we visual system imperfect, our cognitive system has its pitfalls as well. put forward a methodological discussion on how biases might be Even when a system provides information perfectly honest, human studied on all three levels, and which pitfalls and threats to validity biases might distort the information and lead to imperfect or outright exist. We hope that the framework will help spark new ideas and bad decisions. For example, a business person might invest further discussions on how to proceed studying the important topic of biases into a project that had already cost more than expected, as the relative in visualization. prospective investment to finalize the project appears smaller than the retrospective cost of not completing the project. The sunk cost Index Terms: H.1.2 [Models and Principals]: User/Machine fallacy is the reason why many publicly funded projects cost more Systems—Human information processing; H.5.2 [Information Inter- than previously anticipated. Nobody likes to tear down the already faces and Presentation]: User Interfaces—Evaluation/methodology overpriced 80%-complete 100 million dollar airport. We might just invest another 10 million dollars to complete it—and then another. 1 INTRODUCTION Could an accurate visualization have helped the business person? Should it have overemphasized the additional costs? “Look, the earth is flat. I can see it with my own eyes.” At sea-level, the curvature of the earth is too small to be perceivable to the human The body of research on such biases is extremely large. Since eye. The illusion of a flat earth is no hallucination. It is a limitation Tversky and Kahnemann received a Nobel price for their work on of the perceptual system. Yet, the realization that our planet is biases in 2002, research regarding biases has sprouted into all kinds (relatively) spherical dates back to the early Greek philosophers of fields. From distortions in perception to distortions of complex around 600 BC. And the realization did not occur due to paying social phenomena, the spectrum of biases is very wide. A systematic more attention to the visual impression, it came due to considering (reduced) overview of the most prominent biases can be seen in mathematical observations about the rotation of the night-sky and Fig. 1. In this figure, biases are classified in three levels of hierarchy. bodies of water—through science. The first level separates the assumed high-level reasoning behind The scientific method was devised to investigate natural phenom- the existence of the biases. All of them are rooted in the limited ena that are hidden from human sight. Either because they were too perceptual and memory-related capabilities. There is either too much small, too large, too fast, too slow or too rare for human perception. information available, too little meaning in our model, too little time The human body and thus its perceptual system were crafted by to integrate the information, or too much information to memorize. evolution to enable survival of a primate in the savanna. Capabilities The second level of ordering describes strategies to cope with like objective measurement or accurate judgment of the external this reasoning. Each strategy leads to several different distortions or world are neither necessary nor helpful for survival. Being able biases. For example, the availability heuristic (see Fig. 1 at I.a.1.), to make decisions quickly with limited information and limited re- describes the phenomenon that we assume things to be more frequent sources could make the difference between death by saber-tooth or or important depending on how easy, or how available our mental last-minute escape. Therefore, the human mind is equipped with recollection of them is [32]. It’s much easier remembering the 911 heuristics that help decision making with the aim of survival. attack on the World Trade Center, than a toddler drowning in a home Todays world is drastically different! Yet, human perceptual swimming pool. This leads to a misconception. People overestimate and decision making processes remain largely unchanged. People the risk of becoming a victim of a terrorist attack and underestimate nowadays have to deal with different types of information, different the risk of drowning in a swimming pool. Another example: The amounts of information, and make much more delicate decisions. Dunning Kruger effect (see Fig. 1 at III.a.12.) describes the phe- Decisions, such as quickly detecting a critical pattern in an x-ray nomenon that people with little experience in a subject overestimate image, can make the difference between life and death. Decisions, their knowledge in that subject, while people with lots of experi- such as stock-investments derived from numbers displayed on a ence underestimate their knowledge: “The more I learn, the more computer screen, can influence the global economy. To gain trust in I realize how much I don’t know”. The anti-vaxxer thinks he has understood the required field of medicine to evaluate the efficacy *e-mail: [email protected] of vaccinations, while the medical researcher carefully considers †e-mail:ziefl[email protected] different explanations and possible errors in their experimental setup. ‡e-mail:[email protected] However, in such a scenario many other biases are at play. These examples could easily benefit from visualizations depicting the real data. But, what if even with high quality visualizations biases persist. There is little research on biases in the field of visu- alization [8, 10, 33]. Most of the aspects that have been addressed, IV. What Should We I. Too Much Information II. Not Enough Meaning III. Need to act fast remember? a. We notice things 1. Axxxxxxxxxxxvailability Heuristic a. We find stories and 1. Cxxxxxxxxxxxonfabulation a. To act, we must be 1. Pexxxxxxxxxxxltzman effect a. We edit and 1. Sxxxxxxxxxxxpacing effect 2. Axxxxxxxxxxxttentional Bias 2. Cxxxxxxxxxxxlustering illusion 2. Rxxxxxxxxxxxisk compensation 2. Sxxxxxxxxxxxuggestibility already primed in 3. Ixxxxxxxxxxxllusory truth effect patterns even in 3. Ixxxxxxxxxxxnsensitivity to sample size confident we can 3. Efxxxxxxxxxxxfort justification reinforce some 3. Fxxxxxxxxxxxalse memory memory or repeated 4. Mxxxxxxxxxxxere exposure effect sparse data 4. Nxxxxxxxxxxxeglect of probability make an impact and 4. Trxxxxxxxxxxxait ascription bias memories after the 4. Cxxxxxxxxxxxryptomnesia xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx xxxxxxxxxxx often 5. Centext effect 5. Anecdotal fallacy feel what we do is 5. Defensive attribution fact 5. Source confusion 6. Cxxxxxxxxxxxue-dependent forgetting 6. Ixxxxxxxxxxxllusion of validity hypothesis 6. Mxxxxxxxxxxxisattribution of memory 7. Mxxxxxxxxxxxood-congruent memory bias 7. Mxxxxxxxxxxxasked man fallacy important 6. Fuxxxxxxxxxxxndamental attribution error 8. Fxxxxxxxxxxxrequency illusion 8. Rxxxxxxxxxxxecency illusion 7. Acxxxxxxxxxxxtor-observer bias 9. Bxxxxxxxxxxxaader-Meinhof Phenomenon 9. Hxxxxxxxxxxxot-hand fallacy 8. Sexxxxxxxxxxxlf-serving bias 10.xxxxxxxxxxxEmpathy gap 10xxxxxxxxxxx. Illusory correlation 9. Laxxxxxxxxxxxke Wobegone effect 11.xxxxxxxxxxxOmission bias 11xxxxxxxxxxx. Pareidolia 10. Ixxxxxxxxxxxllusory superiority b. We discard 1. Fxxxxxxxxxxxading affect bias 12.xxxxxxxxxxxBase rate fallacy 12xxxxxxxxxxx. Anthropomorphism 11. Hxxxxxxxxxxxard-easy effect specifics to form 2. Nxxxxxxxxxxxegativity bias 12. Dxxxxxxxxxxxunning-Kruger effect 3. Pxxxxxxxxxxxrejudice b. We fill in xxxxxxxxxxx xxxxxxxxxxx generalities xxxxxxxxxxx 1. Group attribution error 13. False consensus effect 4. Sxxxxxxxxxxxtereotypical bias b. Bizarre/funny/ 1. Bizarreness effect characteristics from 2. Uxxxxxxxxxxxltimate attribution error 14. Ixxxxxxxxxxxllusion of control 5. Imxxxxxxxxxxxplicit stereotypes 2. Hxxxxxxxxxxxumor effect 3. Sxxxxxxxxxxxtereotyping 15. Bxxxxxxxxxxxarnum effect 6. Imxxxxxxxxxxxplicit associations visually-striking/ 3. Vxxxxxxxxxxxon Restorff effect stereotypes, 4. Exxxxxxxxxxxssentialism 16. Fxxxxxxxxxxxorer effect anthropomorphic 4. Pxxxxxxxxxxxicture