Please don’t delete empty pages; they keep page numbering in order for double-sided printing. VISUALISING UNCERTAINTY A SHORT INTRODUCTION

A PUBLICATION OF THE ANALYSIS UNDER UNCERTAINTY for DECISION MAKERS NETWORK

Written by Polina Levontin and Jo Lindsay Walton

With additional contributions from Lisa Aufegger, Martine J. Barons, Edward Barons, Simon French, Jeremie Houssineau, Jana Kleineberg, Marissa McBride, and Jim Q. Smith

Layout and design by Jana Kleineberg | Kleineberg Illustration & Design

Published in 2020 by AU4DM, London, UK © Sad Press isbn: 978-1-912802-05-0

This edition was compiled with the support of the Sussex Humanities Lab. With special thanks to Mark Workman of AU4DM.

The Analysis Under Uncertainty for Decision Makers Network (AU4DM) is a community of researchers and professionals from policy, academia, and industry, who are seeking to develop a better understanding of decision-making to build capacity and improve the way decisions are made across diverse sectors and domains. For further details about the network, visit http://www.au4dmnetworks.co.uk/.

Copyright This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without written permission of the rights holders.

Recommended citation Levontin, P., Walton, J.L., Kleineberg, J., Barons, M., French, S., Aufegger, L., McBride, M., Smith, J.Q., Barons, E., and Houssineau, J. Visualising Uncertainty: A Short Introduction (London, UK: AU4DM, 2020). 2 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

FUTURE COLLABORATIONS AND CONTACT

• Jana Kleineberg, Kleineberg Illustration & Design http://www.kleineberg.co.uk/ [email protected] • Polina Levontin [email protected] or [email protected] • Jo Lindsay Walton https://www.jolindsaywalton.com/ [email protected] • Or contact us here http://www.seaplusplus.co.uk/ http://au4dmnetworks.co.uk/ CONTENT VISUALISING UNCERTAINTY

Frontmatter Colophon...... 1 Contact the Authors ...... 2

1. Summary Visualising Uncertainty ...... 4 2. Introduction The Influence of Visualisation ...... 6 3. The Challenges The State of Visualisation Research ...... 8 4. The Framework Developing Visual Solutions ...... 10 5. Types of Uncertainty Categorising Uncertainty ...... 14 6. Cynefin ...... 16 7. Visualising Uncertainty The Toolkit ...... 18 8. The User ...... 26 9. Concluding Remarks ...... 40 10. Bibliography ...... 42 11. Resources ...... 51

Appendix A—Uncertainty Representations ...... 52 Appendix B—Charts, Graphs, Symbols, Metaphors ...... 54 AU4DM Network Colleagues, and special thanks...... 56 4 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

SUMMARY 1 VISUALISING UNCERTAINTY The basis of this short introduction is In order to develop an uncertainty a review of the literature on commu- visualisation format for a case study, nicating uncertainty, with a particu- we must distinguish the different We must distinguish lar focus on visualising uncertainty. types of uncertainty that we believe the different From our review, one thing emerges to be present (see ‘Types of Uncer- ‘Types of Uncertainty’ very clearly: there is no ‘optimal’ tainty’). It is important to make these that we believe format or framework for visualis- distinctions as clearly and as early as to be present. ing uncertainty. Instead, the imple- possible. Definitions and understand- mentation of visualisation techniques ings of uncertainty should also be must be studied on a case-by-case regularly reviewed as the design and See chapter 5, basis, and supported by empirical testing process unfolds. ‘Types of Uncertainty’ testing. Without reliable evaluation meth- Developing solutions for uncer- ods, there is the danger of developing tainty visualisation thus requires dazzling, seductive visualisations that interdisciplinary expertise. The fail to deliver appropriate decision effectiveness of different techniques is support, or even subvert decision- highly context-sensitive, and current making by slowing it down and/or understanding of how to differentiate introducing biases. Self-reporting by relevant contextual factors remains users is not a reliable way to assess the patchy. For this reason, communi- effectiveness of a visualisation format, cation formats should ideally be so evaluation should be performed developed through close collabora- by objective and reproducible tion among researchers, designers, methodologies. and end-users. The building blocks Users of uncertainty information brought together here provide a start- have diverse capacities and needs, ing point for these kinds of dialogues. and there is as yet no deep theory which formalises which differences

Key ideas case-by-case basis review & testing development through collaboration reliable evaluation i different types of uncertainty reproducible methodologies 5

There is no ‘optimal’ format or framework for visualising uncertainty.

are relevant in a given case. This is possible are continually growing. important, because how we visualise In the longer term, expanding the uncertainty is not easily separable repertoire of uncertainty visualisation from how we interpret and reason formats, using robust methodology about that uncertainty. The diverse on a case-by-case basis, will improve identities of decision-makers—our the analysis and communication of various cultural, political, social, uncertainty across a broad spectrum linguistic, institutional, and indi- of decision-making contexts. vidual characteristics—shape our Finally, we must remember that practices in dealing with uncertainty. visualisation is not always the most This means that effective uncertainty appropriate tool. Sometimes words visualisation must ideally do more or numbers do a better job of convey- than encode all the relevant informa- ing a particular type of uncertainty, tion: it must also invite, reinforce, and to a particular audience, in a particu- sometimes even teach appropriate lar context, for particular purposes. modes of interpretation and reason- At the same time, we should be ing. This catalogue offers illustrative conscious that there is almost always discussion of selected heuristics and some visual dimension involved in biases, and how they can interact with how we analyse and share informa- visualisation techniques. This gives a tion, how we design policies and flavour of a vast area of research, and processes, and how we imagine and helps to illuminate some key features plan for the future. Even when visu- of the existing evidence base. alising uncertainty is not the main Although there are many chal- focus, being aware of the visual lenges associated with visualising dimensions of a decision may be help- uncertainty for decision-making, ful in understanding and managing there are also many potential its uncertainties. benefits. In fact, the horizons of the 6 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

INTRODUCTION 2 THE INFLUENCE OF VISUALISATION Decision theory is the study of how future generations or non-human careful design and testing on a case- choices are made. It is primarily stakeholders). by-case basis. Visualisation can grounded in economics, statistics, Classifying, quantifying, and potentially influence decision-making and psychology, but also benefits reasoning about uncertainty is central in many ways, including: from the insights of sociology, to decision analysis. So too is commu- computer science, environmen- nicating about uncertainty. It has long • Whether or not a tal science, design theory, the arts been clear that not just what, but decision is made and humanities, and other exper- how we choose to communicate has • Which stakeholders input tise. The prescriptive side of deci- considerable influence on the inter- into the decision sion theory—often called decision pretations and actions that follow. • How evidence is prioritised analysis—provides many concep- Uncertainty can be communicated tual resources to specify decisions in different formats, including verbal • Whether additional formally and to create recommended descriptors (“high confidence”), evidence is sought numerical ranges, statistical graphics courses of action. What counts as • What forms of reasoning (e.g. probability density functions), a ‘rational’ decision may vary from and analysis are used context to context, but decision pictograms, infographics, or various • Whether biases are active and analysis can help decision-makers to combinations. How uncertainty is the extent of their influence better fulfil their chosen standards of communicated can be fundamental to rationality. For example, if the future the effective transfer of information • How goals are formulated circumstances around a decision can between individuals, agencies, and and success evaluated be divided into scenarios, each of organisations, and is thus crucial to decision-making. Visualisation may • How much workload which can be assigned a probability, decision-making causes then a decision-maker can choose a therefore play an especially significant course of action which is robust across role in analysis and decision-making • How long a decision takes involving multiple stakeholders. But all those scenarios—instead of just • Correctness of a decision optimistically hoping for the best case even when only one person is respon- • Confidence in a decision scenario, or pessimistically steeling sible for every aspect of analysis and themselves against the worst. Deci- decision-making, visualisation may • Kinds of errors made sion analysis also allows us to anato- still play a significant role in how uncertainty is understood—i.e. in • How decision outcomes mise a problem into its more tractable are interpreted constituents, separating the techni- how the decision-maker ‘communi- To support a decision, multi- cal from the value aspects. Then, it cates uncertainty to themselves.’ dimensional uncertainty requires a helps us to identify the role played In many domains, there are already representation that humans (acknowl- by ‘preferences’ that must be gener- tried-and-true methods for classify- edging the variation in our capabili- ated by stakeholders through political ing, quantifying, and propagating ties) find natural to understand and processes, under conditions of more uncertainty in statistically robust operate. Decision-makers often use or less ethical legitimacy (or inferred ways. However, presenting uncer- heuristics to assess uncertainty, and by other means, e.g. in the case of tainty to decision-makers requires 7

Frequently, the challenge is to present visualisations in ways which allow the user intuitively to perceive the uncertainty probabilistically and arrive at effective decisions.

such assessments are often incom- Negative effects of visualisation Training and experience of deci- patible with probability theory, i.e. reported in the literature point to sion-makers plays a role in determin- ‘rational’ treatment of uncertainty. interactions with cognitive biases and ing the impact of a particular visuali- Frequently, the challenge is to present human psychology more generally, sation. Kinkeldey et al. (2015) write visualisations in ways which allow the resulting in delays in decision-making that ‘in order to use data and related user intuitively to perceive the uncer- when extra uncertainty-related infor- uncertainty effectively, users must tainty probabilistically and arrive at mation needs to be processed; irra- know how to interpret data together effective decisions. tional attitudes to risk such as focusing with related uncertainty.’ Boone et Visualisation can be used to coun- on the worst-case scenarios, or focus- al. (2018) conducted experiments teract known biases, such as anecdotal ing on the mean rather than variance; on whether additional training on evidence bias (Fagerlin et al., 2005), confusion between risk and uncer- how to interpret specific graphical side effect aversion (Waters et al., tainty; etc. Several studies explore conventions for uncertainty visuali- 2006, Waters et al., 2007), and risk how visualisation affects cognitive sations could reduce flaws in deci- aversion (Schirillo & Stone, 2005). strategies for dealing with uncertainty, sion-making. Using visualisation to The effects of visualising uncertainty and find that visualisations might lead express a hurricane’s current location are frequently positive—but not to less effort being expended by the and projected path, together with always. Riveiro et al. (2014) write that decision-maker on acquiring and uncertainty, they found that training ‘even if the effects of visualizing uncer- processing information relevant to can reduce misconceptions. However, tainty and its influence on reasoning uncertainty. This can be an undesir- one experiment also revealed an are not fully understood, it has been able consequence: Riveiro et al. (2014) unexpected side-effect: reduced shown that the graphical display of report a study where visualisation of incidence of misinterpretation was uncertainty has positive effects on uncertainty in a military scenario led correlated with lowered risk percep- performance.’ Kinkeldey et al. (2015) to significantly fewer attempts to iden- tion (decreased estimates of hurricane observe: ‘Overall, based on studies tify a target, as well as higher threat damage) relative to the group that did reviewed, uncertainty visualization values assigned to uncertain targets. not receive training. Such evidence has tended to result in a positive effect This is sometimes explained by visu- from past experiments reinforces the on decision accuracy. The evidence alisations triggering an ‘availability imperative to test new visualisation is less clear for decision speed, but it heuristic,’ which puts undue emphasis formats whenever possible. could be observed that usually, uncer- on events that are readily imagined tainty visualization does not slow or easy to recall—such as the worst- down decision-making and in one case case scenario—in disproportion to the it even decreased the decision time.’ chance of occurrence.

Key ideas decision theory stakeholders’ preferences decision analysis multi-dimensional uncertainty robustness reduction of misconceptions through training i 8 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

THE CHALLENGES 3 THE STATE OF VISUALISATION RESEARCH While there is solid evidence that small effect sizes when comparing offer evidence that self-reporting is visualisation influences decision- different visualisation approaches; unreliable, i.e. the user of a visualisa- making, this influence is not biased self-perception (with little tion is not necessarily a good judge of uniformly positive, and theoreti- correspondence between self- how well or poorly the visualisation cal models do not provide sufficient reported impact and actual impact has supported their decision-making. basis to predict case-specific impacts. of visualisations); and transferability This finding goes against the grain of Reviewing the state of knowledge in issues, confounded by the difficulty a typical designer-client relationship, 2005, MacEachren et al. concluded in controlling for differences in indi- in which the designer has done a good that ‘we do not have a comprehensive vidual interpretation. Transferability job if the client is satisfied. In fact, understanding of the parameters that is an especially salient problem since client satisfaction may have little to influence successful uncertainty visu- it implies that studies done with one do with the efficacy of the product: alization, nor is it easy to determine group of people may not be applica- decisions can be improved by visuali- how close we are to achieving such an ble for another, or that the results are sations the client does not favour, or understanding.’ In a follow-up to that reproducible in general. Even on an can be impaired by the visualisations review in 2016, Riveiro asserted that individual level, responses to a visu- the client happens to prefer. As often this conclusion is still valid. This is alisation may vary with factors such happens with good design, the bene- not a reflection of the paucity of stud- as time or stress-inducing constraints; fits may not be noticeable to users. ies, as visualisation is a burgeoning the same visualisation may therefore Several studies have reported that the research topic across diverse fields. influence decision-making in one users of visualisations may become Indeed, a major difficulty in devel- way under a particular set of circum- better at decision-making without oping a broadly applicable theory is stances and in a contrary way under realising it. Kinkeldey et al. (2015) the sheer variety of methodological another. mention one study that revealed a approaches and theoretical frame- When it comes to testing visu- striking lack of correlation between works. Other common problems in alisation formats, there are broadly independently measured perfor- the literature include small sample two types of investigations: objective mance and self-reported confidence sizes for audience testing (statistical (which rely on measured outcomes in making decisions: ‘decision accu- significance); inappropriate subjects such as decision speed and decision racy was significantly higher with (students rather than relevant deci- accuracy) and subjective (which rely uncertainty depicted, meaning that sion-makers); lack of reproducibility; on self-reporting). Several studies user performance and confidence did

Known problems small sample sizes small effect sizes inappropriate subjects biased self-perception i lack of reproducability transferability 9

We do not have a comprehensive understanding of the parameters that influence successful uncertainty visualization, nor is it easy to determine how close we are to achieving such an understanding. (MacEachren et al., 2005)

not correspond.’ The evidence point- et al. (2018), are optimistic: ‘Visu- out that ‘decision-makers often have ing to users’ inability to assess their alizations have the potential to influ- little time to explore uncertainty in own performance is broader than just ence how people make spatial predic- the data.’ In order to ensure that deci- research on visualisation, as Hullman tions in the presence of uncertainty. sion-makers prioritise uncertainty et al. (2008) point out: ‘evidence from Properly designed and implemented information appropriately, the devel- other disciplines suggests that people visualizations may help mitigate opment of visualisation formats must are not very good at making accurate the cognitive biases related to such be placed in a much wider context. judgments about their own ability to prediction.’ This wider context includes the make judgments under uncertainty.’ Of course, creating visualisa- graphic literacy of decision-makers, Even the level of expertise inter- tions that communicate uncertainty the available ensemble of decision acts with visualisation in ways which well is not necessarily a guarantee support tools, and the surrounding are difficult to anticipate. Greater for effective decision support; even technical, social, and institutional expertise has been associated both when understood correctly, uncer- infrastructures. positively and negatively with the tainty may be eschewed by decision- In short, what is lacking currently effectiveness of visualisation. For makers. Uncertainty is not some- is the ability to predict how specific example, in some studies a higher thing people are comfortable with no visualisations will impact particular level of expertise enabled decision- matter how well it is communicated. decision-making processes. Uncer- makers to make better use of visuali- Resistance towards incorporating tainty visualisation can be reward- sation, arriving at decisions faster and uncertainty into decision-making is ing, but it is also a challenging and with greater accuracy. In others, the widely reported in the literature. For unpredictable territory. This is not level of expertise was associated with example, Riveiro et al. (2014) report to say that experience and existing undesirable effects of visualisations; that ‘participants’ greater uncer- research offer no guidance whatso- experiments ‘showed that participants tainty awareness was associated with ever, but to emphasise that any new with a high level of experience had lower confidence.’ Lower confidence, visualisation needs to be tested and the strongest bias towards selecting however, may be a desired outcome of evaluated in its intended context and areas of low uncertainty’ (Kinkeldey visualisation in contexts where over- with the relevant audience, using et al., 2015). confidence is a known problem. robust methodology with an objec- Currently, much of the research Kinkeldey et al. (2015) recommend tive component (not just subjective on uncertainty visualisation relates that decision-makers are supported appraisal). With these caveats in to spatial reasoning. This research with more than just well-crafted visu- mind, the following sections present addresses the potential of visualisa- alisations: ‘decision makers needed some basic approaches and practical tion to improve probabilistic spatial additional information for interpret- examples of visualising uncertainty reasoning which, like all probabilis- ing and coping with uncertainty (e.g. for decision support. tic reasoning, is plagued by cognitive when a high degree of uncertainty is biases. Some researchers, like Pugh a problem and when not).’ They point 10 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

THE FRAMEWORK DEVELOPING VISUAL SOLUTIONS

4 other hand, although designers and ‘First, it is assumed that uncer- Before we begin discussing the foun- dational elements that can be called researchers must cultivate a deep tainty, or at least uncertainty of upon in the visualisation task, let’s understanding of decision-makers’ interest, is both knowable and iden- emphasise that selecting a visualisa- lived experience, they must also work tifiable. Similarly, to be visualized, tion method is not the first step. The with decision-makers to explore how uncertainty must be quantifiable, process itself should begin with the this experience may be misleading, such as through statistical estimates, identification of uncertainty, under- once information from objective quantitative ranges, or qualitative standing of the various components testing has been incorporated. The statements (e.g. less or more uncer- that contribute to uncertainty, and literature on bounded rationality, tain). Moreover, evaluations define discussing the aims of visualisa- heuristics, and cognitive biases offers effectiveness as an ability to identify tion. We recommend considering useful concepts in this regard (see specific uncertainty values, which the framework below or a close section 08 “The User”). assumes that identifying specific equivalent. The following twelve-step guide uncertainty values is useful to deci- As emphasised in the previous demonstrates how design thinking sion-makers and that the values of section, self-reporting is an unreli- can be implemented in the domain of interest can be quantified. Lastly, able method for testing uncertainty uncertainty visualisation. It is largely there is an assumption that the quan- visualisations. Decisions may be based on A. Lapinsky’s ‘Uncertainty tification of uncertainty is beneficial, improved by visualisations the user Visualization Development Strategy applicable to the decision task, and does not favour, or may be impaired (UVDS).’ usable by the decision maker, even by the visualisations the user regards Step 1 is to classify the nature of if users do not currently work with as helpful. This can create additional the uncertainty. We explore possi- uncertainty in that way.’ challenges within the design think- ble approaches in the next section. These pervasive assumptions ing process. On the one hand, as Whatever the approach chosen, it mean that deep uncertainty, i.e. Deitrick and Wentz (2015) warn, will likely reveal that only a portion uncertainty that cannot be quanti- visualisation research has often been of uncertainty lends itself to visuali- fied given available resources, poses ‘normative in nature, reflecting what sation. Deitrick and Wentz (2015) special challenges for visualisation. researchers think decision makers list common assumptions about the Following Step 1, Steps 2 to 9 are need to know about uncertainty,’ conditions under which uncertainty the research phase, subdivided into instead of setting aside preconcep- can be effectively visualised: ‘Understand,’ ‘Decide,’ and ‘Deter- tions and building empathy. On the mine.’ Step 2 ensures that the data

Key ideas self-reporting is unreliable visualisation research shout set aside preconceptions and build empathy i literature on bounded rationality, heuristics, cognitive biases offers useful concepts 11

…although designers and researchers must cultivate a deep understanding of decision-makers’ lived experience, they must also work with decision-makers to explore how this experience may be misleading

Figure 01. 12-Step Strategy for Uncertainty Visualisation. Based on the Uncertainty Visualization Development Strategy (UVDS) by Anna- Liesa S. Lapinsky (2009). Created by Jana Kleineberg.

itself is understood, and takes into use the uncertainty construct? How and make decisions? Some other account such things as the origin is it used? How will the visualisation purpose(s)? Step 5 helps decide on and precision of the data, whether help? It is important to determine an information hierarchy, if multiple it has been modified (e.g. processed the purpose for which presentation is uncertainties are associated with the or aggregated), its format and the required. Is it to explore the problem? inputs. Step 6 formalises a definition format’s limitations, as well as other To facilitate understanding between of the uncertainty. Step 7 looks at the factors. Steps 3 and 4 look at the different stakeholders, roles, or forms specific cause of uncertainty. Step 8 intended target audience: who will of expertise? To analyse a situation determines causal categories. Step 12 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

Decisions may be improved by visualisations the user does not favour, or may be impaired by the visualisations the user regards as helpful.

9 determines visualisation require- in Step 11 does the creative part of themselves, employing evaluation ments, or the needs of the visualisa- the visualisation start. The principle methodologies that are transparent tion — e.g. what should the dominant of appropriate knowledge and the and reproducible. Steps 10 to 12 are features be, what level of understand- semantic principle provide guidance connected, as they may be repeated ing does the user have, and how will here. Different techniques can be multiple times to refine and improve this influence the necessary level of used to create visualisation formats, on the visualisation. detail? What tasks does the user need encoding information in ways likely Following general semantic and to perform, and what information is to support effectively attention and other design principles is no guar- relevant to these tasks? analysis. For example, a saliency antee that visualisations will be Step 10 leads into the actual design algorithm (Padilla, Quinan, et al., correctly interpreted. Rather, a holis- process by preparing the data for 2017) can optionally be used to iden- tic approach to uncertainty visualisa- visualisation: sorting and organis- tify elements likely to attract viewers’ tion includes principles of design, the ing measurements, converting them attention. Once candidate visualisa- testing of visualisation methods, the if necessary, converting uncertainty tion formats have been developed, graphic literacy of end-users, as well as from collected data, and/or combin- these are then tested in Step 12, the overall decision-making context. ing multiple uncertainties. Only ideally with the intended end-users

DESIGN THINKING

In graphic design the ‘design thinking’ approach Empathy: Understanding human needs; learning has been widely adopted. It refers to the cognitive, about the user who will interact with the design. strategic, and practical processes designers use to Definition: Framing and defining the problem; tackle complex problems. It is a flexible approach, shaping a point of view based on user needs and focused on collaboration between designers and insights. users, with an emphasis on bringing ideas to life Ideation: based on how intended users think, feel, and Creating ideas and creative solutions in behave. ideation sessions, e.g. through brainstorming. The process is carried out in a non-linear fashion: Prototyping: Adopting a hands-on approach by the five stages are not always sequential and can building (a) representation(s) to show to others. often occur in parallel and/or iteratively. However, Testing: Developing a solution to the problem the design thinking model identifies and systema- and returning to the original user group for testing tises the five stages one would expect to carry out in and feedback. Results are used to review the empa- a design project. thy stage, redefine problems, and refine the design. 13 DEEP UNCERTAINTY Deep uncertainty may refer to uncertainty which we cannot quantify, which we cannot quantify given our available resources, or whose quantification is on balance undesirable. So the distinction between uncertainty and deep uncertainty is not rigid, but rather is a function of the methods, resources, and choices we bring to bear on classifying and quantifying uncertainty. Identifying some uncertainties as deep uncertainties does not place them beyond quantification once and for all, and does not rule out the possibility that they may be reclassified at a later stage in the process. Furthermore, adopting the designation of deep uncertainty does not mean that decision-makers are justified in excluding these uncertainties from their reasoning, or that developers of decision support tools can safely set them to one side. Indeed, it is even theoretically possible to visually depict deep uncertainty: many artworks (e.g. the abstract impressionist works of Rothko) might be considered representations of deep uncertainties that are perceived intuitively. However, in line with the majority of research done to date, this primer focuses on uncertainties which can be quantified. 14 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

TYPES OF UNCERTAINTIES 5 CATEGORISING UNCERTAINTY WHY CATEGORISE?

Decision-making can be improved by Uncertainty has been decomposed understanding the uncertainty in the in a variety of ways by a multitude data being used. Categorising uncer- of theoretical treatments, and each tainty is a preliminary step towards classification scheme carries its own recognising and dealing with uncer- set of assumptions and motives. In tainty in the decision-making process. particular, sometimes uncertainty is Uncertainty can come in many broken up into different types simply forms, and with many different quali- to help us spot where uncertainty lies, ties. Each type of uncertainty applies and to organise how we investigate to different types of information, and and manage uncertainty, but on the may be quantified, and thus repre- understanding that these distinctions sented, in different ways. However, don’t ultimately prevent integration there is no one-to-one correlation of into a single analysis and decision uncertainty types and visualisation procedure. On other occasions, the techniques. As Chung and Wark purpose of a classification scheme (2016) confirm, often the same tech- is to draw attention to more funda- nique, e.g. colour coding, has been mental differences between uncer- variously applied to depict distinct tainty types, which may be difficult types of uncertainty, as well as subsets or impossible to reconcile. of combined/compounded uncertain- ties from different categories.

Key ideas categorizing is key uncertainty comes in many different forms & qualities i understanding is tailored to context 15

Uncertainty can come in many forms, and with many different qualities.

SOME PROMPTS

French et al. (2016) list the follow- • Accuracy the difference between ing types of uncertainty: stochastic observation and reality uncertainties (i.e. physical random- • Precision the quality of the ness), epistemological uncertain- estimate or measurement ties (lack of scientific knowledge), endpoint uncertainties (when the • Completeness the extent to which required endpoint is ill-defined), information is comprehensive judgemental uncertainties (e.g. • Consistency the extent to which setting of parameter values in codes), information elements agree computational uncertainties (i.e. • Lineage the pathway through inaccurate calculations), and model- which information has been passed ling errors (i.e. however good the model is, it will not fit the real world • Currency the time perfectly, or if it seems to, it is likely span from occurrence to to have little predictive power). There information presentation are further uncertainties that relate • Credibility the reliability to ambiguities (ill-defined meaning) of the information source and partially formed value judge- ments; and then there are social and • Subjectivity the extent ethical uncertainties (e.g. how expert to which the observer recommendations are formulated and influences the observation implemented in society, what the ulti- • Interrelatedness the dependence mate ethical value of a decision and on other information all its consequences will be). Some • Experimental the width of a uncertainties may be deep uncer- random distribution of observations tainties; that is, within the time and data available to support the decision • Geometric the region within process, there may be little chance of which a spatial observation lies getting agreement on their evaluation The two categorisations presented or quantification. illustrate the breadth of approaches Chung and Wark (2016) list these to uncertainty, and suggest that the categories in their review of uncer- choice of a schema for understanding tainty visualisation literature: uncertainty might need to be tailored to the context. 16 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

Excerpt from “Decision Support Tools for Complex Decisions under Uncertainty,” edited by Simon French from contributions from many in the AU4DM network:

6 CYNEFIN Another approach to uncertainty of purposes, and in a different space certainty. In such contexts, decision- called Cynefin — a Welsh word for for another set of purposes. Acquiring making tends to take the form of habitat, and used here to describe the more information and/or conducting recognising patterns and responding context for a decision — categorises analysis may also shift a decision from to them with well-rehearsed actions, our knowledge relative to a specific one space into another. i.e. recognition-primed decision- decision. Cynefin roughly divides In the Known Space, also called making. Such knowledge of cause decision contexts into four spaces Simple, or the realm of Scientific and effect will have come from famil- (see figure 02). Note that placing a Knowledge, relationships between iarity. We will regularly have experi- decision in one of these four spaces cause and effect are well understood, enced similar situations. That means does not preclude certain aspects of so we will know what will happen if we will not only have some certainty that decision being associated with a we take a specific action. All systems about what will happen as a result of different space. It may also occasion- and behaviours can be fully modelled. any action, we will also have thought ally be appropriate to situate a deci- The consequences of any course of through our values as they apply in sion in a particular space for one set action can be predicted with near this context. Thus, there will be little ambiguity or value uncertainty in such contexts In the Knowable Space, also called REPEATABILITY ‘MESSY’ DECISIONS, Complicated, or the realm of Scientific AND INCREASED MANY UNCERTAINTIES Inquiry, cause and effect relation- FAMILIARITY ARE DEEP ships are generally understood, but for Simple / Known space Complex space any specific decision further data is needed before the consequences of any The realm of Scientific Knowledge, The realm of Social Systems, or action can be predicted with certainty. also called the ‘known knowns.’ domain of ‘unknown unknowns.’ The decision-makers will face episte- Rules are in place, the situation is Cause and effect are only obvious mological uncertainties and probably stable. Cause and effect relation- in hindsight and have unpredict- stochastic and analytical ones too. ships are understood; they are able, emergent outcomes. Decision analysis and support will predictable and repeatable. Chaotic space include the fitting and use of models Complicated / No cause and effect relationships Knowable space to forecast the potential outcomes can be determined. of actions with appropriate levels of The realm of Scientific Inquiry, uncertainty. Moreover, although the or domain of ‘known unknowns.’ decision-makers will have experienced Cause and effect relationships exist. such situations before they may be less They are not self-evident but can sure of how their values apply and will be determined with sufficient data. need to reflect on these in making the final decision. 17

Cynefin: Welsh, without direct translation into English, but akin to a place to stand, usual abode, and habitat. It is pronounced /´kλnίvίn / KUN-iv-in.

In the Complex Space, also called the realm of Social Systems, decision- making faces many poorly under- COMPLEX COMPLICATED stood, interacting causes and effects. Probe — Sense — Respond Sense — Analyse — Respond Knowledge is at best qualitative: there Emergent Good Practice are simply too many potential interac- tions to disentangle particular causes and effects. There are no precise disorder quantitative models to predict system behaviours such as in the Known and Knowable spaces. Decision analy- CHAOTIC SIMPLE sis is still possible, but its style will Act — Sense — Respond Sense — categorize — Respond be broader, with less emphasis on Novel Best Practice details, and more focus on exploring judgement and issues, and on develop- ing broad strategies that are flexible enough to accommodate changes as the situation evolves. Analysis may entities and predict their interac- Figure 02. Uncertainty Cynefin begin and, perhaps, end with much tions. The situation is entirely novel more informal qualitative models, to us. Decision-makers will need to sometimes known under the general take probing actions and see what heading of soft modelling or prob- happens, until they can make some lem structuring methods. Decision- sort of sense of the situation, gradu- makers will also be less clear on their ally drawing the context back into one values and they will need to strive of the other spaces. to avoid motherhood-and-apple-pie The central blob in figure 02 is objectives, such as minimise cost, sometimes called the Disordered For more information on improve well-being, or maximise Space. It simply refers to those categories of uncertainty and safety. contexts that we have not had time Cynefin, please see the AU4DM Contexts in the Chaotic Space to categorise. The Disordered Space Uncertainty catalogue “Decision involve events and behaviours beyond and the Chaotic Space are far from Support Tools for Complex our current experience and there are the same. Contexts in the former may Decisions under Uncertainty” no obvious candidates for cause and well lie in the Known, Knowable, or edited by Simon French from effect. Decision-making cannot be Complex Spaces; we just need to contributions from many in the based upon analysis because there recognise that they do. Those in the AU4DM network. are no concepts of how to separate latter will be completely novel. 18 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

VISUALISING UNCERTAINTY 7 THE TOOLKIT As noted, developing an uncertainty CLASSIFYING to represent how successful the policy visualisation solution begins with VISUALISATION will be for any given combination discussing the needs of the user, and TECHNIQUES of values. In the explicit version, understanding the various compo- There are a variety of classifica- by contrast, there is an underlying nents that contribute to uncertainty. tion schemes for visualisation tech- model to predict the outcome of Only once this has been done can niques used in uncertainty visualisa- each policy, and each visualisation assessment of appropriate visualisa- tion. Deitrick (2012) distinguishes encodes the model’s uncertainty in tion techniques begin. Furthermore, between implicit and explicit visu- the transparency/opacity dimension. since there is no general theory of alisation of uncertainty information. Experiments suggest that visualising uncertainty visualisation, whichever Uncertainty is implicitly visualised uncertainty implicitly versus explic- techniques we select will still need to when encoded in the image in such itly can impact decision-making in be tested, potentially through multi- a way that uncertainty cannot be divergent ways (Deitrick, 2012). In ple iterations. Testing should ideally separated out as a feature: that is, this catalogue, we generally focus on be undertaken with the actual end- explicit visualisations. no design element represents uncer- Kinkeldey et al. (2014) review users of the uncertainty visualisation. tainty by itself without also signify- Given well-attested problems with an array of studies in terms of how ing some other value. ‘Explicit visu- uncertainty is visualised, classifying self-reporting, testing should also not alization refers to methods where rely solely on users’ subjective experi- approaches to visualisation accord- uncertainty is extracted, modeled and ing to three theoretical dichotomies. ence. In summary, any toolbox sits quantified separately from the under- within a larger process of preparation (i) Coincident/adjacent distinguishes lying information’ (Deitrick, 2012). information represented together and testing. There are various credible As an example of implicit visuali- approaches to managing this process; with its uncertainty (coincident) from sation, Deitrick offers a scenario in information and associated uncer- this catalogue proposes an approach which a decision-maker is reviewing described in Section 4. tainty that are visualised separately three graphics associated with three (adjacent). (ii) Intrinsic/extrinsic policy options. In each graphic, the distinguishes visualisations achieved vertical and horizontal axes represent through manipulation of existing two variables whose future state is not graphical elements (intrinsic), from known, and the graph space is shaded

Key ideas coincident/adjacent selective attention & visual salience intrinsic/extrinsic using colour to visualise uncertainty i static/dynamic (introducing some examples) 19

Figure 03. Attributes of Graphics. Crated by Jana Kleineberg.

those derived through addition of *glyphs: In typography, a glyph is an Matthews et al. (2008) offer four elements such as grids, glyphs, and elemental symbol within an agreed nested categories of techniques: alter- icons (extrinsic). (iii) Finally, static/ set of symbols, i.e. an individual mark ation (1), addition (2), animation (3), dynamic refers to the potential of of a typeface, such as a letter, a punc- and interaction (4): a visualisation to change through tuation mark, an alternate for a letter. Free graphical variables (e.g. colour, time, as with animation or interactive A glyph is usually a mark that repre- size, position, focus, clarity, fuzzi- techniques. Pragmatic approaches by sents something else. For example, ness, saturation, transparency and Meredith et al. (2008) and Matthews the @ sign is a glyph that commonly edge crispness) can be used to alter et al. (2008) are motivated by the represents the word ‘at’. Thus, e.g. in aspects of the visualizations to question ‘what can be done to visu- flow fields, one would speak of glyphs communicate uncertainty. alise uncertainty?’ In their review of (usually pointing arrows) that show a the approaches to the visualisation of trend or a direction, as there cannot 1. Additional static objects (e.g. uncertainty, Meredith et al. (2008) be a direct pictorial representation labels, images, or glyphs) can be mention the following: of “flow”. added to the visualizations to communicate uncertainty. • adding glyphs* **icons: An icon is a more direct representation of something else; a 2. Animation can be incorporated • adding icons** pictogram or ideogram that shows a into the visualizations, where • adding geometry simplified, comprehensible symbol uncertainty is mapped to of the function or thing it represents. animation parameters (e.g. speed, • modifying geometry Icons are often recognisable depic- duration, motion blur, range or • modifying attributes tions of familiar objects, such as fish, extent of motion). • animation cars, or trees. The sections below will 3. Uncertainty can be discovered provide examples of glyphs and icons. by mouse interaction (e.g. • sonification mouse-over).’ • fluid flow There are only so many graphical • surface interpolants attributes that can be manipulated (figure 03). Various combinations • volumetric rendering of graphical parameters have been • differences in tree structures explored in the context of visual- ising uncertainty. Individual stud- ies and reviews of existing research offer valuable insight into their usefulness. Figure 04. Vector field, or flow field. Source : Wikipedia. 20 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

Figure 05. Hue, Saturation, Value. Created by Jana Kleineberg

SELECTIVE ATTENTION to other elements in the intensity, and orientation — drawn AND VISUAL SALIENCE visualisation as regards these from a variety of different spatial attributes and others scales. The map itself represents the Certain visual attributes can have a visually most important regions in the significant impact on our so-called • Visual salience more broadly image. Under normal viewing condi- ‘preattentive’ perception. For exam- still, the conspicuousness tions, there is believed to be a positive ple, certain stimuli tend to ‘jump of an element relative to its association between the location of out’ at us, regardless of our top-down environment, influenced by all fixation and salience of the stimulus preferences, strategies, and goals in of the above as well as other at those locations. In other words, processing visual information. Influ- factors, e.g. motion, sharpness salience exerts a small but significant ences on preattentive perception: of edges, orientation effect on fixation likelihood, so that Manipulation of these features • Colour hue, saturation, decision-makers are more likely to may impact the allocation of atten- and value fixate on objects with a greater level tion, e.g. the likelihood that the of salience (Milosavljevic, Navalpa- • Size the surface area decision-maker notices an element kkam, Koch, and Rangel, 2012). for of an element at all, and their likelihood of fixat- instance, Milosavljevic et al. (2012) • Position where an element sits ing on it. The term visual salience demonstrated that, during rapid within a visualisation’s overall refers to the conspicuousness of a decision-making tasks, visual sali- space and/or various subspaces visual element relative to the visual ency influences choices more than surroundings in which it appears (Itti personal preferences (Kahneman et • Predictability whether the and Koch, 2000). A salience model al., 1982). Such salience bias increases element is in its expected position takes as input any visual scene and with cognitive load and is particularly • Set size the total number of produces a topographical map of strong when individuals do not have elements in a visualisation the most conspicuous locations, i.e. strong preferences related to different those locations that are brighter, have • Emotional connotations e.g. options. Salience has also been shown sharper edges, or different colours smiling or angry faces, ‘cute’ to influence the fixation order, with than their surroundings. The saliency imagery of animals or babies more salient objects being fixated on map (Koch & Ullman, 1985) usually earlier (Peschel and Orquin, 2013). • Contrast more broadly, considers three channels: colour, how an element compares 21

ANY COLOUR CAN BE DEFINED ACCORDING TO ITS POSITION IN THREE DIMENSIONS: HUE, VALUE, AND SATURATION

We should take care to note some terminological inconsistency, especially around descriptions of colour. for example, other terms for saturation include intensity and purity. Other terms for value include brightness, lightness, and luminosity. All these terms risk being confused with transparency, which is not really a perceptual dimension of the colour itself, but the degree to which the colour(s) underneath are allowed to show through. Sometimes the terms colour and hue are used interchangeably. for example, in a review of visualisation of uncertainty by Aerts et al. (2003) we read: ‘Bertin (1983) describes an extended set of visual variables to portray information, such as position, size, value, texture, color, orientation, and shape. Among these variables, “the strongest acuity in human visual discriminatory power relates to varying size, value and color”.’ Colour in this context may refer to hue, or to a combination of hue and saturation. As well as terminological inconsistency, there is not infrequently some conceptual confusion associated with these dimensions of colour.

USING COLOUR There is another mention of satu- people tend to associate darker values TO VISUALISE ration as an unsuitable graphical with more certainty, and lighter UNCERTAINTY parameter for the visualisation of values with less (e.g. MacEachren, uncertainty in Cheong’s 2016 review, 1992). Any colour can be defined according although the same studies are consid- Hue is determined by the domi- to its position in three dimensions: ered in both reviews and so there is nant wavelength, and is the term that hue, value, and saturation. Figure 05 a risk of double counting and draw- describes the dimension of colour we illustrates the relationship between ing stronger conclusions than the first experience when we look at a these three perceptual dimensions. evidence supports. Cheong (2016) colour (“yellow,” “blue,” etc.). When Studies such as Seipel and Lim further notes inconsistencies across we speak of hue, we are generally (2017) explore the use of hue, satu- various inquiries into the use of hue referring to the colour in its “pure” ration, and value in communicating and value, with some experiments and fully saturated form. Saturation uncertainties. However, Kinkeldey confirming effectiveness while others refers to how pale or strong the colour et al. (2014) report that ‘from current dispute it; these apparent conflicts is. To simplify just a little, it can be knowledge, colour saturation cannot are likely due to studies not being said that the more white you add, the be recommended to represent uncer- directly comparable, e.g. in terms of less saturated the colour becomes. tainty. Instead, colour hue and value experimental subjects. Where value is Value refers to how light or dark a as well as transparency are better effective, the literature suggests that colour is. Adding grey or black will alternatives.’ 22 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

Figure 06 (left). Illustrating value and uncertainty in separate representations, arranged side-by-side.

Figure 07 (bottom). Using hue to directly illustrate uncertainty.

Figure 08 (page 21). Comparing changing different attributes to communicate uncertainty. Based on Cheong et al. (2016).

All created by Jana Kleineberg

change the value: a low value is dark are absorbed (subtracted) from white VISUALISING grey or black, and a high value is light light, so light of another colour UNCERTAINTY: grey or white. reaches the eye (the colour we see). SOME EXAMPLES A distinction is also made between The primary colours of this colour Figures 06 to 11 give some examples pigment primaries (e.g. print) and model are cyan, magenta, and yellow to illustrate these techniques. Figure light primaries (e.g. pixels). Pigment (CMY); combining all three pigment 07 demonstrates the use of hue to primaries use subtractive colour primary colours yields black. By convey uncertainty about spatial mixing: dyes, inks, paints, pigments contrast, light on a monitor display, values. This fictitious example shows absorb some wavelengths of light and projector, etc. uses additive colour- the projected territorial range of an not others. Typical ‘backgrounds,’ ing: mixing together light of two or invasive species. The key or legend, an such as fabric fibres, paint base, and more different colours. The primary essential feature of graphical displays, paper without pigments, are usually colours of this colour model are explains how two hues convey the made of particles that scatter all usually red, green, and blue (RGB). probabilities that a species will spread colours in all directions, meaning All three primary colours together to respective areas. they look white. When a pigment yields white. or ink is added, specific wavelengths 23

Some studies have suggested using likely object classification (with icons These approaches make use of side-by-side representations of the whose transparency represented the metaphors. Kinkeldey et al. 2014 variable and uncertainty relating to level of uncertainty), the probability write: ‘The contention is that fog the variable, e.g. Deitrick and Wentz that the icon was a missile (with trans- and blur are metaphors for lack (2015). For illustration purposes, parency) and, in a fourth condition, of clarity or focus (as in a camera) consider Figure 06, where the left participants could choose among the and thus directly signify uncer- panel depicts projected future size of representations. Task performance tainty. These metaphors have been a metropolitan area of a fictional city was highest when participants could suggested to have the potential to and the right panel shows uncertainty toggle the displays, with little effect enable a better understanding of in these projections. of numeric annotations. As such, the uncertainty (Gershon, 1998) and we An alternative to a side-by-side authors once more support the use of make the assumption that the use of display is an interactive display, graphical uncertainty representations, metaphors can lead to more intuitive enabling users to switch between even when numerical presentations of approaches.’ Metaphors can be both representations of a variable and probability are present.’ useful and misleading: for example, uncertainty. Various studies, involv- Research suggests that representa- particular colour hues carry conno- ing high stakes, high uncertainty, tions involving hue (b), value (c) and tations which might interfere with and time pressure, have shown that transparency (d) worked best (Figure intended signification. For example, in simulations the ability to switch 07). In addition to transparency, discussing climate change model- between alternative representations value, and hue, graphical attributes ling visualisations, Harold et al. 2017 of uncertainty is helpful. Finger and that were found useful in represent- warn against the use of blue that may Bisantz (2000) explored communi- ing uncertainty include resolution, be misinterpreted as representing cating uncertainty in radar contacts fuzziness, and blurring (Kinkeldey water (Figure 11). by degrading or blurring the icons et al., 2014). Metaphors are not universal, and used to represent them. Bisantz et Riveiro et al. (2014) concur, citing associations might differ depending on al. (2011) expanded this research, a couple of visualisation of uncer- the culture and experiences of users. as Riveiro et al. (2014) summarise: tainty evaluations where ‘fuzziness Further, as Kinkeldey et al. (2015) ‘several display methods were used in and location seem to work particu- show, the choice of colour hue can a missile defense game: icons repre- larly well, and both size and transpar- have an impact on the perception of sented the most likely object classi- ency are potentially usable.’ risk, and hence on decision-making fication (with solid icons), the most 24 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

under uncertainty, such as the deci- lives lost if design influences people’s Figure 09 (above). sion about whether or not to follow an decision not to follow an evacuation Using blurring, fuzziness, and evacuation order. To understand the order, where some other hue would transparency to communicate impact of colour hue we must under- have better conveyed an appropriate uncertainty. stand the emotional significance for level of urgency. In situations where an users, which might be different for audience for the visualisation is diverse Figure 10 (bottom). different users based on individual but it is impossible to tailor visualisa- Using pixelation to represent and group-linked factors. Further, tions to distinct groups (as with hurri- uncertainty. the impact of the same colour hue cane warnings), it may not be trivial on a person’s decision-making might to make choices regarding visualisa- Figure 11 (page 23). depend on the circumstances, for tion formats, as trade-offs between Colour and metaphor. Based on instance triggering a different set of the overall efficacy and group specific Harold et al. 2017. The colour blue is heuristics when the person is under impacts might need to be considered. traditionally used to represent water; pressure. The choice of colour hue on thus data might not be read correctly. a map could translate into a number of All created by Jana Kleineberg. 25

EFFECTIVE GRAPHIC DESIGN PRINCIPLES Boone et al. (2018) list the following principles of effective graphic design:

‘Effective graphic design takes account of It also takes account of semantics:

• the specific task at hand (Hegarty, 2011) • compatibility between the form of • expressiveness of the display (Kosslyn, 2006) the graphic and its meaning (Bertin, 1983; Kosslyn, 2006; Zhang, 1996) • data-ink ratio (Tufte, 20 01) • usability of the display, such as including • issues of perception (Kosslyn, 2006; appropriate knowledge (Kosslyn, 2006)’ Tversky, Morrison, & Betrancourt, 2002; Wickens & Hollands, 2000) Ignoring these principles, or failing to implement them effectively, may lead to misunderstandings, or • pragmatics of the display, including making other forms of suboptimal decision support. the most relevant information salient (Bertin, 1983; Dent, 1999; Kosslyn, 2006) 26 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

8 THE USER In order to effectively visualise uncer- decision-making settings, and how tainty, it is necessary to understand it may interact with asymmetries of how mistakes can occur in reception. information, expertise, experience, As emphasised throughout this cata- authority, and accountability among logue, such understandings should analysts, decision-makers, and other be developed through empirical test- stakeholders. ing with users. However, two design Finally, the process of develop- principles do offer some guidance: the ing visualisation formats for decision principle of appropriate knowledge support can be informed by a range (which relates to the user’s familiarity of different models of cognition with conventions) and the semantic and decision-making, including an principle (which relates to ‘natural’ understanding of the decision-maker mappings between visualisations and as a boundedly rational agent, an visualised information). understanding of common heuristics Furthermore, it is useful to appre- and biases related to perceiving and ciate that a visualisation format may reasoning about uncertainty, and an have several different kinds of user, understanding of the counterintui- as well as stakeholders who rely tive quirks of visual perception (i.e. on it in a more indirect fashion. In those that underlie optical illusions). particular, it is useful to be aware of In this section we briefly touch on how the persuasive power of visu- these topics. alisation can play out in distributed

Keywords principle of appropriate knowledge clustering illusion framing effect semantic principle priming confirmation bias visualisation & persuasion anchoring effects set size effect i heuristics & cognitive biases Weber’s law surface size effect confounding mean and variance Ostrich effect & risk compensation bias position effects & predictable locations colour contrast phenomena availability bias emotional stimuli inter-individual variability 27

The user of a visualisation must have knowledge of the conventions necessary to extract its relevant information.

THE PRINCIPLE cautions, or recommendations to help and “up is good” (Tversky, 2011).’ OF APPROPRIATE with interpretation. The principle of According to the semantic principle, KNOWLEDGE appropriate knowledge also invites visual attributes should be mapped AND THE SEMANTIC us to think about the risk that users to underlying data in ‘common PRINCIPLE will interpret a visualisation based sense’ ways, so that most users would on inappropriate conventions. If the correctly guess how to interpret it, Heuristics are short-cut procedures user does apply a different conven- even without knowing the conven- or rules of thumb that may generate tion to the one intended, will this tions used. ‘An example of a match good-enough results under certain produce dissonance, causing the [between visualisation and underly- conditions but are also implicated in user to feel that something is wrong? ing data] is using the length of a line generating biased decisions. Cogni- Or will the visualisation apparently to denote length of time. An example tive biases are systematic errors in accommodate the incorrect conven- of a mismatch would be using higher one’s thinking relative to either social tion, allowing the user to incorrectly values on a graph to show nega- norms for reasoning and/or formal interpret the visualisation indefi- tive numbers’ (Boone et al., 2018). logic. Without testing, it is not possi- nitely? Furthermore, the principle of Another example of a match would ble to know how a particular visu- appropriate knowledge invites us to be data values that are related to one alisation format will interact with a think carefully about the conventions another being physically proximate. user’s heuristics and biases. However, we rely on in encoding and interpret- Padilla et al. (2018) recommend that some principles of good design prac- ing information visually, since prac- we ‘[a]im to create visualizations that tice exist, and are applicable to a tices that feel obvious or inevitable most closely align with a viewer’s wide range of visualisation formats may actually rely on norms that have mental schema and task demands’ and their users. Two such principles, been learned at some point, and that and ‘[w]ork to reduce the number which are related to each other, are are not necessarily shared by all users. of transformations required in the the principle of appropriate knowl- In practice, it may not be possible decision-making process.’ edge and the semantic principle. The principle of appropriate to know the user’s familiarity with The semantic principle raises knowledge simply states that the user various conventions. This is where some interesting theoretical ques- of a visualisation must have knowl- the semantic principle comes in. In tions (see sidebar). However, prac- edge of the conventions necessary the context of creating visualisations tically speaking, the chief drawback to extract its relevant information. of uncertainty, Boone et al. (2018) of the semantic principle is that The conventions of the display are describe ‘the semantic principle of it does not always provide a reli- often encoded in a legend expressing natural mappings between variables able or sufficiently detailed guide to the correspondence between visual in the graphic and what they repre- support the visualisation of complex variables and their meaning. There sent.’ As examples, they offer ‘classic information such as uncertainty may also be additional instructions, metaphors such as “larger is more” information. It is therefore usually 28 DIGGING• VISUALISING UNCERTAINTY DOWN A SHORT INTRODUCTION INTO THE SEMANTIC PRINCIPLE As a principle of design, the semantic efficient and straightforward visual principle teaches that adhering to encoding of data. But again, this ‘natural,’ ‘classic,’ ‘common sense,’ interpretation is not unproblematic: ‘straightforward,’ ‘efficient,’ ‘self- highly compressed information may evident,’ ‘self-explanatory’ mappings be efficiently encoded, but involve can help to reduce misinterpretation several conceptual steps to decode. and/or miscommunication. However, Identifying the semantic principle with it is easier to spot violations of the Zhang’s principle would also make it semantic principle than to give a full difficult to accommodate desirable account of how and why they are redundancy. The semantic principle violations, or to articulate criteria might be understood to favour by which to judge borderline cases. ‘self-evident’ or ‘self-explanatory’ One interpretation of the semantic visualisations, perhaps in the sense principle is as an implicit appeal to that it is not possible to recognise resemblance, e.g. the use of colours that data has been encoded without on a map that roughly resemble an also recognising how it has been aerial photograph (blue for water, encoded, or in the sense that incorrect green for grasslands, etc.). However, it interpretation of the visualisation is unlikely that every implementation eventually generates the knowledge of the semantic principle can be necessary to interpret it correctly. easily explained in this way, and These would however be quite narrow it does leave significant space and demanding interpretations. Other for culturally acquired meanings useful perspectives might be drawn (e.g. blue may be a more intuitive from phenomenological philosophy, representation even of water that is insofar as ‘spatial perceptions black, brown, or green). In their brief and values that are grounded on account, Boone et al. (2018) also cite common traits in human biology Zhang (1996); Zhang suggests that [...] transcend the arbitrariness of no visualisation format is universally culture’ (Tuan, 1979), and/or from the optimal for all the cognitive tasks field of biosemiotics, which theorises users may want to perform on the how meaning may be produced and underlying data, but that ‘there interpreted by non-humans. Overall, does exist a general principle that the semantic principle is a useful can identify correct or incorrect tool for research and design, but we mappings between representations should be a little cautious of taking and tasks’. This principle is that the for granted the ‘naturalness’ of the representation should contain no mappings that Boone et al. allude extraneous information, and should to. Describing these relationships contain all the necessary information as ‘natural’ may obscure how our (e.g. external information about intuitions about them are cultivated interpretive conventions should be and reinforced by social, cultural, unnecessary). This may suggest we institutional, and other factors — even understand the semantic principle when those intuitions are deep-seated as embodying a preference for and widely-held. 29 necessary to supplement the opera- a visualisation format, and fostering tion of the semantic principle with visual literacy, may create opportu- clearly-indicated conventions. Thus, nities to steer decision-makers and when designing visualisation formats, other stakeholders toward or away the principle of appropriate knowl- from particular goals, assumptions, edge and the semantic principle are evidence, procedures, and method- complementary. Users should be ologies. These difficulties can be equipped with knowledge of appro- compounded by the so-called ‘curse priate conventions, and the conven- of knowledge,’ the cognitive bias tions that are chosen should align as which means that experts frequently closely as possible with what most mistake how non-experts perceive users would expect anyway. and reason about information related taken seriously. According to dual to their area of expertise. The ques- process theory, the thought processes VISUALISATION AND tion is then, how do we distinguish which underlie decision-making can PERSUASION valid decision support from improp- be attributed to either System 1 or Often the agent best qualified to erly manipulative formats and prac- System 2 (also sometimes called analyse a decision-context does not tices? There is no easy answer. The Type 1 or Type 2 cognition). Briefly, have the authority to personally make question’s tractability will vary from System 1 thinking refers to rapid, the decision. Visualisation may be one context to context, and may call upon instinctive thinking on the fringes way that such experts communicate value judgements, and legal, political, of consciousness, with a relatively their analysis to relevant decision- and ethical considerations. heavy reliance on heuristics. System makers. In this sense, visualisation For example, in a setting where 2 refers to more conscious, explicit, can be used to overcome gaps between decisions are relatively standardised analytic patterns of thought. These different levels of expertise. Likewise, and fungible, and there is strong two systems compose a spectrum visualisation may be used to commu- consensus about what comprises rather than a strict dichotomy. Sound nicate an analysis to a diverse set of sound decision-making and good visualisation will take into considera- stakeholders, so that they can all outcomes, decision quality may be tion how System 1 and System 2 can come to decisions in accordance with a fairly unproblematic guide to the work together effectively (Padilla et their diverse preferences and spheres legitimacy of the decision support al., 2018). of authority. Visualisations may thus mechanisms used. In more compli- How we allocate attention is heav- become reference points to facili- cated settings, however, there is the ily influenced by what presents itself tate conversations bringing together risk that decision quality is improved as salient to System 1. Research on multiple perspectives, interests, and at the expense of undesirable distri- preattentive processing is concerned forms of expertise. butions of knowledge production and with ‘What visual properties draw In all these communication validation, e.g. certain stakeholders our eyes, and therefore our focus of activities, the questions arise of being institutionally accountable attention, to a particular object in a what counts as legitimate influence, for knowledge which they cannot scene?’ (Haeley, 2007). Those visual and when and how such influence actually account for, insofar as it is properties that draw our eyes — the should be deliberately wielded. The actually understood only by other ‘bottom-up’ properties of a visualisa- choice of visualisation can influ- stakeholders, and/or understood by tion — can play a critical role in what ence how decision-makers and other nobody within the decision-making the decision-maker perceives and stakeholders treat uncertainty, as centre, because it has been so thor- fixates on. Because visual informa- well as other aspects of decision- oughly cognitively offloaded into tion is processed by our low-level making, through mechanisms such decision support systems. visual system, with relatively little as emotional priming, allocation of Dual process theory does offer cognitive effort, the system can be attention, and selection of heuristics. some helpful perspectives. It is widely harnessed by well-designed visu- In particular, ‘[w]hen incorporated acknowledged that visualisation can alisation, in order to draw attention into visualization design, saliency push certain information to the fore- to task-relevant information, and/ can guide bottom-up attention to front, while hiding other information or to minimise and mitigate biases, task-relevant information, thereby in plain sight. But how does this actu- i.e. ‘using vision to think.’ However, improving performance’ (Padilla et ally occur? One mechanism is the use we should be careful not to construe al., 2018). More broadly, the process of visual salience to influence the like- System 1 and System 2 as operating of classifying uncertainty, developing lihood that information is noticed and in a simplistic sequential fashion: 30 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

Choice architects are people in a altering people’s behaviour without position to design the environment closing off any options or imposing in which people make decisions. In significant costs on them. Think- the same way that traditional archi- ing of visualisations as part of choice tects design the buildings that people architecture allows us to more vividly inhabit, choice architects design the describe the pitfalls of improper deci- way choices are presented to decision- sion support. On the one hand, a makers. The theory usually assumes given choice architecture may be too that such choice architecture is ubiq- ‘open,’ offering the decision-maker uitous and cannot be avoided. We can plenty of information, without giving System 1 does not conduct a triage therefore either allow it to develop them sufficient nudges to sift through and then pass on what it determines haphazardly, or we can design it in this information and to understand to be important to System 2. Rather, ways which protect decision-makers it. In this case, a decision-maker may each system dynamically influences from cognitive biases and guide them fail to apply a useful heuristic, and/ the other’s activities. In this sense toward more rational decisions. For or apply one associated with damag- visual perception and reasoning can example, if a visualisation fails to ing bias, without being alerted to it or also be understood as a ‘middle-out’ make appropriate use of System receiving the opportunity to reflect process, rather than a bottom-up 1 to focus the user’s attention on critically on how they are forming process that hands over to a top-down task-relevant information, there is their judgment. On the other hand, process at a certain point. a risk the user will instead focus on a choice architecture may nudge too All this is especially important distractors, reducing decision quality hard, potentially exploit biases to for providing decision support under (Padilla et al., 2018). In the context funnel the decision-maker toward a uncertainty, insofar as there are of visualisation for decision support, particular course of action, making many well-attested cognitive biases choice architecture might be under- them nominally accountable for related to uncertainty information. stood as structure that connects (i) something that has essentially already However, structuring the decision- the diverse cognitive resources of the been decided. That is, when a visu- maker’s attention for the purpose of boundedly rational agent with (ii) the alisation format is designed to be de-biasing cannot be done ad hoc wider decision-making environment foolproof, there can be a danger that (Orquin et al., 2018); it requires a in which such an agent acts. This it impinges on the decision-maker’s robust design and validation of a includes but is not limited to the visu- legitimate freedom of interpretation. decision-making environment, with alisation itself. A related challenge — again attest- sensitivity to how asymmetries in ‘Nudging’ is one of the tools in the ing to the importance of robust test- visual literacy map onto accountabil- choice architect’s toolbox. According ing on a case-by-case basis — arises ity structures. The concept of ‘choice to Thaler & Sunstein (2008), for a because the precise graphical layout architecture,’ from behavioural technique to count as nudging — as that faces the user is often procedurally economics and cognitive psychology, opposed to mandating or forcing — it generated, at least in part, by under- may prove somewhat useful here. must be a technique that allows lying data. In other words, aesthetic features that may impact visual sali- ence (such as a sense of proportion, harmony, balance, ‘rhythm,’ and so on) will vary somewhat according to what data is inputted. To the extent that these features do impact visual

PATTERN CONTRAST EMPHASIS BALANCE RYTHM salience, there is the possibility that they also alter the details of the choice architecture. When the visualisation format is static and only ever needs to encode one information set, then the designer has substantial scope PROPORTION VARIETY UNITY MOVEMENT HARMONY to assess and to modify its overall aesthetics. However, when the visu- Figure 12.Illustrations of selected aesthetic features. Created by Jana Kleineberg. alisation format is dynamic, and/or 31

T H E C O G N I T I V E B I A S C O D E X

We store memories differently based We notice things already primed in on how they were experienced memory or repeated often We reduce events and lists Bizarre, funny, visually striking, or What Should We Tip of the tongue phenomenon Too Much to their key elements anthropomorphic things stick out more Levels–of–processing effect than non-bizarre/unfunny things

Remember? Absent–mindedness Information Next–in–line effect

Part–set cueingSerial–position effect effect

Google effect

Testing effect

Memory inhibitionRecency effect We discard specifics Suffix effect Primacy effect Leveling and sharpening DurationModality neglect effect MisinformationList–length effect effect to form generalities Serial recall effect We notice when something has changed

Availability heuristic Attentional bias Illusory truth effect Mere–exposure effect Context effect Cue–dependent forgetting Mood–congruent memory bias Frequency illusion Baader–Meinhof Phenomenon Peak–end rule Empathy gap Omission bias Fading affect bias Base rate fallacy

Bizarreness effect Humor effect Negativity bias Von Restorff effect We edit and reinforce Stereotypical bias Picture superiority effect Implicit stereotypes Self–relevance effect Negativity bias Implicit associationPrejudice some memories after the fact Anchoring Conservatism Contrast effect Spacing effect Distinction bias Focusing effect Suggestibility Framing effect We are drawn to details False memory Money illusion Misattribution of memory Weber–Fechner law SourceCryptomnesia confusion that confirm our own existing beliefs Confirmation bias Congruence bias We favor simple–looking options Less–is–better effect Post–purchase rationalization Choice–supportive bias and complete information over Occam's razor Selective perception Conjunction fallacy Observer–expectancy effect complex, ambiguous options Experimenter's bias Bike–sheddingLaw of Triviality effect Observer effect Rhyme–as–reason effect Expectation bias Ostrich effect Subjective validation We notice flaws in others Belief bias Information bias Continued influence effect Semmelweis reflex more easily than we Ambiguity bias notice flaws in ourselves To avoid mistakes, Bias blind spot Status quo bias Naïve cynicism we aim to preserve autonomy Social comparison effect Naïve realism Decoy effect and group status, and avoid Confabulation Reactance Clustering illusion Reverse psychology irreversible decisions Insensitivity to sample size System justification Neglect of probability Anecdotal fallacy Backfire effect Illusion of validity We tend to find stories and Endowment effect Masked–man fallacy Recency illusion patterns even when looking Processing difficulty effect Gambler's fallacy Pseudocertainty effect Hot–hand fallacy at sparse data Disposition effect Illusory correlation Zero–risk bias Pareidolia Unit bias Anthropomorphism To get things done, we tend IKEA effect Group attribution error Loss aversion to complete things we've Ultimate attribution error Generation effect Stereotyping invested time and energy in Essentialism Escalation of commitment Functional fixedness Irrational escalation Moral credential effect Sunk cost fallacy Just–world hypothesis Argument from fallacy Authority bias Automation bias IdentifiableAppeal victim to novelty effect Bandwagon effect Placebo effect Hyperbolic discounting Peltzman effect Out–group homogeneity bias We fill in characteristics from To stay focused, we favor the Cross–race effect Risk compensation In–group favoritism stereotypes, generalities, immediate, relatable thing Effort justification Halo effect Cheerleader effect Trait ascription bias Positivity effect and prior histories in front of us Not invented here Reactive devaluation Well–traveled road effect Illusory superiority Illusion of control Mental accounting Appeal to probability fallacy DefensiveFundamental attribution attribution hypothesis error Normalcy bias BarnumForer effect effect Murphy's Law Actor–observerSelf–serving bias bias Zero sum bias Survivorship bias Subadditivity effect Optimism bias Denomination effect The magical number 7 ± 2

Illusion of transparency Egocentric bias Curse of knowledge Spotlight effect Extrinsic incentive error Illusion of external agency Illusion of asymmetric insight

Hard–easy effect Dunning–Kruger effect We imagine things and people Lake Wobegone effect Third–person effect

Restraint bias Declinism False consensus effect Moral luck we're familiar with or fond of To act, we must be confident we Projection bias Impact bias Social desirability bias

Overconfidence effect Outcome bias

Planning fallacy Hindsight bias Pessimism bias Time–saving bias as better can make an impact and feel what Self–consistency bias

Pro–innovation bias Telescoping effect Telescoping we do is important retrospection Rosy

Need To We simplify probabilities and numbers Not Enough Act Fast to make them easier to think about Meaning We project our current mindset and We think we know what assumptions onto the past and future other people are thinking

Figure 13. The cognitive Bias Codex. Source: Wikipedia, by John Manoogian III. when it is used with multiple data sets answers that might not be perfectly domain-specific study or experience. over time, then these features cannot accurate, but that might be adequate In the latter sense, a heuristic can be be micromanaged. Instead to some for everyday life or for a certain associated with the development of extent these features may be deter- purpose. Because of their crude and expertise. The distinction between mined as emergent ‘side effects’ of the approximate nature, however, heuris- a heuristic and a cognitive bias is information that is represented. tics can also lead to systematic errors not completely clear-cut — if you and behaviour that is irrational from are lucky enough to wind up in just HEURISTICS AND the perspective of decision theory. the right context, a bias may lead to COGNITIVE BIASES People can apply heuristics without effective action — but in general we In the 1970s, psychologists began even thinking about or knowing that can say that a heuristic is “a simple to explore the mental tools, called they are doing so. The term heuristic procedure that helps find adequate, heuristics, which humans use to assess often refers specifically to these reflex though often imperfect, answers to probability and make decisions (Tver- patterns of reasoning that are widely difficult questions” (Kahmeman, sky and Kahneman, 1973). In general, attested and apparently ‘deep-seated’ 2011), whereas a bias is a systematic a heuristic (or a heuristic technique features of human psychology. Some- distortion of judgment, often a result or heuristic rule) is a rough-and- times heuristic refers more broadly to of the limitations of the heuristic(s) ready mental process that is used for any kind of mental shortcut, includ- used. problem-solving. Heuristics generate ing those that are acquired through 32 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

Figure 14. Hurricane Michael’s path 2018 leads to landfall across the Florida panhandle on Wednesday as a Category 1 storm, according to the Sunday 10 a.m. update from the National Hurricane Center. Image: NOAA.

A B C Figure 15. The impact of visualisation of uncertainty on subject’s USA USA USA understanding on spatial and temporal

Florida Florida Florida uncertainty in a hurricane forecast. Based on visualisations used by Ruginski et al. (2016) to illustrate their findings. Created by Jana Kleineberg. D E

A. cone w. centerline USA USA B. centerline C. ensemble Florida Florida D. fuzzy cone E. cone only

Dimara et al. (2018) suggest that as well as decision-making process environment. In particular, as Correll there are currently over 154 known and quality (Wall et al., 2017). & Gleicher (2014) put it, ‘how we cognitive biases, including biases or Decision-makers can be consid- visually encode uncertainty and prob- systematic errors related to making ered boundedly rational agents ability can work to “de-bias” data causal attributions, recalling infor- whose ‘selection’ of heuristics (even which would ordinarily fall prey to an mation, testing and assessing a when this is not a conscious selec- otherwise inaccurate set of heuristics hypothesis, conducting estimations, tion) may be influenced by the (by comparison to an outcome maxi- opinion reporting, etc. In visuali- specific uncertainty visualisation mizing classical statistical view).’ sation research, only a handful of format used. One possible objective in On the next few pages, we mention these biases have been scientifically developing uncertainty visualisation a selection of the kinds of biases assessed (for a review, see Valdez et formats, therefore, is the promotion which may become relevant in the al., 2018a). Overall, findings suggest of context-appropriate heuristics and development of an effective visuali- a moderate to strong impact of biases deactivation of inappropriate ones, sation format. As these are intended on selective attention, performance recognising that what is ‘appropriate’ to be illustrative, we make no attempt speed, memory recall and retention, must vary with the user and decision to categorise them in any detail. 33

However, Valdez et al. (2018a) Bar chart with error bars suggest that within visualisation length of bar = proportional to the values they represent. research we can divide cognitive whiskers represent error margin (here ±5) biases into three kinds — percep- Modified box plot tual, action, and social — while whiskers represent error margin also acknowledging a lack of hard center line = median (50% percentile) dots = outliers (extreme datapoints) boundaries. Perceptual biases, such as Weber’s Law and the clustering Gradient plot Point estimate with probability density illusion, occur on the ‘lowest’ level function shown as a gradient of cognitive processing, the sensory- motor-sensory loop, and thus are Violin plot symmetrical probability density mostly irresponsive to training. That is, one cannot ‘unsee’ the effects of such a bias, even if one knows that 0 20 40 60 80 100 it is there. Action biases, such as the availability bias and the ostrich effect, are more related to interpre- Figure 16. Alternative visualisations of the confidence interval. tation and thus more responsive to Created by Jana Kleineberg. training: what is perceived may be an adequate representation of what attention to this fact, which is likely uncertainty that grows, rather than can be objectively known, but it is a general feature of human psychol- the strength of the hurricane (which analysed in systematically incorrect ogy: ‘Even when presented with an is not depicted at all). Ruginski et al. ways. Finally, social biases, such as uncertainty visualization, people still (2016) point to yet another misin- the curse of knowledge and fram- exhibited greater attentional focus on terpretation that the cone represents ing effects, are those biases which the mean and overconfidence in their an area of impact rather that an area can only be understood in relation to understanding of the variance. These made up of possible hurricane trajec- socio-cultural institutions and norms, results have implications for decision tories. One solution proposed was to and the deeply engrained capabili- makers and the consequences related represent trajectories as individual ties and dispositions that we acquire to not considering alternatives to lines, as a set of possible realisa- through lived experience. the most likely outcome.’ However, tions, denser around the most likely Visualisation can alleviate the Pugh et al. (2018) also note that some path. This, however, was found to effects of bias, but visualisation can designs perform better than others. make people less afraid of the hurri- also exacerbate these effects, or create In the context of visualising the cane ­— the path ensemble visualisa- opportunities for biases to manifest predicted path of a hurricane, they tion reduced their estimates of risk, when they otherwise would not. write that it is possible that ‘differ- potentially leading some to ignore The study of cognitive biases within ent forms of visualizations may better evacuation orders, as opposed to visualisation is challenging. ‘Some enhance the understanding of vari- alternative visualisations based on biases might counteract each other, ance, improve transfer effects, and the same predictions. Ruginski et al. and experiments have to be meticu- reduce related overconfidence.’ They (2016) note that ‘the various visuali- lously planned to isolate the desired point in particular to Ruginski et al. zations caused participants to notice effect from other effects’ (Valdez et (2016), who found that ‘the addition different visual properties of the al., 2018a). of a centreline, fuzzy shading, and/or displays and to base their judgments ensemble paths to a cone visualization on different heuristics’; for instance CONFOUNDING MEAN decreased the perception of variance.’ they recount that, ‘The fuzzy-cone AND VARIANCE Another common problem is the and the cone-only visualizations Evidence suggests that current visu- misinterpretation of increase in vari- (both without the centerline) resulted alisation formats are more effective ance over time as if it were an increase in lower damage judgments than the at communicating uncertainty about in the mean. Here the semantic prin- cone-centerline. It is possible that the the mean of a set of sample values ciple of larger size meaning some- presence of the salient center forecast than uncertainty about the variance. thing is bigger is adhered to, but track leads users to cognitively assess Reducing overconfidence is particu- leads to confusion about what gets the intensity of the hurricane to be larly hard. Pugh et al. (2018) draw bigger. In this case, it is the spatial greater.’ 34 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

COLOUR CONTRAST PHENOMENA Our perception of a colour is influ- enced by the colour(s) adjacent. Colour contrast phenomena may

Surround a colour with a lighter color and it Surround a colour with different hues and it be considered a classic instance of a will appear darker; surround a color with a will shift in appearance towards the perceptual bias, in that one cannot darker color and it will appear lighter. complementary hue of the surrounding color. ‘unsee’ the effect, despite knowing that it is there. CLUSTERING ILLUSION The clustering illusion is a part of a perceptual bias which causes respondents to see patterns in small Surround a colour with a less saturated color You can surround two different colours with sets of randomly generated data. For and it will appear more saturated; with a more two other colours to make them appear saturated color it will appear less saturated. more similar. example, apparently significant clus- ters or streaks are perceived in low- Figure 17. Colour contrast phenomena. density scatter-plots (Blanco et al., 2017). The clustering illusion leads Misunderstanding variability fixed interval, and ‘emphasize an “all to irrelevant, inaccurate inferences of is a common problem even among or nothing” approach to interpreta- causal relationships (also called causal experts. When error bars are used to tion ­— values are either within the bar illusion or illusory correlation), and depict confidence intervals for any or they are not’ (Correll & Gleicher, has been shown to decrease the accu- distribution (e.g. continuous, non- 2014). racy and quality of decision-making uniform), one frequent mistake is to They may also often violate the (Blanco et al., 2017). Three principles interpret the uncertainty distribution principle of appropriate knowledge are responsible for the perception of as uniform, bounded, and discrete, if, as Boone et al. (2018) describe, a potential cause-effect relationship. i.e. to assume that every value inside they fail to state ‘what measure of The first two are priority and conti- the error bars is equally probable, error is represented by the bars (e.g. guity, which are represented in the and the probability of a value falling whether they show the standard temporal ordering (i.e. priority) and outside of that range falls sharply to error, standard deviation, or 95% the proximity (i.e. contiguity) of the zero. Correll & Gleicher (2014) also confidence interval),’ especially since stimuli. If two events occur close reported another bias, where a bar ‘even scientists do not always appreci- in time and space, this may create chart was used to indicate the mean ate the differences in the inferences a belief that one caused the other. and error bars to indicate the confi- that can be made in each of these The third principle, the principle of dence interval. Participants treated instances (Belia, Fidler, Williams, contingency, refers to the fact that the bottom part of the margin of & Cumming, 2005).’ causes and their effects must be corre- error, which overlapped with the bar, Correll & Gleicher (2014) also lated, and it is this principle that is as more probable, incorrectly inter- proposed and tested alternative visu- believed to create a clustering illu- preting symmetrical distributions as alisations. Participants were shown sion bias In our perception of contin- skewed. a data point representing a potential gency, we are prone to systematically Traditional error bars, despite outcome, and asked to judge how overestimating causal linkage. One being the most common visualisa- likely this outcome was given the suggested reason for respondents’ tion of uncertainty, are arguably in sample mean and the margin of error, tendency to overestimate data as violation of both the semantic prin- testing across different visualisation correlated is that it provides a feel- ciple and the principle of appropri- formats. It was found that gradated ing of control over the situation, and ate knowledge. Visualisations using visualisations — such as colour reduces anxiety that might occur in these error bars violate the semantic gradients or tapering, violin-like the context of risk and uncertainty principle, insofar as they are inter- shapes — can ameliorate misinter- (Blanco et al., 2017). preted to represent what may be an pretations associated with traditional infinite domain of a function by a error bars, at least in some cases. 35

PRIMING a consequence, in tasks where stand- Priming refers to the phenomenon ardised responses are critical, it has by which response to a stimulus can been proposed that visual aids can be influenced by prior exposure to help to de-bias the decision-making some related stimulus (Kristjansson, process, reducing variability in search 2006). for example, when asked to identification activities, and control- complete the word ‘SO_P,’ respond- ling for effects that cause response ents who have been shown a picture latencies. However, more research in of a shower are relatively more likely this area is still needed (Valdez et al., to choose ‘SOAP,’ and respondents 2018b). who have been shown a picture of ANCHORING EFFECTS bread and butter are more likely to Anchoring consists of the use of a found that anchoring effects take choose ‘SOUP’ (Valdez et al., 2018b). previous stimulus as some sort of place predominantly automatically In the context of visualisation, prim- reference or anchor, which is used and unconsciously, and in situations ing effects challenge the assumption to help make judgments about the of uncertainty, where users are likely that ‘[g]iven the same stimulus and current stimulus, even if the stim- to hold on to a narrative that is read- the same person, one should “see” the uli are unrelated and the anchor ily available, anchoring is associated same thing every time’ (Valdez et al., is completely random. Anchor- with making more conservative deci- 2018b). When it comes to perceptual ing effects are related to priming; sions (Ellis and Dix, 2015). for exam- processing for visual search targets, priming appears to be one of several ple, when a previous estimate or deci- primed behaviour can be facili- mechanisms that underlie anchoring sion has been shown to be inaccurate, tated by means of visual ‘cues,’ e.g. (Valdez et al., 2018b; Wilson et al., an anchoring effect may prevent the in certain uses of colour to represent 1996). The anchor provides an initial decision-maker from making suffi- certain types of content, or by repeat- reference point which the decision- cient adjustments on the next itera- ing cues in expected positions or maker then adjusts by incorporating tion. Anchoring can in effect act as a locations. Such techniques have the relevant beliefs (the ‘anchor-and- stability bias, meaning that it is one potential to enhance performance adjust’ heuristic), but since these of those biases may make one cling speed, accuracy and recognition of adjustments are often insufficient, the to the status quo (Kahneman, 2011). identified search item, as well as to In visualisation research, anchor- initial choice of anchor tends to carry decrease response latencies. Prim- ing has been studied in terms of undue weight. In relation to deci- ing appears to be of particular value search tasks within naturalistic sion-making generally (Langeborg when the task contains ambiguity. As visual scenes (Boettcher et al., 2018). and Eriksson, 2016), studies have

Figure 18. Clustering Illusion. Source: CaitlinJo, Wikipedia. 36 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

Anchor objects are objects that hold aided by uncertainty visualization a relatively high amount of predictive selected higher priority values and information about objects with which more hostile and suspect identities. they frequently co-occur in spatially This suggests that, when safety was consistent arrangement. For exam- an issue, the experts put themselves in ple, across many different scenes, a the “worst-case scenario” in the pres- bathroom sink may hold information ence of uncertainty.’ about the likely locations of plughole, taps, toothbrush, toothpaste, soap, WEBER’S LAW mirror, and so on. By tracking eye How we judge and categorise sensory movements, it has been demonstrated magnitude is affected by a number of that the presence of relevant anchor biases (Poulton 1979). As an illustra- objects within scenes alters search tion, the smallest perceptible change strategy, resulting in less scene cover- in the brightness of a light source will Figure 19. Weber-Fechner Law 01, age overall. Relevant anchor objects be different depending on the initial source Wikipedia, MrPomidor. can somewhat enhance perceptual intensity of the light. Weber’s Law is a processing by faster reaction times, On each side, the lower square contains mathematical formula that states that and less time between fixating the 10 more dots than the upper one. the minimum difference in intensity anchor and the target (Boettcher et However the perception is different: needed to perceive a change between al., 2018). Anchoring has also been On the left side, the difference between two given stimuli is proportional to studied in visualisation research in upper and lower square is clearly the stimuli (Carr, 1927; Harrison terms of how anchoring during train- visible. On the right side, the two et al., 2014; Kay and Heer, 2016; ing may influence how analysts use squares look almost the same. Valdez et al., 2018a). Using Weber’s an exploratory visual analysis system Law, we can make predictions about (Wesslen et al., 2019). whether a given change in a stimu- Anchoring may become prob- lus intensity will be perceived, and lematic when decision-makers use about what magnitude of change will a ‘worst case’ scenario as an anchor. be perceived. The formula requires an Nadav-Greenberg et al. (2008) asked empirically derived constant, Weber’s participants to forecast wind speeds fraction (k). The law has been shown based on visualisation of median not to hold for extremes, e.g. a very wind speeds as wall as various repre- dim or very bright light. sentations of uncertainty informa- The law implies the risk of a Figure 20. Weber-Fechner Law 02, tion. They found that where uncer- response bias when judging and cate- source Wikipedia, MrPomidor. tainty information was displayed as gorising sensory magnitudes (Poul- an upper boundary of projected wind ton, 1979). As an illustration, such an Circles in the upper row grow in speeds, forecasts showed a bias toward effect could be modelled in correlated arithmetic progression; they make an higher wind speeds. Kinkeldey et al. data representations (such as scatter- impression of growing initially fast (2015) note in their review: ‘Thus, plots), with first evidence showing and then slower. the authors give the warning that that the just-noticeable difference in correlation strength is indeed differ- Circles in the lower row grow in in a real-world setting providing ent in different parts of the correlation geometric progression: each one is worst-case maps could lead to more spectrum (Harrison et al., 2014). The larger by 40% than previous one. They false alarms. They explain this effect findings suggest that a user’s ability make an impression of growing by the with evidence from past research to identify correlation in visualisa- same amount at each step. that anchors (in this case the depic- tion of the worst case) unconsciously tion formats can be modelled using influence people’s judgments (Chap- Weber’s Law, and the authors inter- man and Johnson, 2002). Similar pret this to mean that the underlying results were provided by Riveiro et information-bearing visual features al. (2014) in a target identification also follow Weber’s Law. They there- experiment, where an expert group fore suggest that Weber’s Law, and 37

biases are not well-researched in visu- primarily used in market research alisation research; however, it has and assumes that ‘if people are decid- been suggested that in the context of ing between two products (“target” uncertainty, in order to mitigate poor and “competitor”), a third product decision-making, automated systems (“decoy”) that is close to the target should be put in place, highlighting but objectively suboptimal to both critical, uncomfortable information attributes, can make the target look to counteract the occurrence of such more attractive’ (Dimara et al., 2018). potential biases (Dimara et al. 2018). Expressed in more game theoretic terms, the attraction effect ‘suggests AVAILABILITY BIASES that any decision involving a set of People are influenced by the availabil- points that belongs to the Pareto other perceptual laws from psychol- ity of examples of an event, even when front is influenced by the dominated ogy and cognitive science, can be the ease with which they can think of datapoints below it’ (Dimara et al., used to model and rank the precision examples bears no relation to actual 2018). Within visualisation research, afforded by different visualisation frequency. This bias is called avail- Dimara et al. (2018) embedded the formats. For example, the median ability bias (Tversky & Kahneman, attraction effect within a scatterpoint- just-noticeable-difference of the scat- 1973), and can exert a considerable based task, and invited respondents terplot format can be compared with influence on reasoning and decision to choose between different optimal the median just-noticeable-difference quality. In visualisations, for example, points on the diagram. Their results of other visualisation formats. This an availability bias can be created by confirmed that respondents are more might mitigate the need for exten- having to assemble a set of documents attracted to optimal options situated sive empirical experimentation, and on the screen before undergoing the near decoys. It was suggested that support more targeted testing of visu- analysis stage; information in these dynamic visualisation formats could alisation formats. It could do so by available documents may be given help to de-bias such decisions, includ- excluding certain poorly-performing undue weight in comparison to infor- ing ‘computational aids that highlight formats in advance, and/or by help- mation that is sought out elsewhere optimal decisions based on objective ing to isolate specific design features during analysis (Ellis & Dix, 2015). criteria, but more counterintuitively, that are responsible for differences in a system where users can systemati- performance. FRAMING EFFECT cally delete information as they make comparative decisions at a more local OSTRICH EFFECT AND First mentioned by Goffman in level of analysis’ (Dimara et al., 2018). RISK COMPENSATION 1974, the framing effect identifies BIAS how the presentation or ‘framing’ of information can influence the CONFIRMATION BIAS The Ostrich effect is represented in decision-making process. If options The confirmation bias describes the the tendency to neglect information are presented through positive or tendency to accept information or that generates a feeling of discom- negative semantics, settings, or situ- evidence that confirms preexisting fort, and leads respondents to over- ations, decision-makers are likely to hypotheses, and which results in the look information or to downplay incorporate these features into their information that would be consid- judgements, even when they have no ered negative (Valdez et al., 2018a; bearing on the factual circumstances. Dimara et al., 2018). Similarly, risk For example, a policy option may be compensation bias is the tendency prove attractive when it is presented to adjust behaviour in response to in terms of how much it would save risk, being more cautious during rather than how much it would cost, cases of greater perceived risk and even if both description contain less cautious in situations of protec- mathematically identical information. tion and security (Dulisse, 1997). for One special case of the fram- example, drivers wearing seat-belts ing effect is the attraction effect have been shown to drive somewhat (Mansoor & Harrison, 2017). The less cautiously (Janssen, 1994). Both attraction effect (or decoy effect) is 38 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

denial or dismissal of information object signal strength which depends that functions contrarily to those on object size, number of objects in beliefs (Rajsic et al., 2015). In visu- the visual scene, and object distance alisation research, this bias has been to the centre of the scene.’ (Peschel shown to guide respondents’ atten- & Orquin, 2013). The surface size tion, and to lead to a visual selection effect has shown to exert a robust and of information that has been initially medium to strong effect on fixation prioritised, even when the strategy is likelihood (Peschel & Orquin, 2013). not the most optimal for the task at Larger objects are likely to diminish hand (Rajsic et al., 2015; Padilla et the attention toward smaller objects al., 2018). As a consequence, sugges- (i.e. the decision-maker is more tions have been made to enable likely to fixate on larger objects). The a high-relevance label in a predict- mitigation by analysing a range of surface size effect does not only exist able location as opposed to low- and competing hypotheses that require in visualisations, but has also been medium-relevance objects in predict- careful consideration before reaching found in text-based information, with able locations and unpredictable a conclusion (Dimara et al., 2018; cf. an increase in the surface size related locations, respectively (Orquin et a confirmation bias mitigation soft- to the text significantly and positively al., 2018). In other words, ‘predict- ware [Wright et al., 2006]). impacting selective attention and able locations enhance top-down attention span (Rik & Wedel, 2004). control by allowing decision makers SET SIZE EFFECT to attend or ignore information that POSITION EFFECTS Since attention is a limited resource, they perceive to be important or irrel- AND PREDICTABLE increasing the set size (i.e. the number evant’ (Orquin et al., 2018). LOCATIONS of elements in a data set or a visualisa- EMOTIONAL STIMULI tion) typically leads decision-makers When information is structured in a to fixate on a smaller proportion of a one-dimensional array, viewer have Emotional stimuli of typically nega- set of data, and to an increase response a strong tendency to begin reading tive valence (e.g. angry faces or latency in visual search tasks (Spinks from the top of a column towards the spiders) and those of typically posi- and Mortimer, 2015; Palmer, 1993). bottom (Chen & Pu, 2010; Simola tive valence (baby faces or mini pigs) This effect is a common heuristic and et al., 2011) or from the left of a row both attract attention faster and with can lead to attribute non-attendance, towards the right (Navalpakkam et a higher likelihood than emotion- whereby respondents ignore one or al., 2012). for example, Navalpakkam ally neutral stimuli (Calvo & Lang, more attributes when making deci- et al. (2012) found, contrary to their 2004). When comparing negative sions. Tasks involving increasing hypothesis that semantic factors such versus positive stimuli alone, negative complexity of decision problems have as user interest in the content would pictures tend to create a greater impact demonstrated that the set size is one matter most for sustaining attention, on our visual attention than positive of the strongest predictors of attrib- that the position effect remained stimuli (Nummenmaa, Hyönä, and ute non-attendance. Other factors dominant, with significantly higher Calvo, 2006; for a detailed review, include time pressure and prior expe- dwells at top/left positions than see Pessoa and Ungerleider, 2015). rience with the decision problem. It is other positions. Similarly, in two- In visual search tasks, this is espe- believed that increasing the set size dimensional arrays, the viewer tends cially the case during the first 500ms, does tend to impede visual selec- to fixate on the middle of the array, suggesting that emotional stimuli are tion, even when it allows for desir- whereas the corners often go unno- detected within preattentive process- able enhancement of local feature ticed (Meißner, Musalem, & Huber, ing (Calvo and Lang, 2004). As such, contrasts (Becker and Ansorge, 2016). However, these findings only Calvo and Lang (2004) conclude that 2013). account for Western societies where ‘this early allocation of attention to the reading direction is from left to pleasant and unpleasant stimuli [is] SURFACE SIZE EFFECT right. Furthermore, attention can also a central cognitive mechanism in The surface size is the area the object be guided by controlling the predict- the service of prompt detection of occupies within an environment ability of object locations (Orquin important events and activation of (Peschel & Orquin, 2013), and may et al., 2018). Studies show that motivational resources for approach best be explained as an ‘increase in participants are more likely to fixate or avoidance.’ 39

Graphical literacy, just like literacy, needs to be taught and developed.

INTER-INDIVIDUAL the difficulty that experts experi- conclusions not just from what is VARIABILITY ence when trying to put themselves shown but from a negative space, i.e. in the position of non-experts. To be from what is not depicted: ‘Under- Systematic errors can also arise when an expert in a given domain does not standing how much of what is not visualisation formats fail to take into necessarily entail good awareness of shown is fixed, and what can be account inter-individual differences, when or how a particular judgment varied, is as essential as understand- including cognitive characteristics, draws on that domain knowledge, ing the explicit content of a represen- cultural background, and types and let alone the skill of helping others tation. Alternative interpretations levels of expertise (Conati et al., 2014; to join one in expert perspectives and of the omitted elements of a design Lallé et al., 2015; Green & Fischer, reasoning. Xiong et al. (2019) show are made possible by uncertainty or 2010). In these cases, adapting visu- that ‘when people are primed to see misunderstanding about the inter- alisations accordingly and correcting one pattern in the data as visually pretive conventions to be applied for human factors is key, especially salient, they believe that naive view- to a representation, as well as the when the task increases in complex- ers will experience the same visual context in which it is embedded and ity and cognitive workload required salience.’ the assumptions the generator makes (Valdez et al., 2018a; Micallef et al., Boone et al. (2018) point out the about how the gaps will be filled in. 2017; Parson, 2018). Ensuring stake- visualisation of complex informa- Thus it can be ambiguous by omis- holders are empowered and motivated tion, such as uncertainty informa- sion. In other words, what is implicit to share their beliefs and knowledge tion, is constrained by a general in any representation depends on the is critical to enabling a shared mental lack of knowledge and experience interpretive skills of the recipient and understanding of the task (Birch & with reading a graphical language. the extent of the shared understand- Bloom, 2007). The use of a visualisation as a ‘Another factor that can greatly influ- ing of context established between common reference point offers no ence the effectiveness of a graphic is the sender and the recipient.’ guarantee that stakeholders will use the knowledge the viewer has about it in similar or compatible ways. Even the conventions of the graphic type in robustly designed visualisations may question.’ Graphical literacy, just like sometimes accommodate conflicting literacy, needs to be taught and devel- understandings, which may go unno- oped; such knowledge according to ticed if there is insufficient validation the authors is often insufficient. See and/or insufficient opportunities for also ‘The semantic principle and the reflection and dialogue. Moreover, principle of appropriate knowledge,’ communication about the develop- discussed earlier in this section. ment of a visualisation format, or Lastly, Stacey and Eckert (2001) the meaning of a completed visu- warn about another potential source alisation, can be impeded by the of misinterpretation that stems from ‘curse of knowledge.’ This refers to the fact that the user is drawing 40 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

9 CONCLUDING REMARKS The literature on communicating and needs. There is not yet any deep uncertainty is vast and mixed. This theory to formalise which differ- catalogue offers a very brief selection, ences are relevant in any given case. guided by several criteria: assessment Uncertainty representation is not of the quality of the research; of its easily separable from interpretations relevance to the decision-making of value, thus individual and group context; and of its capacity to high- differences (cultural, political, social, light the art of the possible in visuali- linguistic, etc) play an important role. sation of uncertainty. One clear recommendations that There is no ‘optimal’ format for emerges from this multidisciplinary communicating uncertainty, nor literature review is that the imple- is it easy to determine if and how a mentation of visualisation tech- format influences decision-making. niques must be studied on a case- Identifying instances in which the by- case basis, and ideally supported communication format has a signifi- by empirical testing. The success of cant impact is key to improving the different techniques for visualising communication of uncertainty. It uncertainty is highly context-sensi- is desirable to develop communica- tive, and current understanding of tion formats through close dialogue how to differentiate relevant contex- between designers and end-users. tual factors appears patchy. In short, Self-reporting is unreliable for assess- current research offers an insufficient ing the effectiveness of a communi- basis for robust generalisations about cation format; evaluations should be the visualisation of uncertainty. performed by methodologies which Complex decision-making inevi- bear hallmarks of reproducible tably involves visual information to research. some degree. From the organisa- Across the spectrum of uncertainty tion of text on a page or screen, to types, a common set of visualisation the internal mental representations techniques are potentially applica- evoked by verbal reasoning, there is a ble. There is no consistent mapping potential visual aspect to all practices between categories of uncertainty and of analysis, communication, and deci- methods for visualising uncertainty, sion-making. Thus even if we attempt and it is an open question whether to simply opt out of the visualisation such a mapping is either possible or of uncertainty, we still risk allowing desirable. The users of uncertainty visuals to shape our processes in ways information have diverse capacities which go unregistered and unstudied. NOVEL APPROACHES TO 41 UNCERTAINTY VISUALISATION

There is currently a paucity of visual professionals working in marketing and and feedback. In this respect, it may also signifiers of uncertainty that can be advertising agencies, and the creative be useful to take a media-archaelogical consistently interpreted. The available industries more generally, respond to approach and examine the historically graphical language has too few elements, commercial incentives to find new and evolving conventions of visual represen- and interpretative communities are too distinctive ways of communicating visu- tation within computer games. In addi- diverse and fragmented to interpret ally. Within popular culture, creators of tion to computer games, other digital it consistently. In this catalogue, we science fiction often depict imaginary applications, as well as analog games such provide some basic building blocks, user interfaces, and imagine how such as tabletop wargames and ‘eurogames,’ as well as perspectives, concepts, and interfaces might be woven into their may offer starting points for thinking methods that may be useful in develop- characters’ everyday working lives. innovatively about the visualisation of ing and testing visualisation formats. Decision analysis can also benefit uncertainty. Finally, there is also now a Throughout this catalogue we empha- from interdisciplinary input, and not just strong and ever-evolving visual dimen- sise the importance of developing these from the most obvious candidate disci- sion to everyday online communication, on a case-by-case basis, using objective plines, such as user experience design. from emoticons and emojis, to reaction and reproducible testing, ideally with the for example, in the digital humanities— GIFs and memes, to filters and editing relevant end-users. However, there are across literary studies, history, heritage tools for user-produced visual content. also contexts in which, for one reason or studies, and other discplines—research- Of course, many of these wider visual another, more informal and exploratory ers have long been exploring new ways cultures are not be specifically concerned approaches are appropriate. of course, of curating texts, artefacts, and other with communicating uncertainty infor- tailored evidence-based solutions are objects of study, and managing and mation (although they are sometimes vital to advance the field. But so too are augmenting human attention and analy- interested in communicating fairly those spaces where designers, artists, sis. To give just one small example, the complex information). They have their data journalists, and researchers, among classic open source application Voyant own native agendas and values, and don’t others, can explore the potential of visu- Tools offers basic analytics for textual exist primarily to complement the needs alisation in free and open ways, or in corpora and a suite of dozens of visuali- of decision-making under uncertainty. response to constraints other than those sation options. Nevertheless, from our perspective, associated with decision support. These Perhaps most significantly of all, visu- such cultures are important in several wider visual cultures are important in alisation and decision making intersect respects. First, they can exert an influ- that they nourish the visual imagination in an extremely rich way in computer ence on graphic literacy. That is, they and are an informal testing ground for games. Computer games often need form spaces in society where certain novel visualisation techniques. to represent relatively complex infor- visual signifiers can be developed and For example, visual and conceptual mation in ways that feel intuitive and popularised, and where people can learn artists frequently engage with data, immersive for players. The conven- and reinforce certain ways of thinking sometimes exploring new ways of repre- tions that are widely understood within visually. In other parts of their lives, the senting data and/or new affordances for gaming communities, and are adopted users of decision support systems may interacting with it. Experimental musi- by major games developers on big budget have had significant exposure to such cians such as Cathy Berberien, Hans- projects, are of interest; so too are the conventions. Second, by looking to Christoph Steiner, and many more potentially more eccentric visualisa- visual innovation within domains such have also developed numerous types of tion formats invented by indie game art, design, cinema, the digital humani- visual notation and an associated body developers. Game design studies offers ties, gaming and game studies, and so of theory. Moreover, data journalists conceptual frameworks around the use on, we can gain direct inspiration for constantly seeking bold, striking, and/ of visual hierarchy, a visual language, novel formats, signifiers, techniques, and or beautiful ways of using data visually, visual themes, calls to action, and so on, so on to test out in an empirically robust and of conveying complexity in ways and the games themselves offer speci- way. Finally, it is plausible that partici- that are clear and persuasive. Likewise mens that test and perhaps exceed such pation in these wider visual cultures communications professionals working concepts. While game development has enriches our visual imaginations more in campaign organisations and NGOs its own distinct goals, and playtesting generally, putting us in contact with a have a strong and longstanding inter- is not conducted according to the same diversity of visual forms that indirectly est in the cognitive and emotional evidentiary standards as most empiri- supports the both the tasks of empathy- effects of visualisation formats, espe- cal science, game development does building and the creative leaps that are cially in relation to environmental crisis tend to be heavily iterative, and incor- often necessary to develop an effective and economic inequality. Similarly, porates a great deal of user experience visualisation format. 42 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

10 BIBLIOGRAPHY A… Barnett, V. Comparative Statistical Boettcher, S.E.P., D. Draschkow, Inference, 3rd Ed. Chichester: John E. Dienhart, and M.L. Vo. Aerts, Jeroen C.J.H., Keith C. Wiley and Sons (1999). “Anchoring Visual Search in Scenes: Clarke, and Alex D. Keuper. Assessing the Role of Anchor “Testing Popular Visualization Becker, S.I. and U. Ansorge. Objects on Eye Movements During Techniques for Representing Model “Higher Set Sizes In Pop-Out Visual Search.” Journal of Vision, Vol. Uncertainty.” Cartography and Search Displays Do Not Eliminate 18, No. 13 (2018). Geographic Information Science, Vol. Priming or Enhance Target Selec- 30, No. 3 (2003): 249–61. tion.” Vision Research, Vol. 81 (2013): Bogomolova, Svetlana, Harmen 18–28. Oppewal, Justin Cohen, and Jun Appleton, Katy, Andrew Lovett, Yao. “How The Layout of a Unit Trudie Dockerty, and Gisela Bennett, P.G. “Towards A Theory Price Label Affects Eye-Movements Sünnenberg. “Representing Uncer- of Hypergames.” Omega Vol. 5 and Product Choice: An Eye-Track- tainty in Visualisations of Future (1977): 749–51. ing Investigation.” Journal of Business Landscapes.” Proceedings of the XXth Beyth-Marom, Ruth. “How Research (2018). ISPRS Congress (2004). Probable Is Probable? A Numerical Boone, A.P., P. Gunalp, and M. Atalay, A.S., H.O. Bodur, and Translation of Verbal Probability Hegarty. “Explicit Versus Action- D. Rasolofoarison. “Shining in Expressions.” Journal of Forecasting, able Knowledge: The Influence of the Center: Central Gaze Cascade Vol. 1, No. 3 (1982): 257–69. Explaining Graphical Conventions Effect on Product Choice.” Journal on Interpretation of Hurricane Vol. 39, No. 4 Birch, S.A.J., and P. Bloom. “The of Consumer Research, Forecast Visualizations.” Journal (2012): 848–66. Curse of Knowlege in Reason- ing about False Beliefs.” Psycho- of Experimental Psychology Applied, logical Science Vol. 18, No. 5 (2007): Vol. 24, No. 3 (September 2018): 382–86. 275–95. B… Blanco, F. “Positive and Negative Booth, P., N. Gibbins, and G. Banks, David L., Jesus M. Rios Implications of the Causal Illusion.” Spyros. “Toward a Theory of Aliaga, and David Ríos Insua. Consciousness and Cognition Vol. 50 Analytical Behaviour: A Model Adversarial Risk Analysis. Boca (2017): 56–68. of Decision-Making in Visual Raton: Chapman & Hall / CRC Analytics.” In Proceedings of the 52nd Press (2015). Blight, B.J.N. and L. Ott. “A Hawaii International Conference on Bayesian Approach to Model Inad- Barnes, Alan. “Making Intelligence System Sciences. Honolulu: Univer- equacy for Polynomial Regression.” Analysis More Intelligent: Using sity of Hawaii at Manoa (2019): Biometrika, Vol. 62, No. 1 (1975): Numeric Probabilities.” Intelligence 1607–16. 79–88. and National Security, Vol. 31, No. 3 (2016): 327–44. 43

Bostrom, A., L. Anselin, and J. Chen, Li and Pearl Pu. “Eye- Conti, Stefano and Anthony Farris. “Visualizing Seismic Risk Tracking Study of User Behavior in O’Hagan. “Bayesian Emulation and Uncertainty: A Review of Recommender Interfaces.” In Paul of Complex Multi-Output and Related Research.” Annals of the New De Bra, Alfred Kobsa A., David Dynamic Computer Models,” Jour- York Academy of Sciences, Vol. 1128, Chin (eds), User Modeling, Adapta- nal of Statistical Planning and Infer- No. 1 (2008): 29–40. tion, and Personalization. Berlin: ence, Vol. 140, No. 3 (2010): 640–51. Springer (2010). Brynjarsdóttir, Jenný and Anthony Correll, Michael, Dominik O’Hagan. “Learning About Physi- Cheong, Lisa, Susanne Bleisch, Moritz, and Jeffrey Heer. “Value- cal Parameters: The Importance of Allison Kealy, Kevin Tolhurst, Suppressing Uncertainty Palettes.” Model Discrepancy,” Inverse Prob- Tom Wilkening, and Matt Duck- Proceedings of the 2018 CHI Confer- lems, Vol. 30, No. 11 (2014): 114007. ham. “Evaluating the Impact of ence on Human Factors in Computing Visualization of Wildfire Hazard Systems (2018). Budescu, David V., and Thomas Upon Decision-Making Under S. Wallsten. “Processing Linguistic Correll, Michael. “Error Bars Uncertainty.” International Journal of Probabilities: General Principles and Considered Harmful: Exploring Geographical Information Science, Vol. Empirical Evidence.” In Psychology 30, No. 7 (2016): 1377–1404. Alternate Encodings for Mean and of Learning and Motivation, Vol. 32 Error.” IEEE Transactions on Visu- (1995): 275–318. Chung, Jae and Steven Wark, and alization and Computer Graphics, Vol. Joint and Operations Analysis 20, No. 12 (2014): 2142–51. Budescu, David V., Han-Hui Por, Division Defence Science and and Stephen B Broomell. “Effec- Technology Group. Visualising De Finetti, B. Theory of Probability, tive Communication of Uncertainty Uncertainty for Decision Support Vol. 1. Chichester: John Wiley and in the IPCC Reports.” Climatic (2016). Sons (1974). Change, Vol. 113, No. 2 (2012): 181–200. Clarke, Valerie A., Coral L. Deitrick, Stephanie and Elizabeth Ruffin, David J Hill, and Arthur A. Wentz. “Developing Implicit L. Beamen. “Ratings of Orally Uncertainty Visualization Methods Motivated by Theories in Decision C… Presented Verbal Expressions of Probability by a Heterogeneous Science.” Annals of the Association of Vol. 105, No. Cairo, Alberto. “Introduction: Sample.” Journal of Applied Social American Geographers, 3 (2015): 531–51. Infographics and Visualization.” In Psychology, Vol. 22, No. 8 (1992): Alberto Cairo (ed.), The Functional 638–56. Deitrick, Stephanie. “Evaluating Art: An introduction to Information Implicit Visualization of Uncer- Graphics and Visualization. Berkeley: Comes, Tina, Niek Wijngaards, tainty for Public Policy Decision New Riders (2012). and Bartel Van De Walle. “Explor- ing the Future: Runtime Scenario Support.” Proceedings AutoCarto Cairo, G., and P.J. Lang. “Gaze Selection for Complex and Time- (2012). Patterns when Looking at Bound Decisions.” Technological Dhami, Mandeep K., David R. Emotional Pictures: Motivationally Forecasting and Social Change, Vol. Mandel, Barbara A. Mellers, and Biased.” Motivation and Emotion, 97, No. 1 (2015): 29–46. Philip E. Tetlock. “Improving Vol. 28, No. 5 (2004): 221–43. Conati, C., G. Carenini, B. Intelligence Analysis With Decision Calvo, M.G. and P.J. Lang. Steichen, and D. Toker. “Evaluat- Science.” Perspectives on Psychologi- “Gaze Pattern When Looking at ing the Impact of User Charac- cal Science, Vol. 10, No. 6 (2015): Emotional Pictures: Motivation- teristics and Different Layouts on 753–57. ally Biased Attention.” Motivation an Interactive Visualization for Dieckmann, Nathan F., Robin and Emotion, Vol. 28, No. 3 (2004): Decision Making.” In EuroVis ‘14: Gregory, Ellen Peters, and Robert 221–43. Proceedings of the 16th Eurographics Hartman. “Seeing What You Want Carr, H. “An Interpretation of The Conference on Visualization (2014): To See: How Imprecise Uncer- Weber-Fechner Law.” Psychologi- 371–80. tainty Ranges Enhance Motivated cal Review, Vol. 34, No. 4 (1927): Reasoning.” Risk Analysis, Vol. 37, 313–19. No. 3 (2017): 471–86. 44 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

Dimara, E., G. Bailly, A. Beze- Ellis, G., and A. Dix. “Decision French, S. “Cynefin: Uncertainty, rianos, and S. Franconeri. “Miti- Making Under Uncertainty in Visu- Small Worlds and Scenarios.” Jour- gating the Attraction Effect with alisation?” IEEE workshop on Visu- nal of the Operational Research Society, Visualizations.” IEEE Transac- alization for Decision Making Under Vol. 66, No. 10 (2015): 1635–45. tions on Visualization and Computer Uncertainty. VIS2015, Chicago French, S. “Modelling, Making Graphics (2018). (2015). Inferences and Making Decisions: Dimara, E., P. Dragicevic, and Evans, J.R. and D.L. Olson. The Roles of Sensitivity Analysis.” A. Bezerianos. “Accounting for Introduction To Simulation and Risk TOP, Vol. 11, No. 2 (2003): 229–52. Availability Biases in Information Analysis. Upper Saddle River: Pren- Visualization.” In G. Ellis (ed.), tice Hall (2002). French, S. and D. Rios Insua. London: DECISIVe: Workshop on Dealing with Statistical Decision Theory. Evans, Jonathan. “Dual-Processing Arnold (2000). Cognitive Biases in Visualisations. Accounts of Reasoning, Judgment, Paris: IEEE VIS2014 (2014). French, S., A.J. Maule, and K.N. and Social Cognition.” Annual Papamichail. Dimara, E., S. Franconeri, C. Review of Psychology, Vol. 59 (2008): Decision Behaviour, Plaisant, A. Bezerianos, and 255–78. Analysis and Support. Cambridge: P. Dragicevic. “A Task-Based Cambridge University Press (2009). Taxonomy of Cognitive Biases for Friedman, Jeffrey A., Joshua D. Information Visualization.” IEEE F… Baker, Barbara A. Mellers, Philip Transactions on Visualization and E. Tetlock, and Richard Zeck- Fagerlin, A., C. Wang, and P.A. Computer Graphics (2018). hauser. “The Value of Precision in Ubel. “Reducing the Influence of Probability Assessment: Evidence Draper, D. “Assessment and Propa- Anecdotal Reasoning on People’s from a Large-Scale Geopolitical gation of Model Uncertainty (With Health Care Decisions: Is a picture Forecasting Tournament.” Interna- Discussion),” Journal of The Royal worth a thousand statistics?” Medi- tional Studies Quarterly, Vol. 62, No. Statistical Society B57 (1995): 45–97. cal Decision Making, Vol. 25, No. 4 2 (2018): 410–22. (2005): 398–405. Dulisse, B. “Methodological Issues Fuchs, R., and H. Hauser. “Visu- in Testing the Hypothesis of Risk Figueiras, Ana. “ Towards the alization of Multi-Variate Scientific Compensation.” Accident Analysis Understanding of Interaction Data.” Computer Graphics Forum, & Prevention, Vol. 29, No. 3 (1997): in Information Visualization.” Vol. 28, No. 6 (2009): 1670–90. 285–92. Presented at the 2015 19th Interna- tional Conference on Information Visualisation (2015). E… G… Finger, Richard, and Ann M. Einhauser, W., U. Rutishauser, Bisantz. “Utilizing Graphical Gamerman, Dani and Hedibert F. and C. Koch. “Task-Demands Can Formats to Convey Uncertainty in a Lopes. Markov Chain Monte Carlo: Immediately Reverse the Effects Decision Making Task.” Proceedings Stochastic Simulation for Bayesian Boca Raton: Chapman & of Sensory-Driven Saliency in of The Human Factors and Ergonomics Inference. Hall / CRC Press (2006). Complex Visual Stimuli.” Journal of Society Annual Meeting, Vol. 44, No. Vision, Vol. 8, No. 2 (2008): 1-19. 1 (2000): 13–16. Garcia-Retamero, R., and M. Eiselt, H.A., and Vladimir French, S. “Communicating Galesic. “Who Profits from Visual Marianov. “Multicriteria Deci- Geographical Risks in Crisis Aids: Overcoming Challenges in sion Making Under Uncertainty: Management of Radiological Acci- People’s Understanding of Risks [Corrected].” A Visual Approach.” International dents.” Risk Analysis, Vol. 39, No. 1 Social Science and Vol. 70, No. 7 (2010): Transactions in Operational Research, (2019): 9–16. Medicine, Vol. 21, No. 4 (2014): 525–40. 1019–25. French, S. “Cynefin, Statistics and Gelman, A. et al. Decision Analysis.” Journal of the Bayesian Data 3rd Ed. London: Chapman Operational Research Society, Vol. 64, Analysis. No. 4 (2013): 547–61. and Hall (2013). 45

Gilbert, D.T., R.P. Brown, E.C. H… Ho, Emily H., David V. Budescu, Pinel, and T.D. Wilson. “The Illu- Mandeep K. Dhami, and David R. Halvorson, Craig S., Alain sion of External Agency.” Journal of Mandel. “Improving the Commu- Bouchard, Daniel Lehrfeld, Anna- Personality and Social Psychology, Vol. nication of Uncertainty in Climate 79, No. 5 (2000): 690. Liesa S. Lapinski, Jérôme Lavoie, Science and Intelligence Analysis.” Theodore T. Saito, and Jean Roy. Behavioural Science & Policy, Vol. 1, Gleicher, Michael, Danielle “Evaluation of Information Visuali- No. 2 (2015): 43–55. Albers, Rick Walker, Ilir Jusufi, zation Approaches for An Enhanced Charles D. Hansen, and Jonathan Recognized Maritime Picture.” Hollingworth, A. “Guidance of Roberts, C. “Visual Comparison for Presented at Optics and Photonics in Visual Search by Memory and Information Visualization.” Infor- Global Homeland Security IV (2008). Knowledge.” Nebraska Symposium on mation Visualization, Vol. 10, No. 4 Motivation, Vol. 59 (2012): 63–89. (2011): 289–309. Hannus, A., S.F. W. Neggers, F. W. Cornelissen, and H. Bekker- Goerlandt, Floris, Habtamnesh ing. “Selective Attention for Action. Goite, Osiris A. Valdez Banda, New Evidence From Visual Search Hullman, J., X. Qiao, M. Correll, A. Kale, and M. Kay. “In Pursuit Anders Höglund, Paula Ahonen- Studies.” In Attention In Action: of Error: A Survey of Uncertainty Rainio, and Mikko Lensu. “An Advances from Cognitive Neurosci- Visualization Evaluation.” Analysis of Wintertime Naviga- ence, ed. G. W. Humphreys and M. IEEE tional Accidents in the Northern J. Riddoch, 131–51. Hove and New Transactions on Visualization and Vol. 25, No. 1 Baltic Sea.” Safety Science, Vol. 92 York: Psychology Press (2005). Computer Graphics, (2017): 66–84. (2018). Harold, Jordan, Irene Lorenzoni, Hullman, Jessica. “Why Evaluating Goffman, E. Frame Analysis: An Kenny R. Coventry, and Asher Uncertainty Visualization is Error Essay on the Organization of Experi- Minns. Enhancing the Accessibil- Prone.” ence. Cambridge: Harvard Univer- ity of Climate Change Data Visuals: Proceedings of The Beyond sity Press (1974). Recommendations to the IPCC and Time and Errors on Novel Evaluation guidance for researchers. Norwich: Methods for Visualization - BELIV Goldstein, M. and J. C. Rougier. Tyndall Centre for Climate Change ’16 (2016). “Reified Bayesian Modelling and Research (2017). Inference for Physical Systems (With Discussion).” Journal of Statis- Harrison, L., F. Yang, S. Fran- I… tical Planning and Inference, Vol. 139 coneri, and R. Chang. “Ranking (2009): 1221–56. Visualisations of Correlation Using Itti, L., and C. Koch. “A Saliency- Weber’s Law.” IEEE Transactions on Based Search Mechanism for Goldstein, Michael. “Bayesian Visualization and Computer Graphics, Overt and Covert Shifts of Visual Analysis for Computer Simula- Vol. 20, No. 12 (2014). Attention.” Vision Research, Vol. 40 tors (With Discussion).” In J.M. (2000): 1489–1506. Bernardo et al. (eds). Bayesian Harsanyi, J.C. Rational Behaviour Statistics 9. Oxford: Oxford Univer- and Bargaining Equilibrium in Games sity Press (2011). and Social Situations. Cambridge: J… Cambridge University Press (1977). Green, T.M., and B. Fisher. Janssen, W. “Seat-belt Wearing “Towards The Personal Equation of Healey, C. and J. Enns. “Attention and Driving Behavior: An instru- Interaction: The Impact of Person- and Visual Memory in Visualization mented-vehicle study.” Accident ality Factors on Visual Analytics and Computer Graphics,” in IEEE Analysis and Prevention, Vol. 26, No. Interface Interaction.” Presented at Transactions on Visualization and 2 (1994): 249–61. IEEE Symposium on Visual Analytics Computer Graphics, Vol. 18, No. 7 Science and Technology (2010). (2012): 1170-1188. Jeffreys, H. Theory of Probability, 3rd Ed. Oxford: Oxford University Press (1961). 46 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

Johannessen, Trond Vegard, Kinkeldey, Christoph, Alan M. L… and Anna Mette Fuglseth. “The MacEachren, and Jochen Schiewe. Effectiveness of Data Presentation “How to Assess Visual Communica- Lallé, S., D. Toker, C. Conati, and Formats: An Exploratory Study,” tion of Uncertainty? A Systematic G. Carenini. “Prediction of Users’ Learning Curves for Adaptation NOKOBIT, Vol. 22, No. 1 (2014). Review of Geospatial Uncertainty While Using an Information Visu- Visualisation User Studies.” The alization.” In Cartographic Journal, Vol. 51, No. 4 IUI 2015: Visualization / Video / Augmented Reality (2015): K… (2014): 372–86. 357–68. Kahneman, D. Thinking, Fast and Kinkeldey, Christoph, Alan M. Langeborg, L., and M. Eriksson. Slow. New York: Farrar, Straus and MacEachren, Maria Riveiro, and “Anchoring In Numeric Judgments Giroux (2011). Jochen Schiewe. “EvaluatingtThe Effect of Visually Represented of Visual Stimuli.” Frontiers in Kahneman, D., P. Slovic, and Geodata Uncertainty on Deci- Psychology, Vol. 7 (2016): 225. A. Tversk y. Judgment Under sion- Making: Systematic Review, Lapinski, Anna-Liesa S. “A Strat- Uncertainty: Heuristics and Biases. Lessons Learned, and Recommen- egy for Uncertainty Visualization Cambridge: Cambridge University dations.” Cartography and Geographic Design.” Technical Memorandum, Press (1982). Information Science, Vol. 44, No. 1 Defence R&D Canada (2009). Karlsson, Niklas, George Loewen- (2015): 1–21. Leach, Adrian W., Polina Levon- stein, and Duane Seppi. “The Knight, F.H. Risk, Uncertainty and tin, Johnson Holt, Laurence T. Ostrich Effect: Selective Attention Profit. Boston, MA: Hart, Schaf- Kell, and John D. Mumford. to Information.” Journal of Risk and fner & Marx; Houghton Mifflin “Identification and Prioritization Uncertainty, Vol. 38, No. 2 (2009): Company (1921). of Uncertainties for Management 95–115. Koch, C., and S. Ullman. “Shifts In of Eastern Atlantic Bluefin Tuna Kay, M., and Jeffrey Heer. “Beyond Selective Visual Attention: Towards (Thunnus Thynnus).” Marine Policy, Weber’s Law: A Second Look At The Underlying Neural Circuitry.” Vol. 48 (2014): 84–92. Ranking Visualisations of Correla- Human Neurobiology Vol. 4 (1985): Leskens, J. G., C. Kehl, T. Tutenel, tion.” IEEE Transactions on Visu- 219–27. T. Kol, G. Haan, G. Stelling, alization and Computer Graphics, and E. Eisemann. “An Interactive Vol. 22, No. 1 (2016): 469–78. Korb, K.B. and A.E. Nicholson. Simulation and Visualization Tool Bayesian Artificial Intelligence. Boca Kelton, Andrea Seaton, Robin R. Raton: Chapman and Hall/CRC for Flood Analysis Usable for Prac- Pennington, and Brad M. Tuttle. (2004). titioners.” Mitigation and Adaptation “The Effects of Information Pres- Strategies for Global Change, Vol. 22, entation Format on Judgment and Kovach, C.K., and R. Adolphs. No. 2 (2017): 307–24. Decision Making: A Review of the “Investigating Attention in Complex Levontin, Polina, Paul Baranow- Information Systems Research.” Visual Search.” Vision Research, Vol. ski, Adrian W. Leach, Alexandria Journal of Information Systems, Vol. 116, Part B (2015): 127–41. Bailey, John D. Mumford, Antoni 24, No. 2 (2010): 79–105. Kristjansson, A. “Simultaneous Quetglas, and Laurence T. Kell. Kent, Sherman. “Words of Estima- Priming along Multiple Feature “On the Role of Visualisation in tive Probability,” Studies in Intelli- Dimensions in a Visual Search Fisheries Management.” Marine gence, Vol. 8, No. 4 (1964): 49–65. Task.” Vision Research, Vol. 46, Policy, Vol. 78 (2017): 114–21. No. 16 (2006): 2554–70. Kesselman, Rachel F. Verbal Lim, K.T.K., E. Fradkin, A. Probability Expressions in National Kurowicka, Dorota. Uncertainty Miloschewski, D.A. Selby, A. Intelligence Estimates: A Compre- Analysis with High Dimensional Trendl, N. Chater, M.J. Fiecas, hensive Analysis of Trends from the Dependence Modelling. John Wiley & M.R. Munafò, D. Read, and J.L. Fifties through Post 9/11. Mercyhurst Sons Ltd (2006). Hutton. “Statistical Practice and College (2008). Replication Success in Behavioural Science: An Evaluation of Published Studies.” (Submitted).

47

Lim, Kenneth. “Microthesis: Matthews, Michael L. and Lisa Milosavljevic, M., V. Navalpa- Statistical Practice and Replication Rehak. “Improving the Maritime kkam, C. Koch, and A. Rangel. Success in Behavioural Science.” Surface Picture with a Visualiza- “Relative Visual Saliency Differ- London Mathematical Society News- tion Aid to Provide Rapid Situation ences Induce Sizable Bias In letter (July 2018). Awareness of Information Uncer- Consumer Choice.” Journal of tainty.” IEEE Toronto International Consumer Psychology, Vol. 22, No. 1 Lohse, G.L. “Consumer Eye Move- Conference Science and Technology for (2012): 67–74. ment Patterns on Yellow Pages Humanity (2009): 533–38. Advertising.” Journal of Advertising, Moore, Jeanne. “Data Visualiza- Vol. 26, No. 1 (1997): 61–73. Matthews, Michael L., Lisa tion in Support of Executive Deci- Rehak, Anna-Liesa S. Lapinski, sion Making.” Interdisciplinary and Sharon Mcfadden. “Improving Journal of Information, Knowledge, M… The Maritime Surface Picture with and Management, Vol. 12 (2017): a Visualization Aid to Provide Rapid 158–138. MacEachren, Alan M. “Visualiz- Situation Awareness of Information ing Uncertain Information.” Carto- Uncertainty.” Pesented at the IEEE Morgan, M. Granger. Best Prac- graphic Perspectives, Vol. 13 (1992). Toronto International Confer- tice Approaches for Characterizing, Communicating and Incorporating MacEachren, Alan M. and Robert ence Science and Technology for Humanity (2009): 533–38. Scientific Uncertainty in Climate Deci- E. Roth. “Visual Semiotics & sion Making. DIANE Publishing Uncertainty Visualization: An Matthews, Michael L., Lisa (2009). Empirical Study.” IEEE Transac- Rehak, Julie Famewo, Tamsen tions on Visualization and Computer Taylor, and Jeremy Robson. Graphics, Vol. 18, No. 12 (2012). “Evaluation of New Visualization N… MacEachren, Alan M., Anthony Approaches for Representing Uncer- tainty In The Recognized Maritime Nadav-Greenberg, L., S.L. Joslyn, Robinson, Susan Hopper, Steven and M.U. Taing. “The Effect of Gardner, Robert Murray, Mark Picture.” Defence R&D Canada – Atlantic (2008). Uncertainty Visualizations on Deci- Gahegan, and Elisabeth Hetzler. sion Making in Weather Forecast- “Visualizing Geospatial Informa- McMahon, Rosemarie, Michael ing.” Journal of Cognitive Engineering tion Uncertainty: What We Know Stauffacher, and Reto Knutti. and Decision Making, Vol. 2, No. 1 and What We Need to Know.” “The Unseen Uncertainties (2008): 24–47. Cartography and Geographic Informa- in Climate Change: Review- tion Science, Vol. 32, No. 3 (2005): ing Comprehension of an IPCC Navalpakkam, V., R. Kumar, L. 139–60. Scenario Graph.” Climatic Change, Li, and D. Sivakumar. “Attention Vol. 133, No. 2 (2015): 141–54. and Selection In Online Choice Mansoor, H., and L. Harrison. Tasks.” In J. Masthoff, B. Mobasher, “Data Visualisation Literacy and Meißner, Martin, Andres Musa- M. Desmarais, and R. Nkambo Visualisation Biases: Cases for lem, and Joel Huber. “Eye Track- (eds), User Modeling, Adaptation, and Merging Parallel Threads.” IEEE- ing Reveals Processes that Enable Personalization. Berlin Heidelberg: VIS 2017: Dealing With Cognitive Conjoint Choices to Become Springer (2012): 200–211. Biases In Visualisations (2017). Increasingly Efficient with Practice.” Nielsen, Jakob. “F-Shaped Pattern Mason, Jennifer Smith, Alexander Journal of Marketing Research, Vol. 53, No. 1 (2016): 1–17. for Reading Web Content.” Nielsen Klippel, Susanne Bleisch, Aidan Norman Group (2006). Visualization to Support Reason- Visual Design of Scatterplots.” ing and Decision Making.” Spatial IEEE Transactions on Visualization Cognition & Computation, Vol. 16, and Computer Graphics, Vol. 23, No. No. 2 (2016): 97–105. 6 (2017): 1588–99. 48 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

Nummenmaa, L., J. Hyönä, and Pang, A.T., C.M. Wittenbrink, Pugh, A.J., C.D. Wickens, N. M.G. Calvo. “Eye Movement and S.K. Lodha. “Approaches to Herdener, B.A. Clegg, and C.A.P. Assessment of Selective Attentional Uncertainty Visualization.” The Smith. “Effect of Visualization Capture by Emotional Pictures.” Visual Computer, Vol. 13 (1997): Training on Uncertain Spatial Emotion, Vol. 6, No. 2 (May 2006): 370–90. Trajectory Predictions.” Human 257–68. Factors, Vol. 60, No. 3 (May 2018): Parkhurst, D., K. Law, and E. 324–39. Niebur. “Modeling the Role of Salience in the Allocation of Overt O… Visual Attention.” Vision Research, R… Olivers, C., and M. Meeter. Vol. 42 (2002): 107–23. “On the Dissociation Between Rajsic, J., D.E. Wilson, and J. Pärnamets, Philip, Roger Johans- Compound and Present/Absent Pratt. “Confirmation Bias In Visual son, Kerstin Gidlöf, and Annika Tasks in Visual Search: Intertrial Search.” Journal of Experimental Wallin. “How Information Avail- Priming is Ambiguity Driven.” Psychology: Human Perception and ability Interacts with Visual Atten- Visual Cognition Vol. 13 (2006): Performance, Vol. 41, No. 5 (2015): tion During Judgment and Decision 1–28. 1353–64. Tasks.” Journal of Behavioral Decision Orquin, Jacob L., Nina Chrobot, Making, Vol. 29, No. 2–3 (2016): Rik, P., and M. Wedel. “Attention and Klaus G. Grunert. “Guiding 218–31. Capture and Transfer in Advertis- Decision Makers’ Eye Movements ing: Brand, Pictorial, and Text-Size Parsons, P. “Promoting Represen- with (Un)Predictable Object Loca- Effects.” Journal of Marketing, Vol. 68 tational Fluency for Cognitive Bias tions.” Journal of Behavioral Deci- (2004): 36–50. Mitigation In Information Visuali- sion Making, Vol. 31, No. 3 (2018): zation.” Presented at Dealing with 341–54. Riveiro, Maria, Tove Helldin, Cognitive Biases in Visualizations: a Göran Falkman, and Mikael Orquin, Jacob L., Sonja Perkovic, VIS Workshop (2017). Lebram. “Effects of Visualizing and Klaus G. Grunert. “Visual Uncertainty on Decision-Making Peschel, A.O., and J.L. Orquin. “A Biases In Decision Making.” Applied in a Target Identification Scenario.” Review of the Findings and Theories Economic Perspectives and Policy, Vol. Computers & Graphics, Vol. 41 on Surface Size Effects on Visual 40, No. 4 (2018): 523–37. (2014): 84–98. Attention.” Frontiers in Psychology, Osborne, M.J. An Introduction Vol. 4 (December, 2013): 902. Riveiro, Maria. “Visually Supported to Game Theory. Oxford: Oxford Reasoning Under Uncertain Condi- Pessoa, L., and L. G. Ungerleider. University Press (2003). tions: Effects of Domain Expertise “Visual Attention and Emotional on Air Traffic Risk Assessment.” Perception.” In G. Rees Itti and J.K. Spatial Cognition & Computation, Tsotsos (eds), Neurobiology of Atten- P… Vol. 16, No. 2 (2016): 133–53. tion. San Diego: Elsevier (2005). Padilla, L. M., S. H. Creem- Rodriguez, Lulu, and Daniela V. Popper, Karl R. “The Propensity Regehr, M. Hegarty, and J.K. Dimitrova. “The Levels of Visual Interpretation of Probability.” The Stefanucci. “Decision Making with Framing.” Journal of Visual Literacy, British Journal for the Philosophy Visualizations: A Cognitive Frame- Vol. 30, No. 1 (2016): 48–65. of Science, Vol. 10, No. 37 (1959): work Across Disciplines.” Cognitive 25–42. Rogers, Simon and Mark Research: Principles and Implications, Girolami. A First Course In Machine Vol. 3 (2018): 29. Poulton, E. Christopher. “Models Learning. CRC Press (2015). for Biases In Judging Sensory Palmer, J. “Set-Size Effects in Magnitude.” Psychological Bulletin, Visual Search: The Effect of Atten- Vol. 86, No. 4 (1979): 777–803. tion is Independent of the Stimulus for Simple Tasks.” Vision Research, Vol. 34, No. 1 (1993): 1703–21. 49

Ruginski, Ian T., Alexander P. Simola, J., J. Kuisma, A. Öörni, Stacey, Martin and Claudia Eckert. Boone, Lace M. Padilla, Le Liu, L. Uusitalo, and J. Hyönä. “The “Managing Uncertainty in Design Nahal Heydari, Heidi Kramer, Impact of Salient Advertisements Communication.” Presented at the S. Mary Hegarty, William B. On Reading and Attention On International Conference of Engi- Thompson, Donald H. House, and Web Pages.” Journal of Experimental neering Design, Glasgow (2001). Sarah H. Creem-Regehr. “Non- Psychology: Applied, Vol. 17 (2011): Expert Interpretations of Hurricane 174–90. Sütterlin, Bernadette, Thomas A. Forecast Uncertainty Visualizations.” Brunner, and Klaus Opwis. “Eye- Skeels, Meredith, Bongshin Lee, Tracking the Cancellation and Focus Spatial Cognition & Computation, Vol. 16, No. 2 (2016): 154–72. Greg Smith, and George Robert- Model for Preference Judgments.” son. “Revealing Uncertainty for Journal of Experimental Social Psychol- Information Visualization.” Proceed- ogy, Vol. 44, No. 3 (2008): 904–11. ings of the Working Conference on S… Advanced Visual Interfaces (2008): Sacha, D., H. Senaratne, B.C. 376–79. T… Kwon, G. Ellis, and D.A. Keim. Smith Mason, Jennifer, David “The Role of Uncertainty, Aware- Thaler, R., and C. Sunstein. Nudge: Retchless, and Alexander Klippel. ness, and Trust In Visual Analytics.” Improving Decisions about Health, “Domains of Uncertainty Visualiza- Yale: Yale IEEE Transactions on Visualization Wealth, and Happiness. tion Research: A Visual Summary University Press (2008). and Computer Graphics, Vol. 22, No. Approach.” Cartography and 1 (2016): 240–49. Geographic Information Science, Vol. Towal, R.B., M. Mormann, and C. Saltelli, A., K. Chan, and E. M. 44, No. 4 (2016): 296–309. Koch. “Simultaneous Modeling of Visual Saliency and Value Compu- Scott. Sensitivity Analysis. Chiches- Smith, J.Q. Bayesian Decision ter: John Wiley and Sons (2000). tation Improves Predictions of Analysis: Principles and Practice. Economic Choice.” Proceedings of the Saltelli, S., S. Tarantola, and F. Cambridge: Cambridge University National Association of Sciences, Vol. Campolongo. “Sensitivity Analysis Press (2010). 110, No. 40 (2013): E3858-67. As An Ingredient of Modelling,” Snowden, D. “Complex Acts of Tversky, A., and D. Kahneman. Statistical Science, Vol. 15, No. 4 Knowing – Paradox and Descrip- (2000): 377–95. “Availability: A Heuristic for Judg- tive Self- Awareness.” Journal of ing Frequency and Probability.” Schirillo, James A., and Stone, Eric Knowledge Management, Vol. 6, No. Cognitive Psychology, No. 5 (1973): R. “The Greater Ability of Graphi- 2 (2002): 100–111. 207–232. cal Versus Numerical Displays to Spiegelhalter, D., M. Pearson, and Increase Risk Avoidance Involves a I. Short. “Visualising Uncertainty Common Mechanism.” Risk Analy- V… about the Future.” Science, Vol. 333 sis, Vol. 25, No. 3 (2005): 555–566. (2011): 1393–1400. Valdez, A.C., M. Ziefle, and Schwartz, Steven H. Visual Percep- M. Sedlmair. “A Framework for Spiegelhalter, David. “Risk and tion: A Clinical Orientation, 5th Ed. Studying Biases in Visualisation Uncertainty Communication.” New York: Mcgraw-Hill Education Research.” In Geoffrey Ellis (ed.), Annual Review of Statistics and its (2017). Cognitive Biases in Visualizations Application, Vol. 4 (2017): 31–60. Cham: Springer (2018). Seipel, S., and N.J. Lim. “Color Spinks, J., and D. Mortimer. “Lost Map Design for Visualization in Valdez, A.C., M. Ziefle, and M. in the Crowd? Using Eye-Tracking Flood Risk Assessment.” Inter- Sedlmair. “Priming and Anchor- to Investigate the Effect of Complex- national Journal of Geographical ing Effects in Visualization.” IEEE ity on Attribute Non-Attendance in Information Science, Vol. 31, No. 11 Transactions on Visualization & Discrete Choice Experiments.” BMC (2017): 2286–2309. Computer Graphics, Vol. 24, No. 1, Medical Informatics and Decision (2018): 584–594. Making, Vol. 16 (2016): 14. 50 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

Visschers, V.H., R.M. Meertens, Wesslen, R., Santhanam, S., X… W.W. Passchier, and N.N. De Karduni, A., Cho, I., Shaikh, S. Vries. “Probability Information in and Dou, W. “Investigating Effects Xiong, C., L. van Weelden and Risk Communication: A Review of Visual Anchors on Decision- S. Franconeri. “The Curse of Knowledge in Visual Data Commu- of the Research Literature.” Risk Making about Misinformation.” nication,” in Analysis 29, No. 2 (2009): 267–87. Computer Graphics Forum, Vol. 38 IEEE Transactions on (2019): 161-171. Visualization and Computer Graphics Von Mises, Richard. Probability, (2019): 1. Statistics, and Truth. Courier Corpo- Wheaton, Kristan J. “The Revo- ration (1957). lution Begins on Page Five: The Changing Nature of Nies.” Inter- Z… Von Neumann, J. and O. Morgen- national Journal of Intelligence and stern. Theory of Games and Economic Counterintelligence, Vol. 25, No. 2 Zuk, T., and S. Carpendale. Behaviour, 2nd Ed. Princeton (2012): 330–49. “Visualization of Uncertainty and University Press (1947). Reasoning,” Proceedings of the 8th Wilson, T.D., C.E. Houston, international symposium on Smart K.M. Etling, and N. Brekke. “A Graphics (2007): 164–77. W… New Look at Anchoring Effects: Basic Anchoring and its Ante- Wall, E., L. Blaha, L. C. Paul, K. cedents.” Journal of Experimental Cook, and E. Alex. “Four Perspec- Psychology: General, Vol. 125, No. 4 tives on Human Bias in Visual (1996): 387. Analytics.” In Geoffrey Ellis (ed.), Cognitive Biases in Visualizations, Windschitl, Paul D., Andrew R Cham: Springer (2018). Smith, Aaron M. Scherer, and Jerry Suls. “Risk It? Direct and Wallsten, Thomas S., Samuel Collateral Impacts of Peers’ Verbal Fillenbaum, and James A. Cox. Expressions about Hazard Likeli- “Base Rate Effects on the Interpre- hoods.” Thinking & Reasoning, Vol. tations of Probability and Frequency 23, No. 3 (2017): 259–91. Expressions.” Journal of Memory and Language, Vol. 25, No. 5 (1986): Wolfe, J.M. “Guided Search 2.0. A 571–87. Revised Model of Visual Search.” Psychonomic Bulletin & Review, Vol. Waters, E.A., Weinstein, N.D., 1, No. 2 (1994): 202–38. Colditz, G.A., & Emmons, K.M. “Reducing Aversion to Side Effects Wright, W., D. Schroh, P. Proulx, in Preventive Medical Treatment A. Skaburskis, and B. Cort. “The Decisions.” Journal of Experimental Sandbox for Analysis,” Proceedings Psychology: Applied, Vol. 13, No. 1 of the SIGCHI Conference on Human (2007): 11. Factors in Computing Systems (2006): 1-10. Waters, E.A., Weinstein, N.D., Colditz, G.A., and Emmons, Wright, William, David Sheffield, K. “Formats for Improving Risk and Stephanie Santosa. “Argu- Communication in Medical Trade- ment Mapper: Countering Cogni- off Decisions.” Journal of Health tive Biases in Analysis with Critical Communication, Vol. 11, No. 2 (Visual) Thinking.” Presented at (2006): 167–182. 21st International Conference Infor- mation Visualisation IV (2017): Weick, K.E. Sensemaking in Organ- 250– 55. isations. Thousand Oaks, CA: Sage (1995). 51

11 RESOURCES ADDITIONAL IMAGES/ GENERAL BACKGROUND PLATFORMS FOR DATA ILLUSTRATIONS VISUALISATIONS How to create interactive data visualisations. from www.flaticon.com, made by • Microsoft authors/Freepic 3 • Nesta Sparks lecture by Cath https://powerbi.microsoft. Sleeman com/en-us/ authors/Freepic 12 https://www.youtube.com/ • R shiny authors/Eucalyp 13 watch?v=ctSl8tYEEDY https://shiny.rstudio.com/gallery/ authors/Eucalyp 24 • Visualising the uncertainty in • Tableau authors/Eucalyp 27 data by Nathan Yau (UCLA) https://www.tableau.com/ https://flowingdata. authors/Freepic 28 com/2018/01/08/visualizing- • D3 authors/Eucalyp 33 the-uncertainty-in-data/ https://github.com/d3/ d3/wiki/Gallery authors/monkik 35 • Visualising conflict data authors/Freepic 35 https://www.acleddata.com/ SOCIAL MEDIA ON VISUALISING DATA authors/wanicon 36 DESIGN authors/Eucalyp 37 • Financial Times @ftdata • Free images authors/Eucalyp 38 https://pixabay.com/ • NYT Graphics @nytgraphics https://unsplash.com/ • https://twitter.com/ https://thenounproject.com/ GuardianVisuals • Catalogue of examples • The Pudding @puddingviz http://www.rethinkingvis. or https://pudding.cool/ com/#all • Andy Kirk @visualisingdata or https://datavizcatalogue.com/ http://www.visualisingdata.com/ • Tutorials https://flowingdata.com/ category/tutorials/ 52 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

APPENDIX A UNCERTAINTY REPRESENTATIONS SIMULATIONS Some uncertainty formats attempt to present a range of possible future realities simultaneously. Another option is to present them sequen- tially. Many decision-makers and analysts will construct a base case, an upside, and a downside scenario. By examining these in turn, it is possible to develop a sense of where risks and opportunities lie. Uncertainty can also be represented in a simulation where random- ness is built into the model at appropriate points. Running the model again and again, and comparing the different outputs, can provide intuition for the fuzziness of predictions. Pros Cons Showing simulations provides Too much weight might be a sense of build-up and a link placed on individual outcomes with individual outcomes. and obscure the overall picture. Showing all data at once can be challenging for interpretation and lead to data overload.

OBSCURITY Blurriness is a powerful visual metaphor for displaying uncertainty (fog). Pros Cons The metaphor makes sense: How is fuzziness or results that are more uncertain obscurity perceived? are displayed with a blurry (not Are various levels actually sharp/clear) edge, which makes interpreted, and to what degree? it less visually prominent.

ERROR BARS Graphical representations of the variability of data; used to indi- cate the error or uncertainty in a reported measurement. Pros Con Lines or bars represent a range of Details in the data can get lost. values, so you can see that a mean “Within-the-bar bias”: viewers or median represents only part of judge points that fall within the an estimate. Especially useful when coloured bar as being more likely comparing multiple estimates. than points equidistant from Widely used, therefore the mean, but above the bar. easily understood. 53

DISTRIBUTIONS Show the spread of possible values with a histogram or a vari- ant of it. You might see something a median would never show. Pros Cons By showing the variation, Many people don’t understand a user can make a more distributions, so a careful educated judgement about the explanation needs to be accuracy and trustworthiness given in the annotations. a sample. It is oddly skewed? Sometimes variation is just Are there multiple peaks? Or noise, or the details might is it an expected bell curve? obscure the overall view, impression, or key point.

MULTIPLE OUTCOMES For projections and forecasts, it can be helpful to see various outcomes of what might happen. Pros Cons Uncertainty is displayed more The chart can become confusing explicitly; it is shown that if there is too much noise or there is not one set path, but too many possibilities. Like multiple possible paths which many visualisation formats, it is may diverge and/or converge. easily manipulated to promote a desired analysis or strategy.

DECISION TREES In this context, a decision tree refers to a visualisation format made up of nodes and branches. It is often laid out to be read left-to-right, or top-down; with a single root node as the starting point, branching out into possible futures. One common convention is to use squares to represent deci- sions and circles to represent chance events. A decision tree can be useful for understanding how a plethora of interconnected decisions, whose different outcomes can be assigned estimated probabilities, will play out under different future scenarios.

WORDS Not everything has to be visualized. Sometimes words better describe uncertainty. Some rules of thumb: avoid absolutes when describing numbers; treat estimates as such when you use them; account for the uncertainty in the numbers. 54 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

APPENDIX B CHARTS, GRAPHS, SYMBOLS 55

SYMBOLS, METAPHORS, VISUAL ANALOGIES 56 • VISUALISING UNCERTAINTY A SHORT INTRODUCTION

This work was supported byThe Alan Turing Institute through a project titled ‘The Communication of Uncertainty’, March 2019.