Communication of Statistical Uncertainty to Non-expert Audiences A thesis submitted for the degree of Master of Philosophy (Mathematics) by Jessie Roberts B.Sc. Biotechnology Innovation, Queensland University of Technology School of Mathematical Sciences Science and Engineering Faculty Queensland University of Technology Australia 2019 Contents Acknowledgementsv Declaration vii Abstract ix 1 Introduction1 1.1 Aims and objectives ................................. 6 1.2 The Australian Cancer Atlas ............................. 7 1.3 Research contributions ............................... 8 1.4 Thesis structure ................................... 8 2 Literature Review 11 2.1 Uncertainty ..................................... 12 2.2 Why is Uncertainty Information important to decision-makers? .......... 21 2.3 Communicating Statistical Uncertainty ....................... 24 2.4 Uncertainty communication ............................ 32 2.5 Uncertainty Communication design ........................ 34 2.6 Spatial epidemiology and disease mapping .................... 37 3 Research Activity 1.A: Grey literature review of internet published cancer maps. 43 3.1 Introduction ..................................... 43 3.2 Aim and Research Question ............................. 44 3.3 Methods ....................................... 45 3.4 Summary findings .................................. 46 3.5 Implications for the Australian Cancer Atlas .................... 53 3.6 Conclusion ...................................... 55 ii CONTENTS 4 Research Activity 1.B: User centred uncertainty communication design (Australian Cancer Atlas as a case study) 57 4.1 Introduction ..................................... 57 4.2 Methods ....................................... 58 4.3 Results ........................................ 65 4.4 Discussion & insights for the Australian Cancer Atlas. ............... 76 4.5 Conclusion ...................................... 80 5 Research Activity 2: User study - Uncertainty representation in an online game. 85 5.1 Introduction ..................................... 85 5.2 Aim ......................................... 87 5.3 Online Simulation - Impact of uncertainty communication methods on decision making ........................................ 87 5.4 Methods ....................................... 88 5.5 Results ........................................ 102 5.6 Discussion ...................................... 114 6 Discussion 119 6.1 Uncertainty communication design ......................... 120 6.2 Testing uncertainty representation methods .................... 125 6.3 Critique & limitations ................................ 127 6.4 Future Work ..................................... 129 6.5 Conclusion ...................................... 130 A Appendix: Literature Review 131 A.1 Uncertainty Representation in Mapping & GISciences ............... 131 B Appendix: Research Acitivity 1. A - Grey Literature Review 133 B.1 Search Protocol ................................... 133 Pre-Scoping ........................................ 133 Search Details ....................................... 134 B.2 Database of identified cancer atlases ........................ 139 iii CONTENTS C Research Activity 1.B - User-centred design for uncertainty communication 163 C.1 Project Partners Workshop ............................. 163 C.2 NEUVis Audience Profiles .............................. 179 D Appendix: Online Game 191 D.1 Untransformed Performance Data ......................... 191 D.2 Logit(PR) by game mode: LME output and diagnostic plots ............ 191 D.3 logit(PR) by risk profile - LME model output and diagnostic plots ......... 194 E Ethics Approvals 199 E.1 Focus Groups Recruitment Flyer and Consent Form ................ 199 E.2 Online Game .................................... 204 Bibliography 207 iv Acknowledgements I would like to acknowledge the many people that have contributed to the following thesis and more importantly the development of my academic research skills and competencies. I express my appreciation and thanks to my supervisors Distinguished Professor Kerrie Mengersen & Dr Kate Helmstedt for their guidance and support. Further to this I would like to specifically acknowledge the contribution and support of the following people: 1. Phil Gough - with whom I collaborated closely with to design, develop and imple- ment the online game (Chapter 4), as well as the design, data collection and data analysis of the Cancer Atlas focus groups (Chapter 3). 2. The Cancer Council QLD and specifically Peter Baade and Susanna Cramb - I am very grateful for their guidance and feedback along the journey and their collaboration for the work detailed in Chapter 3, including financial support to conduct the cancer mapping focus groups. 3. Nicholas Dendle - who through his VRES summer project of 2016/17 translated a paper version of the pirate game to a functioning online game (Chapter 4). 4. Matt Sutton - who implemented the optimisation solution needed for part of the analysis of the online game data (Chapter 4) 5. The QUT ACEMS and BRAG community for their many coffees, lunches, board games and generous feedback. 6. Shannon Ryan - for editing and proof-reading services. 7. Lawrence Jones for his support and encouragement. v Declaration I hereby declare that this thesis contains no material which has been accepted for the award of any other degree or diploma in any university or equivalent institution, and that, to the best of my knowledge and belief, this thesis contains no material previously published or written by another person, except where due reference is made in the text of the thesis. QUT Verified Signature Jessie Roberts June 2019 vii Abstract Introduction Statistical uncertainty is present in modelled and data derived information and plays an important role in how outputs of scientific and quantitative analyses are interpreted and used by decision makers, many of which are non-experts. Despite the importance of uncertainty information, there still remains impediments to successfully communicating this information to the non-expert audience. Standardisation of uncertainty representation methods as well as methods and guidelines for communication uncertainty in a way that is accessible to non-expert audiences are two recognised impediments to this challenge. I contribute to both of these in this thesis. A significant motivation of this thesis is to inform the design of an Australian national cancer atlas, which is used as a case study within this research. Aim and Objectives This research aims to contribute to a growing body of literature on uncertainty com- munication. It achieves this aim through four distinct objectives: 1 - provide a greater understanding of the current methods used to visualise uncertainty in disease mapping; 2 - investigate a tool for systematically identifying the sources of uncertainty within a multidisciplinary research project; 3 - explore a framework for including uncertainty within the design of scientific communication material; and 4 - and investigate the impact of three different uncertainty visualisation methods on decision making. Methods The first, second and third objectives are connected to the Australian Cancer Atlas, which is used as a case study throughout this thesis. Objective 1 is achieved through a grey literature review of currently available cancer maps. Objective 2 is achieved through ix CONTENTS exploring the use of an uncertainty taxonomy (developed by Morgan and Henrion (1990)), as a tool to diagnose uncertainty sources within the atlas. And objective 3 explores the NEUVis design framework (Gough, Wall, and Bednarz, 2014) for its effectiveness in uncertainty communication design, including the use of focus groups to understand how people interpret uncertainty within cancer mapping. The final and 4th objective is achieved through a quantitative user study which uses an online game to evaluate if players behaviour differs depending on if uncertainty is represented as: the upper and lower bounds of an interval, the semantic bounds of an interval, a point estimate ± error, or a point estimate (no uncertainty). Representation methods are evaluated in terms of the players ability to maximise their potential reward (performance), as well as how players distribute available resources (risk-averse behaviour). Findings/Results 1. The grey literature review identified 33 publicly available cancer maps of which 13 contained uncertainty information. Many different approaches were used to represent uncertainty, there were no common approaches. Uncertainty was more common in maps that contained interactivity. 2. Morgan and Henrion (1990)’s taxonomy of uncertainty, in its current form, did not prove to be a useful tool for diagnosing sources of uncertainty within the Australian Cancer Atlas. The technical detail of the taxonomy and lack of connection to the project or the target audiences hindered its applicability. 3. The extension of the NEUVis design framework to include uncertainty information worked well to navigate the uncertainty communication design process and sup- ported a cross disciplinary team to identify audiences and their needs. The design tool provided a platform for mapping audience needs to uncertainty information within the Australian Cancer Atlas. 4. The user study evaluating the effect of different uncertainty representation methods on user behaviour showed that, in terms of maximising potential reward (perfor- mance), players performed better in the online game when a point estimate