Literature Review of Visual Representation of the Results of Beneﬁt–Risk 68 10 † 69 11 Assessments of Medicinal Products 70 12 71 13 72 Q2 Christine E

Journal Code Article ID Dispatch: 13.09.15 CE: Jan April Same P D S 3 8 8 0 No. of Pages: 13 ME:

1 pharmacoepidemiology and drug safety 2015 60 2 61 Published online in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/pds.3880 3 62 4 REVIEW 63 5 64 6 65 7 66 8 67 9 Literature review of visual representation of the results of benefit–risk 68 10 † 69 11 assessments of medicinal products 70 12 71 13 72 Q2 Christine E. Hallgreen1*, Shahrul Mt-Isa1, Alfons Lieftucht2, Lawrence D. Phillips3, Diana Hughes4, 14 Susan Talbot5, Alex Asiimwe6, Gerald Downey5, Georgy Genov7, Richard Hermann8, Rebecca Noel9, 73 15 Ruth Peters1, Alain Micaleff10, Ioanna Tzoulaki1, Deborah Ashby1 and On behalf of PROTECT Benefit–Risk group 74 16 75 17 Q3 1School of Public Health, Imperial College London, London, UK 76 2 18 GlaxoSmithKline UK, Middlesex, UK 77 3 19 Department of Management, London School of Economics and Political Science, London, UK 78 4Pfizer, New York, NY, USA 20 5Amgen Limited, Uxbridge, UK 79 21 6Bayer Pharma AG, Berlin, Germany 80 22 7European Medicines Agency, London, UK 81 8 23 AstraZeneca LP, Wilmington, DE, USA 82 9 24 Eli Lilly and Company, Lilly Corporate Center, Indianapolis, IN, USA 83 10MerckSerono International SA, Geneva, Switzerland 25 84 26 85 27 ABSTRACT 86 28 Background The PROTECT Benefit–Risk group is dedicated to research in methods for continuous benefit–risk monitoring of medicines, 87 29 Q4 including the presentation of the results, with a particular emphasis on graphical methods.1 88 30 Methods A comprehensive review was performed to identify visuals used for medical risk and benefit–risk communication. The identified 89 31 visual displays were grouped into visual types, and each visual type was appraised based on five criteria: intended audience, intended mes- 90 32 sage, knowledge required to understand the visual, unintentional messages that may be derived from the visual and missing information that 91 may be needed to understand the visual. 33 Results Sixty-six examples of visual formats were identified from the literature and classified into 14 visual types. We found that there is 92 34 not one single visual format that is consistently superior to others for the communication of benefit–risk information. In addition, we found 93 35 that most of the drawbacks found in the visual formats could be considered general to visual communication, although some appear more 94 36 relevant to specific formats and should be considered when creating visuals for different audiences depending on the exact message to be 95 37 communicated. 96 Conclusion We have arrived at recommendations for the use of visual displays for benefit–risk communication. The recommendation re- 38 fers to the creation of visuals. We outline four criteria to determine audience–visual compatibility and consider these to be a key task in cre- 97 39 ating any visual. Next we propose specific visual formats of interest, to be explored further for their ability to address nine different types of 98 40 benefit–risk analysis information. Copyright © 2015 John Wiley & Sons, Ltd. 99 41 100 42 key words—review; benefit–risk; decision-making; medicines; communication; visual representation; pharmacoepidemiology 101 43 102 Received 8 February 2015; Revised 8 August 2015; Accepted 27 August 2015 44 103 45 104 46 105 47 BACKGROUND 106 48 107 49 This review was carried out as part the Innovative Med- 108 50 icine Initiative Pharmacoepidemiological Research on 109 51 *Correspondence to: C. E. Hallgreen, Imperial Clinical Trial Unit, School of Outcomes of Therapeutics by a European Consortium 110 52 Public Health, Imperial College London, St Mary’s Campus, Norfolk Place, (PROTECT) project work package 5 benefit–risk inte- 111 53 Paddington, London W2 1PG, UK. Email: [email protected] gration and representation (PROTECT BR group). PRO- 112 †The contents of this paper have previously been presented at various scientific 54 conferences, and a preliminary full report of this review has also been published TECT BR group is dedicated to research in methods for 113 55 online on the project website http://www.imi-protect.eu/. continuous benefit–risk (BR) monitoring of medicines, 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. 116 58 117 59 118 1 60 2 c. e. hallgreen et al. 2 61 3 including both the underpinning modelling and the pre- obtained in full and assessed against the inclusion 62 4 sentation of the results, with a particular emphasis on criteria described in the following. 63 5 graphical methods.1 This literature review of visual rep- Furthermore, we identified visual formats linked to 64 6 resentation and visual/graphical formats for BR commu- BR assessment methodologies from a recent review2 65 7 nication followed a review of methods for medicinal BR and highlight prominent visual formats associated with 66 8 assessment2; both were used to provide input to the each method. 67 9 PROTECT BR group case studies exploring the utility 68 10 of BR methods and visual formats for communication 69 11 in connection with BR assessment. Inclusion criteria and data extraction 70 12 When communicating about BR, it is important to 71 First screening included articles if title or abstract re- 13 be aware that we distinguish between efficacy and 72 ferred to communication, presenting information on 14 safety data on the one hand and benefit and risk on 73 risk, efficacy or safety data or BR and/or graphical 15 the other, requiring interpretation of the efficacy and 74 representation/information. In the following, full text 16 safety data for their clinical and therapeutic relevance. 75 screening articles were included that present or discuss 17 Benefits are defined as favourable effects and risks as 76 one or more visual formats to communicate benefitor 18 unfavourable effects, separate from the uncertainty of 77 risk information or information in connection to BR as- 19 experiencing the effects.3 78 sessment. From each relevant article, examples of the 20 Visual representation of BR information for 79 visual formats presented or discussed were extracted 21 decision-making of medicinal products is not 80 and also any relevant discussion and comment on the 22 completely exclusive to PROTECT BR group. We 81 strengths and weaknesses of the visual format. 23 gained insight from other resources as a starting point 82 24 for this review, including the recent BR Methodology 83 25 Project commissioned by the European Medicines 84 Appraisal criteria and strategy 26 Agency (EMA) and the US Food and Drug Adminis- 85 27 tration commissioned study to investigate the value The authors C. E. H., S. M. and A. L. identified distinct 86 28 of adding quantitative summaries of benefits and risks visual formats from the literature and grouped them 87 29 in standardised formats including visual displays and into visual types, such as, but not limited to, bar charts, 88 30 numerical formats.4 More general initiatives on visual pie charts or line graphs. The visual types were ap- 89 31 representation of data and communication by special praised at group level initially. We made some com- 90 32 interest groups and individuals are available on the In- ments on special cases or variation of the visual types 91 33 ternet.5–14 where necessary. 92 34 The aim of this review is to evaluate the usefulness The authors C. E. H., S. M. and A. L. appraised each 93 35 of different visual types for the representation and group of visual type against five criteria: intended audi- 94 36 communication of BR assessment information. ence, intended message, knowledge required to under- 95 37 stand the visuals, unintentional message that may be 96 38 associated with the visuals and any missing informa- 97 39 METHOD tion from the visuals that may be needed to understand 98 40 Literature search strategy them (for appraisal criteria description, see Table 7 or 99 41 Supporting Information). We then consolidated that 100 42 We conducted a comprehensive review of the litera- appraisals discussed and resolved any conflicts with 101 43 ture and searched for articles on BR communication the study team through emails, teleconferences and a fi- 102 44 and visual formats for risk communication, published nal face-to-face meeting. 103 45 after the year 2000 on Scopus up until February Because we were not able to formally test individ- 104 46 2014, PubMed, Web of Science and PsycINFO (for ual’s comprehension, we approached the appraisal 105 47 Q5 details of search terms, see Supporting Information). process theoretically based on two sets of principles 106 48 The reference list of articles that met our inclusion for visual display design: Wickens’ principles of dis- 107 49 criteria was screened for relevant publications. In addi- play design15 and Cleveland’s elementary perceptual 108 50 tion, we included related materials that were known to tasks.16,17 109 51 us at the time from the PROTECT BR group case stud- We framed our recommendation of visual formats to 110 52 ies, other initiatives, scientific conferences and be used in medical BR communication and representa- 111 53 websites on the Internet. tion through nine key BR questions. The key BR ques- 112 54 One reviewer (C. E. H.) examined titles and ab- tions were adapted from the work of the Communities 113 55 stracts of identified articles. Relevant articles were and Local Government (CLG) on visual representation 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2015 116 58 DOI: 10.1002/pds 117 59 118 1 Q1 60 visuals for benefit–risk representation 3 2 61 3 of data in the public sector, as appeared on the CLG extracted additional 33 examples of visual formats as- 62 4 DataViz website.7 sociated directly with BR methodologies identified in 63 5 a separate literature review of BR methodologies.2 In 64 6 Table 1, the visual types that are connected to specific T1 65 7 RESULTS BR assessment methodologies are presented. 66 8 The extracted visual formats were classified into 13 67 fi 9 Searches identi ed 4855 potentially relevant articles visual types, of which several include sub-groups of 68 fi 10 from the scienti c literature. Following title and ab- the variations with specific properties and ways of pre- 69 11 stract screening, more than 500 were examined in full sentation (Table 2). The classifications were based on T2 70 16,18–71 12 text and of those, 55 were deemed eligible. the well-accepted terminologies of the visual formats 71 fi 13 Q6 Fourteen additional sources for visuals were identi ed from our past experience. 72 4,7–14,72–76 14 including websites and reports. From the 55 In Table 3, we present a selection of visual formats T3 73 fi 15 identi ed articles and the 14 additional sources, we ex- that have more specific use in data representation and 74 16 tracted 66 examples of visual formats (for details of therefore may be more unfamiliar to lay readers. This 75 17 search, see Supporting Information). In addition, we is to give an idea of how an unfamiliar visual format 76 18 might look. The examples in Table 3 include specialist 77 19 visual formats aimed at general audiences, such as a 78 20 Table 1. Overview of visual representation connected to benefit–risk value tree, a risk scale and a pictogram. These also in- 79 21 methodologies clude three variations of bar charts communicating 80 22 Visual specific information in specific structures (waterfall 81 23 representation of Other visual representations of special plot, difference display and tornado diagram) and vi- 82 24 Approach results interest sual formats that communicate statistical information 83 25 PrOACT- ‘Effects’ table N/A 84 26 URL 85 27 PhRMA Table, dot/forest Tree diagram to represent model. 86 BRAT plot, bar graph Table 2. Visual types and visual type sub-groups 28 MCDA Bar graph, Table for evidence data, tree diagram 87 29 ‘difference display’ to represent model, line graph for Visual type Sub-group Reference 88 30 sensitivity analysis. 89 SMAA Bar graph, dot/ Table for evidence data, tree diagram Area graphs Area graph 16,23,38,46,58,67,76 31 forest plot and distribution plot to represent Distributions plots 14,67 90 32 model, line graph and scatter plot for Volume graphs 46,67 91 53 33 sensitivity analysis. Frontier graph 92 BRR Bar graph, dot/ Scatter plot or contour plot for Bar chart Simple bar chart 16,22,23,29,35,40,46,49,50,59,62,67 34 forest plot, line sensitivity analysis. Tornado diagram Grouped bar chart 16,24,32,40,46,50,54,59,64,66,67,69 93 – 35 graph may be suitable to simplify further the Divided/stacked 16,23,25,28 94 30,33,37,39,46,47,54,67–71,73 36 results. bar chart 95 NNT/ Dot/forest plot, line Contour plot for sensitivity analysis. Difference 72 37 NNH graph, scatter plot Tornado diagram may be suitable to diagram 96 38 simplify further the results. Tornado diagram 97 39 INHB Line graph, scatter Contour plot for sensitivity analysis. Waterfall plots 98 plot Box plot 28,34,46,50,56,67 40 Impact Dot/forest plot, line Contour plot for sensitivity analysis. Cartoons, symbols 21,46,50,52,55,64,67,74,77 99 41 numbers graph, scatter plot Tornado diagram may be suitable to and icons 100 16,28,40,46,67 42 simplify further the results. Dot chart Dot chart 101 QALY Bar graph, dot/ Line graph or scatter plot for Forest plot 20,68,73,78 – 43 forest plot sensitivity analysis. Line graphs Line graph 16,22 24,28,35,46,50,54,60,66,67 102 44 Q- Bar graph, dot/ Line graph or scatter plot for Frontier area 17 103 45 TWiST forest plot sensitivity analysis. graph 104 PSM N/A Network graph to represent model. Maps Statistical maps 16,22,31,67 46 MTC N/A Network graph to represent model. Sector maps 105 47 DCE Bar graph Line graph or scatter plot for (tree map) 106 sensitivity analysis. Pictograms 18,22,23,25,30–33,35,36,40–42,44,47, 48 50,52,54,57,59,60,65,67,70,71,77 107 49 PrOACT-URL, problem, objective, alternative, consequence, trade-off, un- Pie charts Pie charts 16,22,23,28,35,40,46,50,63,67,71 108 50 certainty, risk tolerance, linked decisions; PhRMA BRAT, Pharmaceutical Nightingale rose 50 109 Research and Manufacturers of America Benefit–Risk Action Team; 52 51 Speedometer 110 MCDA, multi-criteria decision analysis; SMAA, stochastic multi-criteria Risk scales/ladder 18,22,26,32,45,48,49,61,64,65 fi – 52 acceptability analysis; BRR, bene t risk ratio; INHB, incremental net Scatter plot 16,28,46,66,67 111 fi 53 health bene t; NNT/NNH, numbers needed to treat/numbers needed to Tables 23,30,38,40,46,65,67 112 harm; QALY, quality-adjusted life years; Q-TWiST, quality-adjusted time 30,38,47 54 Tree diagram Tree diagram 113 without symptoms and toxicity; PSM, probabilistic simulation method; Value tree 78 55 MTC, mixed treatment comparison; DCE, discrete choice experiment. 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2015 116 58 DOI: 10.1002/pds 117 59 118 1 60 4 c. e. hallgreen et al. 2 61 † 3 Table 3. Examples of selected visual formats, from the top left, the value tree, the risk scale, a pictogram, a waterfall plot, a difference display, a tornado 62 diagram, a box plot, a dot plot and in the bottom right corner a forest plot 4 63 5 64 6 65 7 66 8 67 9 68 10 69 11 70 12 71 13 72 14 73 15 74 16 75 17 76 18 77 19 78 20 79 21 80 22 81 23 82 24 83 25 84 26 85 27 86 28 87 29 88 30 89 31 90 32 91 33 92 34 93 35 94 36 95 37 96 38 97 39 98 40 99 41 100 Note: For example, the criteria could be favourable clinical events such as improvement in cholesterol levels or reduced disease progression, and/or 42 unfavourable outcomes such as increased risk of diarrhoea, psychiatric disorders or cardiovascular disorders. Alternatives A and B on these visuals may refer 101 43 Q7 to alternative treatments such as rimonabant and placebo for weight loss. 102 † fi 44 For more examples, see Supporting Information or www.imi-protect.eu/bene t-risk. 103 45 104 46 105 47 such as the box plot and forest plot. A dot plot is also We found that several visual formats could be used 106 48 shown, which is a part of a forest plot (middle part to in each of the pre-specified BR questions, depending 107 49 show the values of any point estimates). The forest on the exact message to be communicated and to 108 50 plot is sometimes referred to as a ‘range’ graph. whom different visuals could be relevant. Table 5 T5 109 51 To facilitate the recommendations from this review, gives an overview of which visual formats have the 110 52 we adapted the CLG DataViz’s common questions on potential to be used in connection to the common BR 111 53 visual data representation in the public sector to the questions. This is shown together with the information 112 54 BR scenario.7 The nine adapted BR questions are of level of expertise that is considered to be required to 113 55 T4 shown in Table 4. interpret the visual format and how the visual formats 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2015 116 58 DOI: 10.1002/pds 117 59 118 1 60 visuals for benefit–risk representation 5 2 61 3 Table 4. Adaptation of CLG DataViz’s data exploration question to BR questions (non-chronological), the CLG question is presented in the left column and 62 our adaptation to BR assessment in the right. The first CLG question was specified into two BR questions 4 63 5 Example (possible question to visually representing BR 64 6 CLG questions Adaptation to BR assessment results of a weight loss drug) 65 7 How to compare data? 1. How do I represent the (raw) magnitudes of quantitative How do I represent the percentages of people who achieved a 66 8 data such as the probabilities of events to describe data 10% weight loss and experienced diarrhoea for those taking 67 9 and to put them into context? rimonabant versus placebo? 68 2. How do I represent the magnitude of the final BR metrics How do I represent the overall BR score to easily compare 10 to allow easy comparison of the BR balance to be made? rimonabant and placebo for weight loss? 69 11 What is changing over time? 3. How do I represent how the magnitude of a measure is How do I represent the relationship between the score for 70 12 changing against a range of another measure such as time weight loss and preference weight? 71 or a range of preference values? 13 What is the distribution 4. How do I visualise the distributions or uncertainty of How do I represent the viability of data for weight loss 72 14 of an indicator variable? safety and efficacy data, preferences or a BR metric? observed in different trials? 73 15 What are the components 5. How do I represent the contributions from the different How do I show visually which of the adverse events in the 74 of an indicator variable? criteria (components) in a BR analysis to allow better BR analysis contributes most (or least) to the overall BR 16 perception of the key drivers? score? 75 17 What is the relationship 6. How do I represent the strength of the relationships How do I represent the relationship between people who 76 fi 18 between indicator variables? between bene t and risk metrics, for example, to visualise achieved 10% weight loss and those who experienced 77 many data points such as patient-level data or to visualise diarrhoea, to visually explore whether diarrhoea and weigh 19 the extent of correlation between criteria? loss occur together? 78 20 How significant are the 7. How do I represent the degree of statistical significance in How do I represent to which extent rimonabant is a more 79 21 differences? the difference between alternatives? preferred option compared to placebo, given the current 80 evidence and assumptions? 22 How to visualise qualitative 8. How do I represent and present qualitative data such as How do I represent the rates of depression and diarrhoea 81 23 data? text descriptions meaningfully and simply to support associated with rimonabant and placebo to indicate that 82 24 judgement without introducing extra cognitive burden? diarrhoea may be a more unfavourable side effect? 83 How to visualise categorical 9. How do I represent categorical data such as groups of How do I represent the percentages of people by the level of 25 data? patients, discrete events and categorical value function improvement in HDL cholesterol, for example, ‘improved’, 84 26 without distorting the data they are presenting? ‘did not change’, ‘got worse’? 85 27 CLG, Communities and Local Government; BR, benefit–risk; HDL, high-density lipoprotein. 86 28 87 29 are ranked according to Cleveland’s elementary per- Information or the PROTECT BR review of visual for- 88 30 ceptual tasks.16 For a more in-depth description of mats for the representation of BR assessment of med- 89 31 the appraisal of each visual type, see Supporting ication Stage 2.79 90 32 91 33 Table 5. Information on the level of expertise required for interpreting visual types, the rank of visuals according to Cleveland’s elementary perceptual tasks 92 and the visual types’ ability to communicate \messages connected to the central benefit–risk (BR) questions, as indicated by an ‘x’ 34 93 35 Network Tree Simple Grouped Dot Line 94 36 Cartoons maps Pictogram Table diagram bar chart bar chart chart graph 95 37 Level of expertise required E E E E E E E E E 96 38 Rank at elementary perceptual task (1–7) —— ———111197 39 Represent magnitudes of measures and xx x xxx98 ease comparison 40 Represent change in a magnitude of a xx 99 41 measure over the range of another measure 100 42 Represent the distribution or uncertainty 101 of a measure 43 Represent contributions from different x 102 44 criteria to BR 103 45 Represent the strength of relationships 104 between measures 46 Represent degree of statistical 105 47 significance 106 48 Represent qualitative data x x x x x 107 Represent categorical data x x x 49 108 50 E (easy)—no or very little expertise is required of the users to understand the visuals presented. It is accessible to patients, general public and suitable for mass Q8 109 Q9 media communication. The visual may be presented to user without much explanation. 51 M (intermediate)—some experience with straightforward BR assessment methodology may be required of the users in is not necessary to understand the the- 110 52 oretical foundation of the model. It is accessible to practicing physicians and patients’ representatives who need to understand and communicate BR to patients, Q10 111 53 caregivers or general public. The visuals may be presented to users without much explanation but would benefit from annotations or experts’ explanation. 112 D (difficult)—some experience and familiarity with complex BR assessment methodology, decision analysis and statistics may be required to fully exploit and 54 understand these visuals. It is accessible to BR experts in regulatory agencies, pharmaceutical companies and academia and is suitable for specialist publication 113 55 only for making high-level decisions. The visuals may also benefit from clear annotations and labelling to avoid presenting misleading information. Q11 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2015 116 58 DOI: 10.1002/pds 117 59 118 1 60 6 c. e. hallgreen et al. 2 61 3 Table 5. (Continued) 62 4 63 5 Risk 64 ladder/risk Area Pie Box Difference Forest Scatter Statistical 6 scale graph chart Speedometer plot display plot plot map 65 7 66 8 Level of expertise required E E E E M M M M M 67 Rank at elementary perceptual task (1–7) 1 5 6 6 1 1 1 2 2 9 Represent magnitudes of measures and xxxxxxxx68 10 ease comparison 69 11 Represent change in a magnitude of a xx 70 measure over the range of another measure 12 Represent the distribution or uncertainty xxx 71 13 of a measure 72 14 Represent contributions from different x 73 criteria to BR 15 Represent the strength of relationships x 74 16 between measures 75 17 Represent degree of statistical xx 76 significance 18 Represent qualitative data 77 19 Represent categorical data x x x 78 20 79 21 80 Table 5. (Continued) 22 81 23 82 Stacked Distribution Waterfall Tornado Frontier Sector 24 bar chart plot plot diagram graph map 83 25 84 Level of expertise required M M D D D D 26 Rank at elementary perceptual task (1–7) 3 5 3 3 3 5 85 27 Represent magnitudes of measures and xxx86 28 ease comparison 87 Represent change in a magnitude of a xxx 29 measure over the range of another measure 88 30 Represent the distribution or uncertainty xxx89 31 of a measure 90 Represent contributions from different xx 32 criteria to BR 91 33 Represent the strength of relationships x 92 34 between measures 93 Represent degree of statistical x 35 significance 94 36 Represent qualitative data 95 37 Represent categorical data x x 96 38 97 39 Effective visual representations of BR information assessment information. We demonstrate that none of 98 40 are not only limited to pictorial representations but the visual types can be used for all purposes (Table 5), 99 41 also include other components of the visual represen- which concurs with the existing finding that there is 100 42 tation. This may result in the inclusion of words that not one single visual type that is consistently superior 101 43 are prone to misinterpretation or misleading. There to others for the communication of BR information to 102 44 is also a risk of potentially presenting insufficient in- various stakeholders.4 This is partly due to the differ- 103 45 T6 formation. Table 6 gives an overview of some issues ent types of information to be presented and also partly 104 46 to be considered with visual representation of BR as- due to the differences in an individual’s perception, 105 47 sessments. We also hypothesised (but have not tested) understanding and preference of visuals. 106 48 that certain visual types may easily be associated with Firstly, we want to point out the importance of con- 107 49 the specific issues, based on the visual display exam- sidering the intended audience for the visual communi- 108 50 ples extracted from the literature. cation. Some visuals such as the simpler bar charts may 109 51 be used for a variety of groups from general public to 110 52 DISCUSSION trained experts, whilst others like the pictogram or the 111 53 waterfall plot may have more targeted users. As for 112 54 This review set out to appraise different visual types the intended audience, the intended message is a main 113 55 for the representation and communication of BR factor in creating visuals. Although different messages 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2015 116 58 DOI: 10.1002/pds 117 59 118 1 60 visuals for benefit–risk representation 7 2 61 3 Table 6. Overview of potential risk of misinterpretation related to visual communication 62 4 Examples of visual types related to 63 5 Issue Description the issue 64 6 65 Verbal labels 7 Gradable adjectives Adjectives are easy and natural to be used in the presentation of BR assessment and may Risk scales 66 8 better capture a person’s emotions and intuitions25,49 and can have the ability to put a 67 ‘ ’ ‘ ’ 9 treatment into context. Examples of gradable adjectives are high risk and very high risk . 68 Risk of misinterpretation is especially high if verbal labels are not accompanied by numerical 10 representation.60 69 11 Technical terms This could be medical or statistical terms that are not understood by an untrained audience. Any visual type 70 fi 12 Examples of technical terms are con dence intervals, densities, utilities and cardiovascular 71 events. 13 Numerical It is important to be consistent in the use of numerical format when making comparison.49 Any visual type 72 14 representation There is a general consensus that relative frequencies are superior to percentages or 73 25,33,38,49 15 probabilities for a transparent communication of risk information. 74 Relative risk (RR) A relative risk is a ratio of two incidence rates. RR may lead people to systematically Forest plot 16 underestimate or overestimate treatment effects, depending on the effect size.26,33,38 75 17 RR does not, on its own, provide all the necessary information to the audience because it is 76 38 18 relative to a measurement that might be unknown to the audience. 77 Denominator An example of denominator neglect is the arbitrary and inconsistent use of denominators Pictograms 19 neglect when describing frequencies in different situations. For example, a frequency of an Numerical representation as 78 20 unfavourable effect of 1 in 5 (1:5) may be perceived as safer than a frequency of an frequencies 79 unfavourable effect of 20 in a 100 (20:100), although they are exactly the same.25,50,54,60 21 30 80 Logarithmic scales When visuals presenting logarithmic scales are not clearly labelled, they can cause users to Risk scales (which are often used 22 perceive consecutive risks as being additive rather than multiplicative, for example, reducing for an untrained audience) Q1281 23 a probability with 1 in 10 to 1 in 100 may be perceived as being the same as reducing a Forest plot showing relative risks or 82 24 probability with 1 in 100 to 1 in 1000. odds ratios 83 Missing part-to- Emphasises the foreground information without sufficient background could lead to a Bar charts 25 whole information misperception of the difference in the measures such as the probabilities between two Pictograms 84 26 events.18 Dot charts 85 27 Area/volume graphs 86 Abundance of A long list of risks for a drug in comparison with short list of benefits, for example, may be Tables 28 events perceived as an unfavourable benefit–risk balance without taking into account the actual Tree diagrams 87 29 quantitative data. 88 30 The right column states which visual formats are specifically related to a problem; this, however, does not mean that the problem should not be considered in 89 31 connection with other visual formats. 90 32 91 33 92 34 93 35 can be communicated by a variety of visual types, the Table 7 outlines four criteria for determining audience– T794 36 level of detail that needs to be communicated can influ- visual compatibility. 95 37 ence the choice of visual type, for example, the stacked In addition to determining audience–visual compat- 96 38 bar chart can be used to communicate how each of the ibility when creating visuals for communication in BR 97 39 criteria contribute to the overall BR balance, but if the assessment, we recommend applying Wickens’ princi- 98 40 contributions from several criteria are similar, it can ples of display design15 and the GlaxoSmithKline 99 41 be difficult to discriminate their individual contribu- graphics principles.80 Although these principles were 100 42 tions; a grouped bar chart might be a better choice. not developed specifically for the visual representation 101 43 Whether the chosen visual representation causes an un- of BR assessments in medicine, they do offer some ad- 102 44 intended message, or gives an unjust impression of cer- vice on the design of general visual representation, 103 45 tainty to the presented BR balance, should also be which are easily adaptable for our purpose. 104 46 considered. Furthermore, one should consider what Despite our focus on static visual representations, 105 47 knowledge is required to interpret the visual, and this the principles can also be applied to interactive/ 106 48 is often related to technical skills such as understanding dynamic visual representation. We acknowledge that 107 49 the logarithmic scale or medical terms. In addition, it is interactive/dynamic visuals may be of great value be- 108 50 also important to ensure that the visual includes all nec- cause they enable active participation of the audience 109 51 essary information to correctly interpret and under- that can increase attention and perception. Through 110 52 stand the visual. This could be as simple as making increasing,81 their use to display analysis results are 111 53 sure that the axes have the right labelling, or more still uncommon and are not substitutes for poorly 112 54 extensively as verbally describing/explaining the vi- designed static version of the visual display; the use 113 55 sual to the users to enhance the important information. and choice of colours may also effect perception 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2015 116 58 DOI: 10.1002/pds 117 59 118 1 60 8 c. e. hallgreen et al. 2 61 3 Table 7. Criteria to determine audience–visual compatibility prior to generating visuals 62 4 1. Intended audience. Specify the intended main audience/user and verify whether the final visual is still suitable for the initially intended group 63 5 of audience. 64 • 6 The main user(s) of the visual could be the general public/media, patient, prescriber, regulator or expert (medical, statistical and decision analyst). 65 If the visual is intended for more than one group of users, consider criteria 2–4 in the following for each group. 7 2. Message. Specify the main message of the visual and verify that the final visual still communicates the intended message clearly and that it is 66 8 free from unintentionally misleading or confusing information. 67 • 9 The main intended message could be information about the BR balance, input data, probability of an event, uncertainty related to input data or BR, 68 sensitivity of the benefit risk analysis, integrated BR balance, the BR process and so on. 10 •Unintentional misleading/confusing message could be due to the visual display design itself, or the lack of user’s knowledge that was not anticipated 69 11 in the design stage. Unintentional messages could be incoherent reflection of the original data, any misleading assurance of the BR balance, the amount 70 fi 12 of certainty/uncertainty of the BR balance are not presented suf ciently and so on. 71 3. Knowledge required. Specify the expected level of knowledge required to understand and to extract information from the visual. Verify that the final 13 visual is at an appropriate level for the intended group of audience. 72 14 •Knowledge requirement could be any technical skills (e.g. understanding of logarithmic scale and concepts used in descriptive statistics), any medical 73 15 knowledge (e.g. severity of condition, reversible effects/events, passing events and conditional relationships) and any background information about 74 the measures in the visual (e.g. population affected). Ensure that the required knowledge is easily accessible by the users. 16 4. Message not communicated. For all of the aforementioned, verify in the final visual that there are sufficient representations of the information for 75 17 the intended message to be communicated and understood clearly. 76 18 BR, benefit–risk. 77 19 78 20 79 21 and/or comprehension.79 We did not explore colours number of benefit and risk criteria can be perceived 80 22 in greater detail but recommend that users refer as an unbalanced BR profile without taking into ac- 81 23 to other research on colours when developing their count the actual quantitative data. 82 24 visual representations, for example, Color Brewer The risk ladder/scale can facilitate comparison and 83 25 (http://colorbrewer2.org) and J’FLY (http://jfly.iam. judgement; it often provides information on other risks 84 26 u-tokyo.ac.jp/color). for comparison to particularly assist the general public 85 27 We set out to propose visual types that could be of and patients as well as regulators in perceiving the 86 28 interest when presenting information related to nine magnitude of risks under discussion.50 For the risk 87 29 central BR questions (Table 4). Here, particularly scale, it is important to make sure that, if used, a loga- 88 30 Cleveland’s elementary perceptual tasks have been rithmic scale is clearly marked and understood by the 89 31 our focus.16,17 audience. Risk ladders or scales are designed to ease 90 32 A table can serve as a useful BR communication the communication of risks by anchoring the risks 91 33 tool because of its simple structure, flexibility and against commonly understood scenarios; however, it 92 34 the ease with which it can be adapted. Readability is important to make sure the anchors are understood 93 35 can be enhanced through the use of colour coding to and relevant to the audience. 94 36 represent grouping and relationships, as carried out The pictogram has generally proven to be quickly 95 37 in the Benefit–Risk Action Team (BRAT) frame- and better comprehended than other graphical formats 96 38 work.82 For tables, it is important to be aware that they when used to communicating individual statis- 97 39 can be thought of as containing a list, with a long list tics35,40,65 and can help to prevent patients from being 98 40 of risks perceived as having unfavourable BR balance biased by other factors.33 Therefore, the pictogram is 99 41 without taking into account the actual quantitative data of interest as an easily comprehended visual format 100 42 of their severity and incidence. The table is suitable for when communicating to the general public about the 101 43 many audiences from general public to experts. It relative frequencies of favourable effects and the inci- 102 44 communicates well the criteria considered in a BR as- dence of unfavourable effects. 103 45 sessment, their hierarchical structure and the statistical The bar chart includes several special cases, where 104 46 summaries associated with the favourable and the simple bar chart, stacked/divided bar chart and 105 47 unfavourable effects. The two main examples are the grouped bar chart are the most familiar, the bar chart 106 48 key BR table from BRAT82 and the effect table from is usually easy to read and interpret. For the stacked 107 49 PrOACT-URL.72 bar chart, one should be aware that it can be more dif- 108 50 Tree diagrams can communicate qualitative infor- ficult to rank order the categories than for the grouped 109 51 mation, such as which benefits and risks are pivotal bar chart. Bar charts often best represent categorical 110 52 to the BR balance, and can represent the hierarchy of data; they only have one value axis, whilst the other 111 53 associations among the criteria, as seen with the axis represents discrete categories such as groups. 112 54 BRAT.82 Like the table, it is important to be aware The simpler bar charts (simple bar chart, stacked bar Q13113 55 of the potential downside that an imbalance in the chart and the grouped bar chart) could be suitable 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2015 116 58 DOI: 10.1002/pds 117 59 118 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 oyih 05Jh ie os Ltd. Sons, & Wiley John 2015 © Copyright Table 8. Overview of visual representations recommended for further consideration Ease of Key BR question Visual format interpretation Possible misinterpretations

To represent the comparison of the magnitudes of the final BR metrics, for example, Simple bar graph Easy Effects can be emphasised by not showing part-to-whole scores or expected utilities between alternatives. information Stacked bar graph Easy Effects can be emphasised by not showing part-to-whole information Difficult to compare the categories across options Risk of misinterpretation by reading of the values corresponding to height of the bar section instead of the actual length To represent the comparison of the magnitudes of quantitative data, for example, Table—‘effects table’, Easy Incorrectly perceived as list, could give a false impression on BR probabilities of events. ‘source table’ balance Hierarchies may be perceived when reading a table because information appears by lines and could be read as such Risk scales/ladder Easy Risk of unclear rational for risks chosen as anchors for comparison —‘community risk Inaccurate and inconsistent interpretation of logarithmic scales bene for visuals scale’ Pictogram/pictograph/ Easy Risk of misinterpretation when different total number of icons icon array (numerator) are used in a series of pictograms The absolute number of icons can influence the perceived likelihood The pictograms do not represent the entire population Partial displayed figures tend to be rounded up in interpretation

To represent how the magnitude of a measure is changing against a range of another Line graph Easy Difﬁcult to estimate the vertical difference between two curves on ﬁ t

measure, for example, time, preference values. the same graph –

Misleading when they are used to represent ranks, nominal or representation risk ordinal measures Dot chart/forest plot Easy Waterfall plot (bar Difficult Risk of misinterpretation because a bar begins where the above bar chart) ends. To represent the distributions or uncertainty of efficacy or safety data or a BR metric. Distribution plot (area Difficult Difficult to judge the size of a difference between two areas graph) Forest plot Intermediate Confidence intervals around the point estimates can cause attention to the criteria with larger confidence interval hraopdmooyadDu Safety Drug and Pharmacoepidemiology Tornado diagram Difficult Box plot Intermediate Require statistical knowledge To represent the contributions of the different criteria (categories) in the BR analysis. Stacked bar graph Intermediate Effects can be emphasised by not showing part-to-whole information Difficult to compare the categories across options Risk of misinterpretation by reading of the values corresponding to height of the bar section instead of the actual length Difference display (bar Intermediate Small differences can disappear compared with larger graph) To represent the contributions of the different criteria (categories) in the BR analysis. Grouped bar graph Intermediate Effects can be emphasised by not showing part-to-whole (continued) information fi

O:10.1002/pds DOI: To represent the strength of relationships between bene t and risk metrics, for Scatter plot Intermediate Overlapping points cannot be distinguished example, for many data points like patient-level data or correlated criteria. Could draw attention to relationship in data that are not clinical relevant Nominal scales can be misunderstood to have same interpretation as the continuous scale

2015 , Tornado diagram Difﬁcult

(Continues) 9 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 1 60 10 c. e. hallgreen et al. 2 61 3 for a large variety of audiences such as the general 62 tor nBR 4 fi public through the media, patients, physicians, regula- 63 5 tors and other experts for communication about the fi- 64 6 nal BR metric to visualise the contributions of the 65 7 different criteria (components) in the BR analysis 66 8 and categorical data. Special cases of the bar chart in- 67 9 clude the tornado diagram, the difference display and 68 10 the waterfall diagram (Table 3). The special cases 69 11 have many of the same features as the simpler bar 70 dence interval 12 fi charts but will generally require more explanation to 71 13 be clearly understood. The difference display is rele- 72 14 vant to represent, for a trained audience, the contribu- 73 15 tions of the different criteria in the BR analysis and 74 16 was also recommended as a visual for displaying re- 75 17 sults of BR analysis in the recent report from EMA 76 18 72 Q14 77 dence intervals around the point estimates can cause attention BR Methodology Project. The tornado diagram is cult to judge the size of a difference between two areas fi 19 fi proposed for the communication of uncertainty of 78 information information balance Hierarchies may be perceivedinformation when appears reading by a lines table and because could be readrisk as criteria such to represent BR balance Imprecise information 20 to the criteria with larger con the BR metric and visualisation of the relationships 79 21 between benefit and risk metrics and correlated 80 22 criteria, again for a trained audience. Finally, the wa- 81 23 terfall plot can be used to communicate about the 82 Ease of 24 fi 83 interpretationIntermediate Dif Possible misinterpretations level of contribution each bene t and each risk pro- 25 vides to the overall BR balance. 84 26 The dot plot has similar features compared with the 85 27 simple bar chart and offers a very high data–ink ra- 86 28 tio.67 The forest plot is a special case of dot plot, 87 29 which contains more statistical underpinnings, and 88 30 can be used to represent summary measures such as 89 31 mean risk difference and risk ratios as well as their as- 90 graph) Simple bar graphGrouped bar graphDot Easy plot Easy Effects can be emphasised by not Effects showing can part-to-whole be emphasised by Easy not showing part-to-whole Risk of falsely perceiving relationship or variability in data 32 Forest plotTree Intermediate diagram Con Cartoons/icons Easy Easy Risk of misinterpreting the value Misunderstanding tree due if to overweight cultural of differences bene sociated uncertainty via confidence intervals, as in 91 33 BRAT,82 and is most suitable to a specialist audience 92 34 such as statisticians, physicians, regulators and other 93 35 experts. 94 36 Line graphs communicate the relationship of 95 37 changes in one measure such as frequency or probabil- 96 38 ity of an event over a range of values in another effect 97 39 —time, dose levels and so on. A line graph is a very 98 40 common type of visual display many people come 99 41 across in various media such as in the newspaper or 100 42 on television (e.g. stock values line graph and trends 101 43 in historical weather or the forecast). Although general 102 44 awareness may not be the best measure of broad appli- 103 45 cability of visual understanding in BR assessment, 104 46 105 cance in the difference between alternatives. Distribution plot (area such exposure to line graphs may make them suitable 47 fi for communication to most people. 106 48 Scatter plots allow users to perceive the strength of 107 49 relationship between any two uncertain quantities and 108 50 can also reflect the variability in the data. Scatter plots 109 51 are fairly intuitive and do not need any specialised 110 risk. 52 – 111 ts knowledge in order to understand them. 53 fi Box plots (also known as the box and whiskers dia- 112 54 gram) are used to convey statistical information by 113 To represent and present qualitative data, for example, text descriptions. Table Easy Incorrectly perceived as list, could give a false impression o To represent the statistical signi To represent categorical data,value for function. example, groups, discrete events and categorical BR, bene 55 Table 8. (Continued) Key BR questionpresenting Visual format a summary of the dataset in terms of their 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2015 116 58 DOI: 10.1002/pds 117 59 118 1 60 visuals for benefit–risk representation 11 2 61 3 position in the data. The box plot can be used to repre- CONCLUSIONS 62 4 sent the distributions of uncertainty for efficacy and 63 5 safety data. Because of the technical constructions of Our main recommendation for the creation of visuals 64 6 box plot, they may be limited to experts or trained au- for BR assessments is to determine the compatibility 65 7 dience who have some understanding on statistical between a visual and its target audience. This is carried 66 8 summary measures such as medians, means, quartiles out by considering the intended audience for the visual, 67 9 and outliers. the main message the visual should communicate and 68 10 The area graphs and volume charts suffer from peo- the knowledge required to understand and to extract in- 69 11 ple’s ability to perceive area and volume differently.16 formation from the visual. We specifically suggest 70 12 In the case of volume chart, it becomes worse because evaluating whether any message may be missed or 71 13 of our limitation to accurately judge the size of three- any unintended message could be drawn from a visual. 72 14 dimensional objects. The only area graph we find of in- Secondly, we aim to help BR analysis experts and 73 15 terest is the distribution plot, which may look like a line decision-makers to navigate through the many visual 74 16 graph, but the information is actually being communi- types using a series of common BR questions. An 75 17 cated by the area under the curve. The distribution plot overview of the key BR questions and the visuals pro- 76 18 is a well-known way of representing data distributions posed is provided in Table 8, together with the ease of T8 77 19 for experts or a trained audience who have some under- interpretation for each visual format and possible mis- 78 20 Q15 standing on statistics. It can be used to represent the interpretation to take in consideration. 79 21 distribution or uncertainty of a measure, showing the 80 22 patient-level distribution of data, and to communicate DISCLAIMER 81 23 about the statistical significance in the difference be- 82 24 tween alternatives to an expert audience. The processes described and conclusions drawn from 83 25 Cartoons/icons or pictograms can be used to indi- the work presented herein relate solely to the testing 84 26 cate if something is a positive or a negative outcome, of methodologies and representations for the evalua- 85 fi 27 inform about specific patient groups (e.g. men or tion of bene t and risk of medicines. This report nei- 86 28 women) and indicate the direction of a change. Picto- ther replaces nor is intended to replace or comment 87 29 grams or cartoons have the potential to cross the on any regulatory decisions made by national regula- 88 30 language barrier and would be particularly useful for tory agencies or the EMA. 89 31 people who are sighted or partially sighted but are The views expressed in this article are the personal 90 32 unable to read. It is important that pictograms, car- views of the author(s) and may not be understood or 91 fl 33 toons, icons or symbols used in BR visual representa- quoted as being made on behalf of or re ecting the po- 92 34 tions are recognisable images that the intended users sition of the EMA or one of its committees or working 93 35 would have had experience seeing in the past to sup- parties. 94 36 port their understanding.15 Cultural differences may 95 37 be the most prohibitive when it comes to cartoons, CONFLICT OF INTEREST 96 38 icons and symbols because the images may not be 97 39 common or could even be offending to some cultures. The PROTECT Consortium has the right of 98 40 The pie chart is an often a widely used visual; how- commenting, but authors retain the right of accepting 99 comments and/or suggestions. The Consortium 41 ever, the reading of angles means that it scores fairly fi 100 42 low on Cleveland’s elementary perceptual task scale, reviewed and approved the nal paper. 101 43 and it is difficult to rank order categories and compare 102 16 44 between pie charts. KEY POINTS 103 45 Statistical maps in the form of geographical maps • There is not one single visual type that is consis- 104 46 may not be very relevant for use in the BR assessment. 105 ‘ ’ tently superior to others for the communication 47 A different type of statistical map is the sector map ; of BR information to various stakeholders. 106 48 it is used as a type of graphical method to detect and • Creating visuals for communication in BR as- 107 49 display differences in adverse event rates between sessments is too important to consider the com- 108 50 treatment groups. The sector map provides a high- 109 patibility between a visual and its target Q1 51 level overview of the situation and makes use of col- audience. 110 52 our to encode information that can then be drilled • We propose a number of visual types that could 111 53 down to the required level of details. However, this be of interest when presenting information re- 112 54 type of representation may be affected by the limita- lated to nine central BR questions. 113 55 tions of area judgement and colour intensity. 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2015 116 58 DOI: 10.1002/pds 117 59 118 1 60 12 c. e. hallgreen et al. 2 61 3 ETHICS STATEMENT 10. Few S. Perceptual Edge ▪▪ [cited 2013]. Available from: http://www. 62 perceptualedge.com. Q24 4 11. Tufte E. The work of Edward Tufte and Graphics Press ▪▪ [cited 2013]. Available 63 5 The authors state that no ethical approval was needed. from: http://www.edwardtufte.com/tufte. Q25 64 12. McCandless D. Information is Beautiful ▪▪ [cited 2013]. Available from: http:// 6 www.informationisbeautiful.net. Q26 65 7 ACKNOWLEDGEMENTS 13. CIRS. Centre for Innovation in Regulatory Science website ▪▪ [cited 2013]. 66 Available from: http://www.cirsci.org. Q27 8 Q17 All persons mentioned in this acknowledgement have 14. Rosling H. Gapminder ▪▪ [cited 2013]. Available from: (http://www.gapminder.org). Q28 Q2967 9 participated in the PROTECT BR group; participants 15. Wickens CD, Lee J, Liu YD, Cordon-Becker S. An introduction to human fac- 68 tors engineering. 2004. Q30 10 have given written consent to be acknowledged in this 16. Cleveland WS, Mcgill R. Graphical perception—theory, experimentation, and 69 11 paper. application to the development of graphical methods. J Am Stat Assoc 1984; 70 79(387): 531–554 PubMed PMID: WOS:A1984TK97500006. English. 12 The research leading to these results was conducted as 17. Carswell CM. Choosing specifiers—an evaluation of the basic tasks model of 71 13 part of the PROTECT Consortium (www.imi-protect. graphical perception. Hum Factors 1992; 34(5): 535–554. PubMed PMID: 72 WOS:A1992KA54500003. English. 14 eu), which is a public–private partnership coordinated 18. Ancker JS, Senathirajah Y, Kukafka R, Starren JB. Design features of graphs in 73 15 by the EMA. The PROTECT project has received sup- health risk communication: a systematic review. J Am Med Inform Assoc 2006; 74 13(6): 608–618. 16 port from the Innovative Medicines Initiative Joint Un- 19. Armstrong K, FitzGerald G, Schwartz JS, Ubel PA. Using survival curve com- 75 17 dertaking (www.imi.europa.eu) under grant agreement parisons to inform patient decision making can a practice exercise improve un- 76 derstanding? J Gen Intern Med 2001; 16(7): 482–485. 18 no. 115004, resources of which are composed of finan- 20. Armstrong K. Methods in comparative effectiveness research. J Clin Oncol 77 19 cial contribution from the European Union’s Seventh 2012; 30(34): 4208–4214. 78 21. Bennett DL, Dharia CV, Ferguson KJ, Okon AE. Patient–physician communica- 20 Framework Programme (FP7/2007-2013) and EFPIA tion: informed consent for imaging-guided spinal injections. JACR J Am College 79 21 companies’ in kind contribution. Radiol 2009; 6(1): 38–44. 80 22. Bostrom A, Anselin L, Farris J. Visualizing seismic risk and uncertainty: a re- 22 view of related research. Ann N Y Acad Sci 2008; 1128:29–40. 81 23 AUTHOR CONTRIBUTIONS 23. Brust-Renck PG, Royer CE, Reyna VF. Communicating Numerical Risk: Hu- 82 man Factors that Aid Understanding in Health Care. ▪▪: ▪▪, 2013; 235–276. Q31 24 24. Brynne L, Bresell A, Sjögren N. Effective visualization of integrated knowledge 83 25 Q18 Billy Amzal, Simon Ashworth, Johan Bring, Torbjorn and data to enable informed decisions in drug development and translational 84 medicine. J Transl Med 2013; 11(1). Q32 26 Callreus, Edmond Chan, Morten Colding-Jorgensen, 25. Burkell J. What are the chances? Evaluating risk and benefit information in con- 85 27 Christoph Dierig, Susan Duke, Adam Elmachtoub, sumer health materials. J Med Libr Assoc 2004; 92(2): 200–208. 86 26. Burkiewicz JS, Vesta KS, Hume AL. Improving effectiveness in communicating 28 David Gelb, Ian Hirsch, Steve Hobbiger, Kimberley risk to patients. Consultant Pharmacist 2008; 23(1): 37–43. 87 29 Hockley, Juhaeri Juhaeri, Silvia Kuhls, Davide 27. Cleveland WS. The Elements of Graphing Data. Hobart Press: ▪▪, 1994. Q33 88 fi 28. Cleveland WS, McGill R. Graphical perception and graphical methods for ana- 30 Luciani, So a Mahmud, Marilyn Metcalf, Jeremiah lyzing scientific data. Science 1985; 229(4716): 828–833. 89 31 Mwangi, Thai Son Tong Nguyen, Richard Nixon, 29. de Bruin WB, Stone ER, Gibson JM, Fischbeck PS, Shoraka MB. The effect 90 of communication design and recipients’ numeracy on responses to UXO risk. 32 John Pears, George Quartey, Sinan B. Sarac, Isabelle J Risk Res 2013; 16(8): 981–1004 PubMed PMID: WOS:000325913900004. 91 33 Stoeckert, Elizabeth J. Swain, Andrew Thomson, Lau- 30. Dolan JG, Qian F, Veazie PJ. How well do commonly used data presentation for- 92 mats support comparative effectiveness evaluations? Med Decis Making 2012; 34 rence Titeux, Hendrika A. van den Ham, Tjeerd P. van 32(6): 840–850 PubMed PMID: WOS:000311802700011. English. 93 35 Staa, Ed Waddingham, Nan Wang, Lesley Wise. 31. Elmore JG, Ganschow PS, Geller BM. Communication between patients and 94 providers and informed decision making. J Nat Canser Inst Monogr 2010; 36 Christine E. Hallgreen was previously employed by ▪▪(41): 204–209. Q34 95 37 Novo Nordisk A/S, Soeborg, Denmark, when this 32. Edwards A, Elwyn G, Mulley A. Explaining risks: turning numerical data into 96 – 38 meaningful pictures. BMJ 2002; 324(7341): 827 830. 97 work started. Alex Asiimwe was previously employed 33. Fagerlin A, Zikmund-Fisher BJ, Ubel PA. Helping patients decide: ten steps to 39 by Eli Lilly, USA, when this work was started. better risk communication. J Natl Cancer Inst 2011; 103(19): 1436–1443. 98 40 34. Fischhoff B. Communicating uncertainty fulfilling the duty to inform. Issues Sci 99 Technol 2012; 28(4): 63–70. 41 REFERENCES 35. Galesic M, Garcia-Retamero R. Graph literacy: a cross-cultural comparison. Med 100 42 Decis Making 2011; 31(3): 444–457. 101 1. Pharmacoepidemiological Research on Outcomes of Therapeutics by a European 36. Gaissmaier W, Wegwarth O, Skopec D, Mueller A-S, Broschinski S, Politi MC. 43 Consortium, WP5 benefit–risk integration and representation. ▪▪ [cited 2014]. Numbers can be worth a thousand pictures: individual differences in understand- 102 Q19 Available from: http://www.imi-protect.eu/wp5.shtml. ing graphical and numerical representations of health-related information. Health 44 – 103 2. Mt-Isa S, Hallgreen CE, Wang N, et al. Balancing benefit and risk of medicines: Psychol 2012; 31(3): 286 296 PubMed PMID: WOS:000303629100003. 45 a systematic review and classification of available methodologies. 37. Gigerenzer G, Edwards A. Simple tools for understanding risks: from innumer- 104 – 46 Pharmacoepidemiol Drug Saf 2014; 23(7): 667–678 PubMed PMID: 24821575. acy to insight. BMJ 2003; 327(7417): 741 744. 105 3. EMA Benefit–Risk Methodology Project Team. Work package 1 report: descrip- 38. Gigerenzer G, Gaissmaier W, Kurz-Milcke E, Schwartz LM, Woloshin S. Help- 47 tion of the current practice of benefit–risk assessment for centralised procedure ing doctors and patients make sense of health statistics. Psychological Sci Publ 106 48 products in the EU regulatory network. 2011 Contract No.: EMA/227124/2011. Interest Suppl 2008; 8(2): 53–96. 107 4. Fischhoff B, Brewer NT, Downs JS. Communicating Risk and Benefits: An 39. Goodyear-Smith F, Arroll B, Chan L, Jackson R, Wells S, Kenealy T. Patients 49 Evidence-based User’s Guide: Food and Drug Administration (FDA); 2011 prefer pictures to numbers to express cardiovascular benefit from treatment. 108 50 14.12.2011. Ann Fam Med 2008; 6(3): 213–217. 109 Q20 5. CTSpedia.▪▪ [cited 2013]. Available from: https://www.ctspedia.org. 40. Hawley ST, Zikmund-Fisher B, Ubel P, Jancovic A, Lucas T, Fagerlin A. The 51 Q21 6. Drugs Box ▪▪ [cited 2013]. Available from: http://www.thedrugsbox.co.uk. impact of the format of graphical presentation on health-related knowledge and 110 52 7. DataViz: improving data visualisation for the public sector ▪▪ [cited 2013]. Avail- treatment choices. Patient Educ Couns 2008; 73(3): 448–455. 111 Q22 able from: http://www.improving-visualisation.org. 41. Henneman L, Oosterwijk JC, Van Asperen CJ, et al. The effectiveness of a 53 8. Spiegelhalter D. Understanding Uncertainty website 2010. Available from: graphical presentation in addition to a frequency format in the context of familial 112 54 http://understandinguncertainty.org. breast cancer risk communication: a multicenter controlled trial. BMC Med In- 113 Q23 fl Q35 55 9. FlowingData [cited 2013]. Available from: http:// owingdata.com. form Decis Mak 2013; 13(1). 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2015 116 58 DOI: 10.1002/pds 117 59 118 1 60 visuals for benefit–risk representation 13 2 61 3 42. Hird M. A simple paper-based patient decision aid. Evid Based Med 2008; 13(6): World Health Organization family planning handbook. Am J Obstet Gynecol 62 Q36 –▪▪ – 4 166 . 2006; 195(1): 85 91. 63 43. Hollands GJ, Marteau TM. The impact of using visual images of the body within 65. Tait AR, Voepel-Lewis T, Zikmund-Fisher BJ, Fagerlin A. The effect of format 5 a personalized health risk assessment: an experimental study. Br J Health on parents’ understanding of the risks and benefits of clinical research: a compar- 64 – – 6 Psychol 2013; 18(2): 263 278. ison between text, tables, and graphics. J Health Commun 2010; 15(5): 487 501. 65 44. Kasper J, Heesen C, Köpke S, Mühlhauser I, Lenz M. Why not?—communicat- 66. Tan JKH, Benbasat I. The effectiveness of graphical presentation for information 7 ing stochastic information by use of unsorted frequency pictograms—a extraction—a cumulative experimental approach. Decis Sci 1993; 24(1): 66 Q38 Q37 – 8 randomised controlled trial. Psychosoc Med 2011; 8: Doc08 Doc. 167–191 PubMed PMID: WOS:A1993KU50500010. English. 67 45. Keller C, Siegrist M, Visschers V. Effect of risk ladder format on risk perception 67. Tufte ER. The Visual Display of Quantitative Information. Graphics Press: ▪▪,2001. Q43 9 in high- and low-numerate individuals. Risk Anal 2009; 29(9): 1255–1264. 68. van Valkenhoef G, Tervonen T, Zwinkels T, de Brock B, Hillege H. ADDIS: a 68 10 46. Kosslyn SM. Graph Design for the Eye and Mind. Oxford University Press: ▪▪, decision support system for evidence-based medicine. Decis Support Syst 69 Q39 2006. 2013; 55(2): 459–475. 11 47. Kurz-Milcke E, Gigerenzer G, Martignon L. Transparency in risk communica- 69. Waters EA, Weinstein ND, Colditz GA, Emmons K. Formats for improving risk 70 12 tion: graphical and analog tools. Ann N Y Acad Sci 2008; 1128:18–28. communication in medical tradeoff decisions. J Health Commun 2006; 11(2): 71 48. Lee DH, Mehta MD. Evaluation of a visual risk communication tool: effects on 167–182 PubMed PMID: 16537286 Epub 2006/03/16. eng. 13 knowledge and perception of blood transfusion risk. Transfusion 2003; 43(6): 70. Waters EA, Weinstein ND, Colditz GA, Emmons KM. Reducing aversion to side 72 14 779–787. effects in preventive medical treatment decisions. J Exp Psychol Appl 2007; 73 49. Lipkus IM. Numeric, verbal, and visual formats of conveying health risks: sug- 13(1): 11–21. 15 gested best practices and future recommendations. Med Decis Making 2007; 71. Wilhelms EA, Reyna VF. Effective ways to communicate risk and benefit. The 74 – 16 27(5): 696 713. virtual mentor: VM 2013; 15(1): 34–41. 75 50. Lipkus IM, Hollands JG. The visual communication of risk. J Natl Cancer Inst 72. EMA Benefit–Risk Methodology Project Team. Work package 4 report: benefit–risk Q40 ▪▪ – 17 Monogr 1999; (25): 149 163. tools and processes. http://www.ema.europa.eu/docs/en_GB/document_library/Re- 76 ’ 18 51. Man-Son-Hing M, O Connor AM, Drake E, Biggs J, Hum V, Laupacis A. The port/2012/03/WC500123819.pdf: EMA, 2012 EMA/297405/2012. 77 effect of qualitative vs. quantitative presentation of probability estimates on pa- 73. QNEXA, Vivus, Inc: hearing before the FDA Advisory Committee 2010. Q44 – 19 tient decision-making: a randomized trial. Health Expect 2002; 5(3): 246 255. 74. Sibutramine, Abbot Laboratories: hearing before the FDA Advisory Committee 78 52. Martin RW. Communicating the risk of side effects to rheumatic patients. Rheu- 2010. Q45 20 – 79 matic Dis Clinics North Am 2012; 38(4): 653 662. 75. Nixon R, Stoeckert I, Hodgson G, Pears J, Tzoulaki I, Montero D. Benefit–Risk 21 53. Norton JD. A longitudinal model and graphic for benefit–risk analysis, with case Wave 1 case study report: natalizumab. London: 2013 2013 01/03/2013. Report 80 22 study. Drug Inf J 2011; 45(6): 741–747. No.: 1:b:iv. 81 54. Okan Y, Garcia-Retamero R, Cokely ET, Maldonado A. Individual differences in 76. Rogers S. UK plastic surgery statistics: the Guardian, 2012 [cited 2013]. Avail- 23 graph literacy: overcoming denominator neglect in risk comprehension. JBehav able from: http://www.theguardian.com/news/datablog/2012/jan/30/plastic-sur- 82 24 Decis Making 2012; 25(4): 390–401 PubMed PMID: WOS:000307165400006. gery-statistics-uk. 83 55. Politi MC, Han PKJ, Col NF. Communicating the uncertainty of harms and ben- 77. Man-Son-Hing M, Laupacis A. Balancing the risks of stroke and upper gastroin- 25 efits of medical interventions. Med Decis Making 2007; 27(5): 681–695. testinal tract bleeding in older patients with atrial fibrillation. Arch Intern Med 84 26 Q41 56. Potter K. Methods for Presenting Statistical Information: The Box Plot. ▪▪: ▪▪,2006. 2002; 162(5): 541–550. 85 57. Price M, Cameron R, Butow P. Communicating risk information: the influ- 78. Levitan BS, Andrews EB, Gilsenan A, et al. Application of the BRAT frame- 27 ence of graphical display format on quantitative information perception—ac- work to case studies: observations and insights. Clin Pharmacol Ther 2011 86 28 curacy, comprehension and preferences. Patient Educ Couns 2007; 69(1–3): Feb; 89(2): 217–224 PubMed PMID: 21178990 Epub 2010/12/24. eng. 87 121–128. 79. Mt-Isa S, Hallgreen CE, Asiimwe A, et al. Review of visualisation methods for 29 58. Rakow T. Differences in belief about likely outcomes account for differences in the representation of benefit–risk assessment of medication: Stage 2 of 2. http:// 88 30 doctors’ treatment preferences: but what accounts for the differences in belief? www.imi-protect.eu/benefitsRep.shtml: 2013 15/02/2013. Report No.: 2:ii. 89 Qual Health Care 2001; 10(suppl. 1): i44–i49. 80. Duke S. Best practices recommendations 2012 [ cited 2013]. Available from: 31 59. Schapira MM, Nattinger AB, McHorney CA. Frequency or probability? A qual- https://www.ctspedia.org/do/view/CTSpedia/BestPractices. 90 32 itative study of risk communication formats used in health care. Med Decis Mak- 81. Buja A, Cook D, Swayne DF. Interactive high-dimensional data visualization. J 91 ing 2001; 21(6): 459–467. – 33 Comput Graph Stat 1996; 5(1): 78 99. 92 60. Siegel CA. Review article: explaining risks of inflammatory bowel disease ther- 82. Coplan PM, Noel RA, Levitan BS, Ferguson J, Mussen F. Development of a 34 apy to patients. Aliment Pharmacol Ther 2011; 33(1): 23–32. framework for enhancing the transparency, reproducibility and communication 93 – fi – 35 61. Smith AF. Discussion of risk pervades doctor patient communication. BMJ of the bene t risk balance of medicines. Clin Pharmacol Ther 2011; 89(2): 94 Q42 2002; 325(7363): 548–▪▪. 312–315 PubMed PMID: 21160469. 36 62. Sprague D, Russo JE, Lavallie DL, Buchwald DS. Influence of framing and 95 37 graphic format on comprehension of risk information among American Indian 96 tribal college students. J Cancer Educ 2012; 27(4): 752–758. SUPPORTING INFORMATION 38 63. Spence I. No humble pie: the origins and usage of a statistical chart. J Educa- 97 tional Behav Stat 2005; 30(4): 353–368. 39 64. Steiner MJ, Trussell J, Mehta N, Condon S, Subramaniam S, Bourne D. Commu- Additional supporting information may be found in the online 98 40 nicating contraceptive effectiveness: a randomized controlled trial to inform a version of this article at the publisher’s web site. 99 41 100 42 101 43 102 44 103 45 104 46 105 47 106 48 107 49 108 50 109 51 110 52 111 53 112 54 113 55 114 56 115 57 Copyright © 2015 John Wiley & Sons, Ltd. Pharmacoepidemiology and Drug Safety, 2015 116 58 DOI: 10.1002/pds 117 59 118 Author Query Form

Journal: Pharmacoepidemiology and Drug Safety Article: pds_3880

Dear Author,

During the copyediting of your paper, the following queries arose. Please respond to these by annotating your proofs with the necessary changes/additions. • If you intend to annotate your proof electronically, please refer to the E-annotation guidelines. • If you intend to annotate your proof by means of hard-copy mark-up, please use the standard prooﬁng marks. If manually writing corrections on your proof and returning it by fax, do not write too close to the edge of the paper. Please remember that illegible mark-ups may delay publication. Whether you opt for hard-copy or electronic annotation of your proofs, we recommend that you provide additional clariﬁcation of answers to queries by entering your answers on the query sheet, in addition to the text mark-up.

Query No. Query Remark

Q1 AUTHOR: As per journal, short title should contain <45 characters including spaces; hence, the provided short title was condensed to ‘Visuals for Benefit–Risk Representation’. Please check the suitability of the suggested short title. Q2 AUTHOR: Please confirm that given names (red) and surnames/family names (green) have been identified correctly. Q3 AUTHOR: Please check that authors and their affiliations are correct. Q4 AUTHOR: As per journal, Abstract should not contain reference citations. Please delete the citation and reword the sentence. Q5 AUTHOR: We conducted a comprehensive review of the literature and searched for articles on BR communication and visual formats for risk communication, published after the year 2000 on Scopus up until February 2014, PubMed, Web of Science and PsycINFO (for details of search terms, see Supporting Information). This sentence has been reworded for clarity. Please check and confirm it is correct. Q6 AUTHOR: Fourteen additional sources for visuals were identified including websites and reports. This sentence has been reworded for clarity. Please check and confirm it is correct. Q7 AUTHOR: Alternatives A and B on these visuals may refer to alternative treatments such as rimonabant and placebo for weight loss. This sentence has been reworded for clarity. Please check and confirm it is correct. Q8 AUTHOR: No or very little expertise is required of the users to understand the visuals presented. This sentence has been reworded for clarity. Please check and confirm it is correct. Q9 AUTHOR: It is accessible to patients, general public and suitable for mass media communication. This sentence has been reworded for clarity. Please check and confirm it is correct. Query No. Query Remark

Q10 AUTHOR:Some experience with straightforward BR assessment methodology may be required of the users in is not necessary to understand the theoretical foundation of the model. The meaning of this sentence is not clear; please rewrite or confirm that the sentence is correct. Q11 AUTHOR: It is accessible to BR experts in regulatory agencies, pharmaceutical companies and academia and is suitable for specialist publication only for making high-level decisions. This sentence has been reworded for clarity. Please check and confirm it is correct. Q12 AUTHOR: ‘which in often used for an untrained audience’ has been changed to ‘which are often used for an untrained audience’ for clarity. Please check if correct. Q13 AUTHOR: The simpler bar charts (simple bar chart, stacked bar chart and the grouped bar chart) could be suitable for a large variety of audiences such as the general public through the media, patients, physicians, regulators and other experts for communication about the final BR metric to visualise the contributions of the different criteria (components) in the BR analysis and categorical data. This sentence has been reworded for clarity. Please check and confirm it is correct. Q14 AUTHOR: The tornado diagram is proposed for the communication of uncertainty of the BR metric and visualisation of the relationships between benefit and risk metrics and correlated criteria, again for a trained audience. This sentence has been reworded for clarity. Please check and confirm it is correct. Q15 AUTHOR: The distribution plot is a well-known way of representing data distributions for experts or a trained audience who have some understanding on statistics. This sentence has been reworded for clarity. Please check and confirm it is correct. Q16 AUTHOR: Creating visuals for communication in BR assessments is too important to consider the compatibility between a visual and its target audience. This sentence has been reworded for clarity. Please check and confirm it is correct. Q17 AUTHOR: All persons mentioned in this acknowledgement have participated in the PROTECT BR group; participants have given written consent to be acknowledged in this paper. This sentence has been reworded for clarity. Please check and confirm it is correct. Q18 AUTHOR: Billy Amzal, Simon Ashworth, Johan Bring, Torbjorn Callreus, Edmond Chan, Morten Colding-Jorgensen, Christoph Dierig, Susan Duke, Adam Elmachtoub, David Gelb, Ian Hirsch, Steve Hobbiger, Kimberley Hockley, Juhaeri Juhaeri, Silvia Kuhls, Davide Luciani, Sofia Mahmud, Marilyn Metcalf, Jeremiah Mwangi, Thai Son Tong Nguyen, Richard Nixon, John Pears, George Quartey, Sinan B. Sarac, Isabelle Stoeckert, Elizabeth J. Swain, Andrew Thomson, Laurence Titeux, Hendrika A. van den Ham, Tjeerd P. van Staa, Ed Waddingham, Nan Wang, Lesley Wise. The meaning of this sentence is not clear; please rewrite or confirm that the sentence is correct. Q19 AUTHOR: Please provide the year of publication for Reference 1. Q20 AUTHOR: Please provide the year of publication for Reference 5. Q21 AUTHOR: Please provide the year of publication for Reference 6. Q22 AUTHOR: Please provide the year of publication for Reference 7. Query No. Query Remark

Q23 AUTHOR: ‘Flowing Data’ has been changed to ‘FlowingData’. Please check if correct. Q24 AUTHOR: Please provide the year of publication for Reference 10. Q25 AUTHOR: Please provide the year of publication for Reference 11. Q26 AUTHOR: Please provide the year of publication for Reference 12. Q27 AUTHOR: Please provide the year of publication for Reference 13. Q28 AUTHOR: ‘Gap Minder’ has been changed to ‘Gapminder’. Please check if correct. Q29 AUTHOR: Please provide the year of publication for Reference 14. Q30 AUTHOR: Please provide accessed date for Reference 15. Q31 AUTHOR: Please provide the publisher name and the publisher location for Reference 23. Q32 AUTHOR: Please provide the page range for this chapter in Reference 24. Q33 AUTHOR: Please provide the city location of the publisher for Reference 27. Q34 AUTHOR: Please provide the volume number for Reference 31. Q35 AUTHOR: Please provide the page range for this chapter in Reference 41. Q36 AUTHOR: Please provide the page ranges for References 42 in the Reference List— not just the ﬁrst page. Q37 AUTHOR: Please check if Reference 44 is presented correctly. Q38 AUTHOR: Please provide the page range for this chapter in Reference 44. Q39 AUTHOR: Please provide the city location of publisher for Reference 46. Q40 AUTHOR: Please provide the volume number for this chapter in Reference 50. Q41 AUTHOR: Please provide the publisher name and the publisher locationfor this chapter in Reference 56. Q42 AUTHOR: Please provide the page range for Rreference 61 in the Reference List—not just the ﬁrst page. Q43 AUTHOR: Please provide the city location of publisher for Reference 67. Q44 AUTHOR: Please provide accessed date for Reference 73. Q45 AUTHOR: Please provide accessed date for Reference 74.

USING e-ANNOTATION TOOLS FOR ELECTRONIC PROOF CORRECTION

Required software to e-Annotate PDFs: Adobe Acrobat Professional or Adobe Reader (version 7.0 or above). (Note that this document uses screenshots from Adobe Reader X) The latest version of Acrobat Reader can be downloaded for free at: http://get.adobe.com/uk/reader/

Once you have Acrobat Reader open on your computer, click on the Comment tab at the right of the toolbar:

This will open up a panel down the right side of the document. The majority of tools you will use for annotating your proof will be in the Annotations section, pictured opposite. We’ve picked out some of these tools below:

1. Replace (Ins) Tool – for replacing text. 2. Strikethrough (Del) Tool – for deleting text.

Strikes a line through text and opens up a text Strikes a red line through text that is to be

box where replacement text can be entered. deleted.

How to use it How to use it  Highlight a word or sentence.  Highlight a word or sentence.  Click on the Replace (Ins) icon in the Annotations  Click on the Strikethrough (Del) icon in the section. Annotations section.  Type the replacement text into the blue box that appears.

3. Add note to text Tool – for highlighting a section 4. Add sticky note Tool – for making notes at to be changed to bold or italic. specific points in the text.

Highlights text in yellow and opens up a text Marks a point in the proof where a comment box where comments can be entered. needs to be highlighted.

How to use it How to use it

 Highlight the relevant section of text.  Click on the Add sticky note icon in the

 Click on the Add note to text icon in the Annotations section.

Annotations section.  Click at the point in the proof where the comment

 Type instruction on what should be changed should be inserted. regarding the text into the yellow box that  Type the comment into the yellow box that appears. appears.

USING e-ANNOTATION TOOLS FOR ELECTRONIC PROOF CORRECTION

5. Attach File Tool – for inserting large amounts of 6. Add stamp Tool – for approving a proof if no

text or replacement figures. corrections are required.

Inserts an icon linking to the attached file in the Inserts a selected stamp onto an appropriate

appropriate pace in the text. place in the proof.

How to use it How to use it  Click on the Attach File icon in the Annotations  Click on the Add stamp icon in the Annotations section. section.  Click on the proof to where you’d like the attached  Select the stamp you want to use. (The Approved file to be linked. stamp is usually available directly in the menu that appears).  Select the file to be attached from your computer or network.  Click on the proof where you’d like the stamp to

 Select the colour and type of icon that will appear appear. (Where a proof is to be approved as it is,

in the proof. Click OK. this would normally be on the first page).

7. Drawing Markups Tools – for drawing shapes, lines and freeform annotations on proofs and commenting on these marks. Allows shapes, lines and freeform annotations to be drawn on proofs and for comment to be made on these marks..

How to use it

 Click on one of the shapes in the Drawing Markups section.  Click on the proof at the relevant point and draw the selected shape with the cursor.  To add a comment to the drawn shape, move the cursor over the shape until an arrowhead appears.

 Double click on the shape and type any text in the red box that appears.

For further information on how to annotate proofs, click on the Help menu to reveal a list of further options: