Real-World Evidence for Health Technology Assessment of Pharmaceuticals: Opportunities and Challenges

REAL-WORLD EVIDENCE FOR HEALTH TECHNOLOGY ASSESSMENT OF PHARMACEUTICALS: OPPORTUNITIES AND CHALLENGES

Amr Ahmed Mahmoud Abdelkader Makady ISBN: 978-94-6182-886-6

Cover Image: Off Page, Amsterdam

Layout and print production: Off Page, Amsterdam

Chapter 4: © The Author(s) 2017. Open Access. This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made REAL-WORLD EVIDENCE FOR HEALTH TECHNOLOGY ASSESSMENT OF PHARMACEUTICALS: OPPORTUNITIES AND CHALLENGES

Real-World Evidence voor Health Technology Assessment van Geneesmiddelen: Kansen en Uitdagingen (met een samenvatting in het Nederlands)

Proefschrift ter verkrijging van de graad van doctor aan de Universiteit Utrecht op gezag van de rector magnificus, prof. dr. G. J. van der Zwaan, ingevolge het besluit van het college voor promoties in het openbaar te verdedigen op woensdag 9 mei 2018 des middags te 4.15 uur

door

Amr Ahmed Mahmoud Abdelkader Makady geboren op 5 januari 1990 te Adan, Koeweit Promotoren: Prof.dr. A. de Boer Prof.dr. J. L. Hillege Prof.dr. O.H. Klungel

Copromotor: Dr. W.G. Goettsch TABLE OF CONTENTS

Chapter 1 Introduction 9 Health, Health Systems & the Role of HTA 11 Evidence-Based Medicine & the Efficacy-Effectiveness Gap: 13 Why is this an issue? Real-World Evidence: Potential complement to RCTs? Growing need & 14 the IMI-GetReal project. 14 Statement of Research Gap & Thesis Objective 15 Outline of Thesis 15 Reference List 17

Chapter 2 What is Real-World Data? A review of definitions based on literature and 23 stakeholder interviews Abstract 24 Introduction 25 Methods 26 Analysis 29 Results 29 Discussion 33 Conclusion 38 Reference List 39

Chapter 3 Policies for Use of Real-World Data in Health Technology 45 Assessment (HTA): A comparative study of 6 HTA agencies Abstract 46 Introduction 47 Methods 48 Results 50 Discussion 65 Conclusion 67 Reference List 69

Chapter 4 Using Real-World Data in Health Technology Assessment (HTA) Practice: 75 A comparative study of 5 HTA agencies Abstract 76 Introduction 77 Methods 78 Analysis 79 Results 80 Discussion 84 Conclusions 88 Reference List 89 Chapter 5 Implementing Managed Entry Agreements in Practice: 93 The Dutch reality check Abstract 94 Introduction 95 Methods 96 Results 99 Discussion 103 Conclusion 109 Reference List 110

Chapter 6 Conditional Financing of Drugs in the Netherlands: 113 Past, Present and Future. Results from Stakeholder Interviews Abstract 114 Introduction 115 Methods 116 Results 118 Discussion 126 Conclusion 129 Reference List 130

Chapter 7 Practical Implications of Using Real-World Evidence in 135 Comparative Effectiveness Research: Learnings from IMI-GetReal Abstract 136 Why is real-world evidence (RWE) relevant for comparative effectiveness research (CER)? 137 Why is individual patient-level data (IPD) important when using 137 RWE in CER? What were IMI-GetReal’s experiences in 138 accessing IPD throughout case studies? What are the consequences of inaccessibility to IPD from 140 RWE repositories on its potential use for decision-making in healthcare? What are potential solutions to addressing issues faced with access to 141 IPD from RWE repositories in the future? Reference List 143

Chapter 8 Use of Social Media in the Assessment of Relative Effectiveness: 149 An explorative review with examples from oncology. Hopeful or hopeless? Abstract 150 Introduction 151 Methods 152 Results 153 Discussion 160 Conclusion 162 Reference List 163 Chapter 9 Social Media as a Tool for Assessing Patient Perspectives on Quality of 167 Life in Metastatic Melanoma: a feasibility study Abstract 168 Introduction 169 Methods 170 Analysis 171 Results 171 Discussion 178 Conclusions 186 Reference List 187

Chapter 10 Real-World Evidence for Health Technology Assessment of 193 Pharmaceuticals: Perspectives on the status quo and the future Re-statement of Research Question 195 Main Findings & Implications 195 Ongoing Initiatives on RWE & Relevance to Main Findings 201 Limitations of Research 205 Which steps can be taken towards more optimal use of RWE in 206 HTA decision making? Recommendations based on findings Conclusion 208 Reference List 209

Chapter 11 Appendices 215 Chapter 2 217 Chapter 3 230 Chapter 4 232 Chapter 5 248 Chapter 6 258 Chapter 9 287

Chapter 12 Summary 301 Samenvatting in het Nederlands 304 Acknowledgements 307 List of publications 309 About the author 311

CHAPTER Introduction 1

Introduction 11 1 social, social, provision provision the World Health Health World the the broad range of of range broad a physical, mental and and mental physical, aims and properties of the organization and its aims. and its aims. organization the population, responding to to responding population, state of complete well-being well-being of complete state a a WHO (1). In this constitution, InWHO (1). this constitution, the absence of ailment). Moreover, Moreover, of ailment). absence holistic definition was devised, was devised, definition holistic complicated concept which may which may concept complicated establishment of establishment a the WHO thus commit to to commit WHO thus financial resources needed to strive to strive needed resources financial Dutch Ministry of Finance published Ministry published Dutch of Finance a the the the health of health the the the achievement of good health on population on population of good health achievement United Nations, representatives from 61 member member 61 from representatives Nations, United the following excerpt from Garrido et al. (2) in Box 1. Garrido in Box (2) from al. et excerpt following development of healthcare systems that guarantee guarantee that systems of healthcare development Global Healthcare Expenditure Database (GHED) of of (GHED) Database Expenditure Healthcare Global the surprise that the people and actions whose primary purpose is to improve and actionspeople primary whose improve purpose is to the first Constitution of Constitution first a The The reader with more context on context with more reader contrary, such resources have very tangible limits. To make make To very limits. tangible have such resources contrary, variety of professions and institutions and and institutions and variety of professions government’s second-largest annual financial expenditure expenditure financial annual second-largest government’s the fundamental principles of fundamental national example, example, national a the a definition for “health”; the the Second World War War and World Second promotion, restoration and protection of health. Health systems systems Health health. of protection and restoration promotion, the the a fact that such figures may imply that immense resources are are resources immense that imply may fact such figures that subsidiary of the the a aims of healthcare systems to provide good health to their respective their respective to health good provide to systems aims of healthcare the the policies and actions of other sectors, in an effort to address and actionspolicies effort an in to address of other sectors, plethora of interventions, such as curative therapy for acute or chronic chronic or acute for therapy curative as such interventions, of plethora state of complete well-being which extends to extends which to well-being of complete state end of a a fundamental goals of improving improving of goals fundamental the the the herculean task, herculean requiring wishes and expectations of individuals, and providing financial protection against against protection financial providing and expectationsand wishes individuals, of a costs of ill-health.” of costs health system consists of all all of consists system health - Hallmarks systems. of health WHO ventured to lay lay to WHO ventured A deliveryof constitution stipulated that health, encompassing such encompassing health, that stipulated constitution WHO states that member states dedicated $6.5 trillion U.S. dollars to healthcare systems systems healthcare dollars to trillion $6.5 U.S. dedicated states member that WHO states fundamental human right. right. human fundamental the “ health. This definition covers covers definition This health. environmental, and economic determinants of health … modern health systems generally generally … modern health of systems health determinants and economic environmental, pursue activities dedicated to activitiesto dedicated activities services, and population individual to both in addition aimed at encompass influencing the By ratifying this constitution, member states of member states constitution, this By ratifying Bearing in mind a report in 2015 which estimated national healthcare expenditures in 2016 to reach €75 €75 reach to in 2016 expenditures healthcare report national estimated in 2015 which HEALTH, HEALTH SYSTEMS & THE ROLE OF HTA OF ROLE THE & SYSTEMS HEALTH HEALTH, the states convened to develop develop to convened states This constituted devising devising constituted This level is level Organization (WHO)Organization as However, citizens. all their to of health healthcare systems, we take note of of take note we systems, healthcare know different meanings across cultures. Eventually Eventually cultures. across meanings know different the encompassing encompassing not merely (i.e., of an individual social well-being is the for measures on preventive awareness raise to programs health and public diseases provide to In order diseases. populations, it may not come as come not it may populations, Box 1 Box towards these aims are colossal. colossal. aims are these towards Shortly after a billion Euros, making them Euros, billion in 2010 alone (3). To draw on on draw To alone (3). in 2010 the available to establish healthcare systems worldwide, one must bear in mind that these these that mind in bear must one worldwide, systems healthcare establish to available To infinite. not are resources (4). However, despite despite However, (4).

decision decision European European a majority of incremental incremental broad range range broad the a the scope of HTA is is of HTA scope the transparent and and transparent United States of of States United a the the more abridged, rapid HTA HTA rapid abridged, more a prior of these components (i.e. (i.e. prior components these of United Kingdom).United intervention being investigated investigated being intervention properties of health and effects intervention (i.e. (i.e. intervention past decade have been increasing increasing been decade have past The The the the the the the Netherlands) or private, not-for-profit not-for-profit Netherlands) or private, aim of informing decision makingaim of informing (7). In new intervention compared to standard standard to compared new intervention the the comprehensive HTA model that systematically systematically model that HTA comprehensive the conduct of HTA to inform decision making for making decision inform for to conduct of HTA decision (8). decision a a the policy question at hand and secondly, an appraisal appraisal an policysecondly, and hand at question the cost-effectiveness of cost-effectiveness scientific, robust process (8). (8). process robust scientific, a systematic evaluation of evaluation systematic the the evidence to reach reach to evidence the Institute for Clinical and Economic Review (ICER), Economic Clinical and Institute for relative effectiveness- (REA) and cost-effectiveness assessment (CEA) of (CEA) cost-effectiveness assessment and (REA) effectiveness- relative policy analysis process including two components; firstly, an assessment assessment an firstly, components; two including policy process analysis a timeframe within which decision makers are often obliged to reach reach often to obliged are makers decision which within timeframe the annual rate of gross domestic product (GDP) growth of growth product domestic (GDP) gross of rate annual the conduct of HTA across nine domains, including those mentioned above (9). (9). above mentioned those including domains, nine across conduct of HTA the consequence of rising healthcare costs and finite budgets, governments are are governments budgets, finite and costs of rising healthcare consequence added therapeutic effect of an intervention compared to current standard of of standard to current compared of an intervention effect added therapeutic the properties of health technologies assessed through HTA may cover cover may HTA assessed through properties technologies health of the reimbursement of health technologies may necessitate necessitate may technologies of health reimbursement a findings from from findings the development of clinical guidelines and ensuring good quality of healthcare (e.g. (e.g. ensuringand guidelines quality clinical good of healthcare of development greatest health gains for their citizens. In an attempt to provide provide to Inattempt an their citizens. for health gains greatest As From an organizational perspective, perspective, an organizational From The The the the matters more complex, healthcare expenditures over over expenditures healthcare complex, more matters beyond beyond constantly faced with challenging questions on how to allocate resources to achieve achieve to resources allocate to how on questions challenging with faced constantly the countries (5). countries policy healthcare resource on to making decision to related approach accountable Briefly (HTA). Assessment Technology Health turn to increasingly governments allocation, pertains to HTA defined, technologies (whether drugs, medical devices, surgical procedures or organizational organizational or procedures surgical devices, medical drugs, (whether technologies as their well as effects, direct their and intended (6)), addressing aspects systems of health with consequences indirect and unintended general, HTA is HTA general, to relevant evidence available all of of assessment) is conventionally is conventionally assessment) different under each operating agencies, HTA to is often delegated systems healthcare could be agencies HTA example, For mandates. structures and different governance Ministries of national length advisory public independent, arm’s at bodies operating Nederland (ZIN), Zorginstituut (e.g. of Health (e.g. entities National Institute for Health and Care Excellence (NICE), (NICE), Excellence and Care Health Institute for National America). Moreover, HTA agencies’ mandates could be limited to one domain of HTA (e.g. (e.g. domain of HTA one to limited be could mandates agencies’ HTA Moreover, America). Wirtschaftlichkeit und technologies; für Instituts Qualität health of effectiveness clinical as HTA, such tasks numerous or encompass Germany) (IQWiG), im Gesundheitswesen the of domains including: legal issues, ethical issues, societal considerations, organizational organizational considerations, societal issues, ethical issues, legal of domains including: years, recent In and safety. cost-effectiveness aspects, effectiveness, clinical developed has (EUnetHTA) network of HTA guides However, However, based on core domains (10). Relative effectiveness of effectiveness Relative (10). domains core on based (i.e. increasingly, and, (11) care) of care) (12) are two examples of such core domains. In this thesis, In this thesis, domains. such core of examples two (12) are of care) to restricted drugs. pharmaceutical on costs and effects incurred by reimbursing reimbursing by incurred and effects costs

Introduction 1 Introduction 13 1 realized realized questions questions past three three past community community the prioritization prioritization medical field, field, medical latter setting, setting, latter the the the intervention. the the the drug is considered drug is considered quality of evidence qualityof evidence a the real-world as it has been has been as it real-world the most valid method for for method most valid the the form of relevant evidence that is is that evidence of relevant form a “real-world”). In “real-world”). drug. As applied to to applied As drug. complementary insights provided by by complementary provided insights the a healthcare system, system, healthcare the a new drug, may also impact may new drug, highest levels of evidence. Moreover, RCTs RCTs Moreover, of evidence. levels highest drug efficacious when delivered under ideal ideal under when delivered drug efficacious relative effectiveness of effectiveness relative theoretical mismatch between between mismatch theoretical a the the the the questions decision makers face are plentiful. As As plentiful. are decision makers face questions control arm (e.g. placebo or active comparator) and and or active comparator) placebo arm (e.g. control degree of clinical experience amongst healthcare amongst healthcare of clinical experience degree different question to that addressed by RCTs, namely: by RCTs, addressed to that question different a the a field of pharmacoepidemiology have aimed to develop to develop aimed have pharmacoepidemiology field of the perspective of the time HTA is conducted. time HTA the ideal context of RCTs. Moreover, in contrast to RCTs whereby whereby RCTs to in contrast Moreover, of RCTs. ideal context new drug can be as efficacious in efficacious new drug can be as principlesEvidence-Based of conduct Medicine to (EBM) (13) the a implementation of implementation the the scientific validity of conclusions reached during an assessment, HTA HTA duringreached an assessment, validity conclusions of scientific the the new drug’s comparative effects in relation to all available alternatives. As As alternatives. available all to relation in effects comparative drug’s new core source of evidence for marketing authorization applications of drugs of drugs marketing applications authorization for evidence of source core intervention. When deciding on resource allocations, decision makers thus thus makers decision allocations, on resource deciding When intervention. intended and unintended effects of effects and unintended intended the other hand, from from other hand, the gulf which has been much under-estimated” (17). Throughout Throughout (17). gulf which has been much under-estimated” the the verynamely: specific question, is drug work when delivered in routine clinical practice settings compared to current current to compared practice settings clinical in routine delivered when drug work a a the the new drug is conventionally compared to one alterative, decision-makers require require decision-makers one alterative, to compared is conventionally new drug On Randomized controlled clinical trials are considered considered trials clinical are Randomized controlled Examples in scientific literature about literature in scientific Examples EVIDENCE-BASED MEDICINE & THE EFFICACY-EFFECTIVENESS THE & MEDICINE EVIDENCE-BASED ISSUE? AN THIS IS WHY GAP: conditions? (13) conditions? RCTs comprise strict inclusion- and exclusion criteria of patients, randomized allocation of of allocation randomized criteria of patients, strict comprise exclusion and inclusion- RCTs and drug experimental an to patients professionals with professionals assessing assessing to designed are RCTs Therefore, of trial subjects. protocols strict and follow-up monitoring address comorbiditiesthan patients different with present and heterogeneous more are patients Other aspects, such as in RCTs. of effects conventionally available at at available conventionally In order to ensure ensure to In order agencies often rely on oftenagencies rely systematic reviews of available scientific literature, categorize categorize literature, scientific available of reviews systematic applying When its interpretation. and upon GRADE based recommendations) (14) (e.g., towards gravitate to agencies tend REA and CEA, to this methodology of Randomized Controlled Trials (RCTs) or meta-analyses of RCTs in their evidence base base evidence their in RCTs of meta-analyses or (RCTs) Trials Randomizedof Controlled as classified conventionally they are since by administered when patients to benefit of evidence is sufficient when there effective practice (i.e. clinical settings in routine physicians whether consider to have as to referred is This (13) of care? standard constitute constitute they are such, regulatory (15). As agencies to submitted decades, numerous advances in advances numerous decades, demonstrated to be in be to demonstrated the on evidence face regularly such, decision makers does early as 1967, Schwartz et Lellouch alluded to alluded to early Schwartz as 1967, Lellouch et (16). observational 1972, In exploratory trials (i.e. trials) confirmatory and RCTs) trials (i.e. in and benefit based on RCTs “Between measurements that stated Cochrane there is there RCTs are designed to answer and answer to designed are RCTs use of use of context effects of of effects real-world real-world the the the the potential uses uses potential whole (2;13). whole drug in clinical drug in clinical real-world. They They real-world. a decision-maker’s decision-maker’s question at hand hand at question the a the the the discrepancy does exist discrepancy exist does world began to resort began to world paper by Garrison et al. Garrison by paper et al. a the the general framework of MEAs of MEAs framework general auspices of such agreements of such agreements auspices population as population the the the dilemma: guaranteeing quick access to to access quick guaranteeing dilemma: a shortcomings associated with heavy reliance heavy with shortcomings reliance associated seminal example is example seminal promise for additional evidence generation on on generation evidence additional for promise A the the wealth of literature abounded whereby authors focused focused authors whereby abounded of literature wealth a “efficacy-effectiveness gap” (23). From (23). gap” “efficacy-effectiveness the need for post-authorization evidence to address uncertainties address to evidence post-authorization need for evidence generated under under generated evidence implementation of RWEimplementation making: in decision recognition of recognition term Real-World Data (RWD) to mean any data on data (RWD) mean any Data to Real-World term the the the the the gap can be worrisome as it implies that evidence for REA and CEA is CEA is REA and for evidence that implies as it worrisome gap can be broad clinical population (18;19). Researchers have also voiced their their also voiced have Researchers (18;19). population clinical broad a need for relevant, alternative sources of evidence to assess assess to evidence of sources alternative relevant, for need analysis of evidence from alternative trial designs on drug effectiveness effectiveness on drug trial designs alternative from of evidence analysis number of potential sources of RWD were delineated, including electronic electronic including delineated, of RWD were sources potential of number the the combination of their exorbitant prices along with significant uncertainties along with significant prices of their exorbitant combination loss in potential health benefits incurred to incurred benefits health potential loss in the A effects of drugs as demonstrated in RCTs and as realized in realized as and in RCTs demonstrated of drugs as effects a different note, HTA agencies have recently witnessed an increase in submissions submissions in increase an witnessed recently have agencies HTA note, different a the the reimbursement of innovative, yet expensive, drugs with less evidence on efficacy with less evidence drugs expensive, yet of innovative, reimbursement evidence base. Consequently, decision makers around makers around decision Consequently, base. evidence term Real-World Evidence was coined to entail knowledge generated based on on based knowledge entail generated to coined was Evidence Real-World term drugs in question (25;26). Historical examples include drugs for orphan diseases and orphan and diseases Historicaldrugs in question (25;26). for drugs include examples synthesis of RWD. Since then, Since RWD. of synthesis the Arguably, these historical and contextual factors on fueled discussions and contextual historical these Arguably, the health intervention (clinical, economic or patient-reported) collected outside collected patient-reported) or economic (clinical, intervention health REAL-WORLD EVIDENCE: POTENTIAL COMPLEMENT TO RCTS? RCTS? TO COMPLEMENT POTENTIAL EVIDENCE: REAL-WORLD PROJECT IMI-GETREAL THE & NEED GROWING health records (EHRs), patient registries and administrative claims databases. Additionally, Additionally, claims databases. and administrative registries (EHRs), patient records health the the in RCT and complement to makingin decision data sources data on coverage alternative policy in health reimbursement journals. which coined (2007) a (30). of RCTs aspectson multiple of oncology drugs (27). Although such drugs potentially address unmet medical need for need for unmet medical address potentially drugs such Although drugs (27). oncology patients, left base evidence in their makers with decision and safety and therefore less evidence on relative effectiveness and cost-effectiveness of cost-effectiveness and effectiveness relative on evidence less and therefore and safety the drugs for patients versus versus patients drugs for Within (27;28). (MEAs) entry managed to agreements to access patient schemes) (29), (CED) development evidence with (particularly coverage with be facilitated new drugs would impact and/or budget cost-effectiveness of effectiveness, (relative) By definition, practice. would thus originate from non-RCT from thus originate sources. would concerns over over concerns on evidence from RCTs for decision making in healthcare has long been evident in literature in literature decision making evident has long been for healthcare in RCTs from on evidence On (6;24). for between between based on data from RCTs designed to provide alternative insights to to insights alternative provide to designed RCTs from on data based effectiveness of drugs (20-22). In 2011, Eichler et al. concluded that that concluded (20-22). of drugs al. et Eichler In 2011, effectiveness as phenomenon this to refer such perspective, allocated, being optimally not resources in healthcare result may this Ultimately, (13;22). in resulting methods for for methods in and safety As mentioned above, above, mentioned As

Introduction 1 Introduction 15 1 IMI- project project form of of form the a Innovative Innovative use of RWE The The experiences experiences use of RWEuse in the the the the clear understanding clear understanding collection and use of collection and use of a European Commission Commission European 5 agencies. Chapters 5 5 Chapters agencies. 5 authors’ involvement in in involvement authors’ use of RWE in HTA. These These RWEuse of in HTA. the the the series of case studies were series were case studies of the the period of 3 years, period of 3 years, unanimous understanding of of understanding unanimous A definitions of RWD available in of definitions in available RWD a a policies and practices of RWEpolicies use the the scope of this thesis. scope design of appropriate studies to generate generate to studies of appropriate design the the topic published between 2007 and 2014, an an and 2014, 2007 between published topic use of RWEof use across potential use of RWE to complement RCTRWE of use complement potential to data lifecycle of drugs. drugs. of lifecycle second section, “Policies for Real-World Evidence Evidence Real-World for “Policies section,second the the broad range of stakeholders; from HTA agencies, agencies, HTA from of stakeholders; range broad the implementation of Conditional Financing; Financing; of Conditional implementation third section, “Real-World Evidence Use in Practice”, Practice”, in Use Evidence “Real-World section,third the a the the The The policies from 6 European HTA agencies on agencies HTA 6 European from policies the public-private partnershippublic-private between practical and cultural obstacles associated with with associated practicalobstacles cultural and a acceptability of these methods amongst relevant stakeholders. stakeholders. relevant methods amongst of these acceptability the analysis and interpretation of RWE of (34). interpretation and analysis Netherlands on the the consortium’s efforts influenced consortium’s access and use of RWEaccess IMI-GetReal within in all case studies conducted the fourth section, “Access to Real-World Evidence”, summarizes summarizes Evidence”, Real-World to “Access fourth section, relatively recent interest in RWE use in decision making in healthcare, and RWE in and making use in decision interest in healthcare, recent relatively the abundance of literature on on of literature abundance the midst of rising awareness on RWE stakeholders, various amongst risingmidst of awareness The The the the European Medicines Agency (EMA), pharmaceutical industry, small-to-medium small-to-medium industry, pharmaceutical Medicines Agency (EMA), European comparative review of review comparative the a first section, “What is Real-World Data?”, examines examines Data?”, is Real-World “What first section, project consortium of consisted This thesis aims to address this gap by exploring exploring by this gap address aims to thesis This In the OUTLINE OUTLINE THESIS OF STATEMENT OF RESEARCH GAP & THESIS OBJECTIVETHESIS & GAP RESEARCH OF STATEMENT in decision making conducted. is examines if and how RWE is used in HTA practice. Chapter 4 explores if RWE 4 explores in practice used is Chapter practice. RWE if and how examines in HTA is used compares and agencies European 5 by what RWD is, whether RWE is used by HTA agencies, in which contexts RWE is used (e.g. for for RWE in which contexts (e.g. is used agencies, RWD RWE whether what HTA is, by used is and or MEA) CEA REA, apparent gap exists in knowledge regarding several aspects of several in knowledge gap exists regarding apparent have stakeholders whether to: not limited are but include, despite despite regarding regarding The The literature and amongst eight different stakeholder groups to provide to provide groups stakeholder different and amongst eight literature RWE and (22;33) In constitutes. it specifically of what Use”, Use”, for RWE making decision for in informing (31;32), and 6 present an in-depth with gained of experiences and 6 present analysis in RWE HTA in HTA and decision making. and HTA Provided Provided in HTA of drugs. in HTA GetReal consortium explore to sought was funded through IMI, IMI, funded through was MEA (36). Medicines Initiative (IMI)-GetRealMedicines Initiative (35). 2014 launched in February project was throughout research effectiveness in and European Federation for Pharmaceutical Industries and Associations (EFPIA). As such, such, As Industries (EFPIA). Pharmaceutical and Associations for Federation and European the to enterprises, academia and patient organizations. Over organizations. patient and academia enterprises, conducted to explore methods for using RWE to improve (relative) effectiveness estimates estimates effectiveness (relative) using RWE improve for methods to explore to conducted examine and to IMI-GetReal, Furthermore, best practices for evidence synthesis from RWE and/or RCTs were explored explored were RWE from RCTs and/or synthesis evidence best practices for Furthermore, Bearing in mind case studies. and reviews literature through study whereby study whereby a fifth section, “Novel The The potential for generating RWE generating for potential the final section, “General Discussion”, summarizes summarizes Discussion”, “General final section, The The challenges, opportunities and possible future approaches for for opportunities approaches challenges, future and possible the increased use of RWE in HTA and decision making.and decision use of RWEincreased in HTA findings and discusses and discusses findings the the social media is used to collect data on patient perceptions on health-related quality-of- on health-related perceptions collect on patient data to social media is used surveys. health (HRQoL) through life from novel sources (i.e. social media) as an alternative to, for example, EHR’s and patient patient and EHR’s example, for to, media) as an alternative social (i.e. sources novel from RWE on relative generate to using social media of applications 8 reviews Chapter registries. from findings 9 presents Chapter Meanwhile, interventions. of effectiveness and discusses their implications on RWE on implications their and discusses making. decision in use Sources for Generating Real-World Evidence”, explores explores Evidence”, Real-World Generating for Sources

Introduction 1 Introduction 17 1

. Drug www. Lancet Lancet The The confusion. confusion. The The www.fda.gov/ the . evidence for decisions about about decisions for evidence results of this trial of this apply? results Milbank Quarterly 2010;88(2):256-76. use of therapeutic interventions. Clinical Clinical interventions. therapeutic use of Luce BR, Drummond M, Jänsson B, Jänsson B, Drummond M, BR, Luce Schwartz JS, Siebert al. et UWE, PJ, Neumann clearingCER: and HTA, EBM, The nuffieldtrust.org.uk/research/effectiveness- and-efficiency-random-reflections- on-health-services 2005;365(9453):82-93. Real-world T. Strack N, Freemantle be should of new medicines effectiveness clinical designed appropriately by evaluated 2010;63(10):1053-8. J Clin Epidemiol trials. A, Breckenridge E, Abadie HG, Eichler et B, Leufkens LL, Gustafsson B, Flamion Schwartz D, Lellouch J. Explanatory and J. Lellouch Schwartz D, J trials. in therapeutical attitudes pragmatic Dis 1967;20(8):637-48. Chronic efficiency: and A. Effectiveness Cochrane Random reflections services. health on https:// from: June 1 Available 1972 for Available Designs BL. Study Strom Pharmacoepidemiologic Textbook Studies. & Wiley John of Pharmacoepidemiology. 17-29. p. 2013. Sons Ltd; BM, Grobbee Psaty Klungel OH, Martens EP, Methods al. Stricker BHC, et SD, Sullivan DE, of drug treatment effects assess intended to J Clin in observational reviewed. are studies 2004;57(12):1223-31. Epidemiol External validity PM. of Rothwell do whom To trials: controlled randomised the on Rawlins De M. testimonio: the the Guyatt GH, Oxman AD, Vist GE, Kunz R, R, Kunz GE, Vist AD, Oxman GH, Guyatt GRADE: al. et Alonso-Coello P, Y, Falck-Ytter quality of rating on consensus an emerging of recommendations. and strength evidence 24;336(7650):924-6. 2008 Apr BMJ Administration. Drug and Food Oct 24 [cited June 2015 Process. Development https:// from: Available 2017]. ForPatients/Approvals/Drugs/default.htm Medicine 2008;8(6):579-88.

23. 20. 21. 22. 18. 19. 15. 16. 17. 13. 14.

. . http://www. http://www. . http://htaglossary. improved conduct of conduct of improved http://www.who.int/about/ the global overview. 2017 April April 2017 overview. global efficacy of medical and safety A . the economic evaluation of health care care health of evaluation economic programmes. Oxford university press; 2015. press; university Oxford programmes. Drummond MF, Sculpher MJ, Claxton K, Sculpher MJ, MF, Drummond Methods for GW. Torrance Stoddart GL, the eunethta.eu/hta-core-model A, Kleijnen Goulden S, d’Andon E, George S, effectiveness Relative Osińska et al. B, P, Vitré similarities pharmaceuticals: of assessment Value in 29 jurisdictions. and differences 2012;15(6):954-60. Health Forum Pharmaceutical High Level HLPF. and Conclusions Final 2005-2008. 2017]. Apr [cited 2008 Recommendations. EUnetHTA. HTA Core Model®. 2017 [cited Model®. [cited 2017 Core HTA EUnetHTA. from: 2017]. Available Nov Drummond MF, Schwartz JS, Jänsson B, Schwartz B, Jänsson JS, MF, Drummond Key et al. Siebert U, PJ, BR, Neumann Luce principles for [cited Aug 2017]. Available from: from: Available 2017]. Aug [cited who.int/mediacentre/factsheets/fs319/en/ DP. Andrulid Behney CJ, HD, Banta Assessing of Office Washington: technologies. 1978. Assessment Technology Assessment Technology Health Aug [cited 2017 Glossary. HTAi International. from: Available 2017]. net/tiki-index.php?page=List+all+terms Work Health Organisation. Spending on on Spending Organisation. Health Work healthcare: Miljoenennota 15-9-2015. 2016. Rijksoverheid. spending? health for future What OECD. Department Economic OECD Policy Garrido MV, Gerhardus A, Röttingen JA, RöttingenA, JA, Gerhardus Garrido MV, health technology Developing Busse R. system care health address to assessment 2010;94(3):196-202. Health Policy needs. . 19-6-2013. Notes Work Health Organisation. Constitution Constitution Organisation. Health Work 2017]. Aug [cited 2017 Principles. WHO: of from: Available mission/en/ health technology assessments for resource resource for assessments technology health Assess Technol Int J decisions. allocation 2008;24(3):244-58. Care Health

REFERENCE LIST REFERENCE 12. 11. 10. 8. 7. 6. 5. 4. 3. 2. 1. 9.

http:// http://pharmacy. . . ISPOR good research ISPOR good research . the ENCePP Code of Conduct for of Conduct for Code ENCePP www.imi-getreal.eu/About- Performance-based risk-sharing The The Conduct of Pharmacoepidemiological Conduct of Pharmacoepidemiological ISPOR Real World Data Task Force Force Task Data World ISPOR Real ISPOR good practices task force. Value Value practices force. ISPOR good task the www.encepp.eu/documents/encepp_ studies/ENCePP%20Code%20of%20 Conduct_20100507.pdf ENCePP. ENCePP. Transparency Scientific and Independence in 2010 Studies. and Pharmacovigilance from: Available Jul 2015]. [cited JM, Alvir Riaz MPP, Ali D, Alemayehu EM, et Perfetto MullinsMardekian CD, J, Issues Analytical Data, of Examination al. Conducting Methods for and Proposed Using Research Effectiveness Comparative Pharm J Manag Care Data”. “Real-World suppl.):S3-S37. 2011;17(9 IMI-GetReal. IMI-GetReal: objectives. Overall from: DecAvailable 2016]. [cited 2017 https:// GetReal/Overall-objectives Briggs A, A, De Towse Garrison LP, Mohr PE, J, G, Grueger Pouvourville et al. design, - good practices for arrangements report evaluation: and of implementation, the 2013;16(5):703-19. Health Berger ML, Dreyer N, Anderson F, Towse Towse F, Anderson N, Dreyer ML, Berger A, Sedrakyan Normand A, SL. Prospective observationalcomparative assess to studies effectiveness: Value report. practices task force 2012;15(2):217-30. Health in Data World Real Using AM. Pleil Challenges, Evaluations: Pharamcoeconomic [cited 2013 Opportunities and Approaches. from: Available 2015]. Jul ucsd.edu/faculty/AppliedPEForum/docs/ Andreas_M_Pleil.pdf Data for Coverage and Payment Decisions: Decisions: and Payment Coverage for Data The 2007;10(5):326-35. Health Value Report.

36. 34. 35. 32. 33. 31.

United United American American taxonomy taxonomy the a The The United States: States: United the efficacy-effectiveness gap: the European Union and European the regulator’s perspective on addressing perspective on addressing regulator’s Garrison LP, Neumann PJ, Erickson P, Erickson P, Neumann PJ, Garrison LP, World Real Using Mullins CD. Marshall D, Carlson JJ, Sullivan SD, Garrison LP, Garrison LP, SD, Sullivan JJ, Carlson DL. Linking Veenstra PJ, Neumann outcomes: health to payment journal of managed care 2015;21(9):632-40. journal care managed of L, Morrell D, M, Qin Lee SD, Faulkner and Pricing A, et al. Sammarco E, Xoxi and insights experiences reimbursement in of performance-basedand examination between schemes reimbursement manufacturers. and payers healthcare 2010;96(3):179-90. Policy Health Carlson JJ, Gries KS, Yeung K, Sullivan SD, K, SD, Sullivan Yeung Gries KS, JJ, Carlson in trends and status Current Garrison LP. performance-based arrangements risk-sharing and medical payers healthcare between health Applied product manufacturers. policy and health economics 2014;12(3):231-8. approach learnedto Lessons States: Pharmacol Clin pathways. payer adaptive 2016;100(6):730-42. Ther Garrison Jr LP, Carlson JJ, Bajaj PS, Towse A, A, Towse PS, Bajaj JJ, Carlson Garrison LP, Jr sector Private al. et SD, Sullivan PJ, Neumann in agreements risk-sharing and prospects. barriers, trends, Lohr KN, Eleazer K, Eleazer KN, policy Health MauskopfJ. Lohr evidence-based for applications issues and practice guidelines. and clinical medicine 1998;46(1):1-19. Policy Health Lipska Leufkens N, I, Hoekman McAuslane J, Does conditional AM. H+Âvels HGM, drugs in Europe oncology new for approval technology health in differences lead to Pharmacol Clin decisions? assessment 1;98(5):489-91. Nov 2015 Ther al. Bridging Bridging al. a Rev Drug Nat variabilityresponse. drug of 2011;10(7):495-506. Discov

30. 29. 28. 27. 26. 25. 24.

Introduction 1

SECTION What is Real-World Data? I

CHAPTER

What is Real-World Data? A review of definitions based on literature and 2 stakeholder interviews

A. Makady1,2, A. de Boer2, H. Hillege3, O. Klungel2, W. Goettsch1,2 (on behalf of GetReal Work Package 1).

1. The National Healthcare Institute (ZIN), Diemen, the Netherlands; 2. Department of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht, the Netherlands; 3. Department of Epidemiology, University Medical Centre Groningen, Groningen, the Netherlands.

Published in Value in Health. Citation is as follows: Makady A, de Boer A, Hillege H, Klungel O, Goettsch W. What Is Real-World Data (RWD)? A Review of Definitions Based on Literature and Stakeholder Interviews. Value in Health. 2017 Aug 20(7);858-865. non- a frequency frequency significant significant a The The non-RCT setting, a non-RCT setting. However, non-RCTHowever, setting. a value of real-world data (RWD), consensus on (RWD), on data consensus of real-world value the non-interventional/non-controlled setting, 3-Data collected in in collected 3-Data setting, non-interventional/non-controlled a majority defineddefinitions of collected in data as RWD aspects of intervention with which non-interventional/non-controlled settings should aspects should settings which non-interventional/non-controlled with intervention of definition of RWD is lacking. We aimed to review definitions publicly available for available RWD publicly definitions review to aimed We of definition is lacking. RWD literature review and stakeholder interviews were used to compile data from 8 groups 8 groups from data compile to used interviews and stakeholder were review literature considerable number of definitions diverged from this concept. Moreover, Moreover, concept. this from diverged definitions of number considerable ABSTRACT of definitions identified per category recorded. per identified of definitions was experimental setting and 4-Other (i.e. do not fit into three categories above). above). categories setting and 4-Otherexperimental three into do not fit (i.e. The The Conclusions 2-Data collected in collected 2-Data of 20 out identified: were 38 definitions assessed. interviews and 20 53 documents were 5 category were 2 definitions, category 9 (24%) were 1 definitions, 38 definitions (53%) were category Differences categorywere 4 definitions. and 4 (11%) 3 definitions were (13%) on opinions differed example, For categories. definition and within, between, identified the 2 interviews in 33 documents. in or identified provided were No definitions abide. a Results A Methods analysis. interviewscoding and subjected documents was to from Data stakeholders. of collected in categories: 1-Data 4 into classified were Definitions identified Background of recognition increasing Despite number of authors and stakeholders did not have an official, institutional definition for definition institutional official, an have not did stakeholders and of authors number variabilityof to disparities definitions in stakeholder among lead RWD may Persisting RWD. when discussing stakeholders use in decision-making. RWD different in order to shed light on similarities and differences between them. between similarities on differences and shed light to in order the

What is RWD? 2 What is RWD? 25 2 review of of review article will a term within within term comparative comparative influence for for influence the the the the ideal study design for for design ideal study setting of RCTs are often not often not are of RCTs setting RCT to complementary source the result, many stakeholders such such stakeholders many result, a a effectiveness of medicines, as well well as of medicines, effectiveness the the specific medicine and intended and and and intended medicine specific issue of making upon based decisions time of initial reimbursement decisions decisions reimbursement initial time of real world has been previously defined by by defined has been previously world real a given indication. given a the the the aspects of real-world effectiveness. Similarly, Similarly, effectiveness. aspectsreal-world of use of broader, more heterogeneous population (2).This (2).This population heterogeneous more broader, the a the disease, delineating treatment pathways in clinical practice, practice, clinical in pathways treatment delineating disease, therapeutic efficacy of medicines from tightly-controlled efficacy tightly-controlled therapeutic from of medicines RCT a use of real-world data (RWD) data as use of real-world the more robust evidence base on base evidence robust more the certain degree of disparity remains among different stakeholders when when stakeholders disparity of certain different among remains degree a similarities and differences between them. Additionally, Additionally, them. between similarities and differences a more heterogeneous populations in clinical practice where medicines medicines practice where clinical in populations heterogeneous more “efficacy-effectiveness gap” (3). gap” “efficacy-effectiveness costs and resource use associated with treatment interventions and and interventions treatment with associated use resource and costs the effectiveness of medicines in of medicines effectiveness the increasing popularity of RWD collection and use for drug development, drug drug popularity RWDof increasing development, drug collection for use and the relative effectiveness of new products. As products. new of effectiveness relative the the the the highly-selective populations examined within within examined populations highly-selective natural historynatural of relative effectiveness compared to existing productsexisting practice (4;5). in clinical to compared effectiveness relative context of drug development, drug regulation and HTA of pharmaceutical products, in in products, of pharmaceutical and HTA regulation drug development, of drug context Additionally, RWD is currently used during drug development to examine aspects examine to such RWD during used development drug currently is Additionally, The The Despite Despite Regulatory with thus faced agencies are the the INTRODUCTION review which data sources stakeholders believe as being RWD and which study designs they as being RWD they believe stakeholders designs study and which sources which data review the identify to order definitions for RWD available in literature and stakeholders’ definitions of stakeholders’ and for literature in available definitions RWD it comes to thoroughly defining RWD (6). Therefore, this articleconduct to aims Therefore, defining thoroughly RWD (6). to it comes as comparable to to comparable co- with varying patients to make-ups, genetic with different administered who present are Consequently, other morbidities. for medications different receive or already morbidities, by accompanied marketing are authorisation for being presented medicines experimental veryefficacy results with validitydata whose but internal safety and provides high that data to easily generalizable be not may establishing for data as determining determining known and unknown confounders (1;2). Additionally, monitoring and follow-up procedures procedures follow-up and monitoring Additionally, (1;2). known and unknown confounders (1;2). trial subjectsfor is often highly controlled disparity findings on of knowledge Such (4;6). interventions may comparator to related outcomes determining or as clinical trial aspects such design inform of early drug development within treatments comparator of effectiveness regulation and HTA, HTA, and regulation Randomised controlled clinical trials (RCT’s) provide provide trials clinical (RCT’s) Randomised controlled demonstrating causality between demonstrating unintended effects under ideal conditions. In conventional RCT’s conducted during Phase conductedPhase during RCT’s conventional In conditions. under ideal effects unintended criteria and inclusion and exclusion stringent on based are patients III drug development, counteract to arms treatment different to randomised subsequently settings and settings as et al. Eichler begun have payers and agencies HTA regulatory agencies, industry, as pharmaceutical options for exploring data with inherent uncertainties on inherent with data conventionally payers healthcare and agencies (HTA) Assessment Technology Health at available evidence RCT-generated exploit assess to literature literature a pharmaceutical pharmaceutical 2 authors. context of of context the the the list of websites consulted). consulted). list of websites a Appendix). Any discrepancies for for discrepancies Any Appendix). search strategy used is presented presented is used strategy search British Pharmaceutical IndustryBritish Pharmaceutical (ABPI) effects of health interventions interventions of health effects the The The the the article definitions on existing shed light will Appendix for for Appendix the the IMI-GetReal consortium (10) (see Table 1). Table IMI-GetReal consortium (see (10) International Society for Pharmacoeconomic and Health and Health Society Pharmacoeconomic International for Association of Association the the the , 2016 (date of search). of search). (date , 2016 st purposes of this guidance, RW data will refer to data obtained by any non- any by obtained data to RW refer will data purposes guidance, of this healthcare sector comes from various sources and includes patient data, data data data, patient includes and sources various from sector comes healthcare the Appendix. To locate grey literature, websites belonging to 8 stakeholder 8 stakeholder to belonging websites literature, grey locate To Appendix. Definition RCTs. in conventional decision-making collected not are for used that Data For from clinicians, hospital data, data from payers and social data. and payers from data data, hospital clinicians, from regarding data for term umbrella An interventional methodology that describes what is happening in normal clinical in normal happening is describesclinical what that methodology interventional practice. data healthcare types of different for term (RWD) data is an umbrella Real-world RWD trials. in controlled randomised in conventional not collected are that the (e.g. benefit, risk, resource use, etc) that are not collected in not are that etc) use, risk, benefit, resource (e.g. conventional randomised controlled trials. Instead, real world data (RWD) is (RWD) data is world real Instead, trials. controlled randomised conventional observations from of routine and retrospectively both prospectively collected and clinical to, not limited but are include, collected Data practice. clinical quality health-related and (PRO) reported outcomes patient outcomes, economic patient including sources (HRQoL). RWD many of life from be obtained can observational and studies. electronic records, medical registries, the literature review, PubMed was used to search scientific literature from January from literature scientific search to used was PubMed review, literature RAND Corporation (9) and (9) and RANDCorporation the ISPOR, ABPI, RAND, and IMI-GetReal definitions for real-world data(RWD). term RWD developed by RWDterm by developed the For For the , 2005 to December 31 December to , 2005 st Term & Source Term 2007 ISPOR, ABPI, 2011 IMI-GetReal, 2015) RAND, 2014 RAND, Table 1- METHODS inclusion and exclusion of articles was resolved by consensus amongst amongst consensus by of articles resolved and exclusion inclusion was Search functions on stakeholder websites were used when available, using terms such as: such as: terms using used when available, were functionsSearch on stakeholder websites world outcome”, “real , data” effectiveness “clinical evidence”, world “real data”, world “real from both scientific results Search effectiveness”. “relative or effectiveness” “comparative pre- to WG) according & (AM 2 authors by screened independently were literature and grey ii in criteria exclusion and inclusion defined (Table in Figure i in in Figure groups were consulted (see Table i in Table (see consulted were groups for for consider to generate RWD. Subsequently, Subsequently, RWD. generate to consider Two qualitative methods were used to compile data from relevant stakeholders; stakeholders; relevant from data compile to used were methods qualitative Two 1 Outcomes Research (ISPOR) (7), (ISPOR) (7), Research Outcomes review and stakeholder interviews. Data compilation from eight stakeholder groups was was groups stakeholder eight from compilation Data interviews. and stakeholder review agencies, (HTA) Assessment Technology performed, namely: Health industry, regulatory agencies, academia, healthcare providers, healthcare insurers/payers, insurers/payers, healthcare providers, healthcare academia, regulatory agencies, industry, ISPOR, on, RWD (e.g. research or commissioning using, and initiatives organisations patient Institute (PCORI)). Research Outcomes Patient-Centered (8), (8),

What is RWD? 2 What is RWD? 27 2 2 data data the the larger study study larger Appendix for for Appendix a preference for for preference a the freedom to invite invite to freedom the 8 previously-mentioned 8 previously-mentioned the 2 authors. type of document, definition(s) typedefinition(s) of document, the scope of questions posed during during posed questions of scope literature review and transcripts of of transcripts and review literature categories for RWD for and definitions categories the the interviews. Tailored questionnaires were were questionnaires Tailored interviews. the the selected documents. Any discrepancies in in discrepancies Any selected documents. the context of stakeholder-specific activities of stakeholder-specific and context coding analysis using MaxQDA software 11.0. software 11.0. MaxQDA using analysis coding the a the term real-world data (RWD)? (RWD)? data real-world term definition of RWD. However, all questionnaires questionnaires all However, of definition RWD. the interviews were conducted as part conducted interviews were of the COREQ checklist provides guidance for explicit and and explicit for guidance checklist provides COREQ the The The codes of repeating themes were iteratively refined and grouped grouped and refined iteratively were themes repeating of codes consolidated criteria for reporting qualitative studies (COREQ) (COREQ) studies reporting qualitative criteria for consolidated the the standardised collection of data on stakeholders’ definitions of RWD, of definitions RWD, stakeholders’ collectionon data of standardised interview to guide discussions (see Figures ii to iv in iv ii to interviewFigures (see discussions guide to categories generated formed formed generated categories documents selected after screening. Data elements included in included elements Data selected documents after screening. the the The The the grounded theory approach in qualitative research (12), data was iteratively iteratively was (12), data research theory in qualitative grounded approach following three questions: three following validity of stakeholder views, participants were provided provided participantsvalidity views, of stakeholder were the sampling of stakeholders and interview protocols were compared to to compared were interview and of stakeholders sampling protocols the the standardised invitation to participate in semi-structured interviews. In order to participate to to In in semi-structured interviews. order invitation standardised authors’ professional network. All representatives were approached by e-mail by approached were network. representatives All professional authors’ if so, what type what of RWD? if so, What is your understanding of understanding your is What for definition specific RWD? own your provide you Could Is RWD in collected/used routinely a standardised data abstraction form was created in Microsoft Excel and used to locate locate to and used Microsoft in Excel created was abstraction data form standardised

interviews extended beyond interviews extended beyond extracted data were resolved by consensus amongst consensus by resolved extracted were data Data extracted from documents selected from selected documents extracted from Data from With regards to stakeholder interviews, stakeholders from from stakeholders interviews, stakeholder to regards With 1. 2. 3. It is important that note to The The A the included included (11) to ensure good quality. quality. good ensure to (11) the Following Following Subsequently, authors. categories. into assessed by 2 authors (AM & WG) independently to identify repeating themes and tag them them tag and themes repeating identify to independently WG) & (AM authors 2 by assessed between consensus by resolved were created in codes discrepancies Any using codes. stakeholder interviews were subjected to subjected to interviewsstakeholder were data sources they consider to be RWD and study designs they consider to generate RWD. generate RWD be to to consider they they consider study designs and sources data This allowed for for allowed This of RWD provided, and data sources considered as RWD and study designs considered to to RWD as considered designs and study considered sources data and of RWD provided, authors (AM Two observational respectively). studies, databases, claims RWD (e.g. generate &WG) extracted from independently data abstraction form were: author(s), publication year, year, author(s), publication were: abstraction form the selectivelyon seniority based sampled function, and were with groups on RWD organisations. work in their respective within use involved senior representatives and/ websites stakeholder from retrieved was representatives identifying for Information or using participate to agreed who stakeholders to sent and group stakeholder each for developed prior to weeks 2 increase increase comprehensive reporting of qualitative studies employing interviews and focus groups. interviews focus and studies employing reportingqualitative of comprehensive recommendations in in recommendations information in information conducted, Interviews were groups). 3 stakeholders to sent of questionnaires examples further for transcribed and subsequently analysis. recorded and perspectiveson policies on RWD thus (6), colleagues they deemed relevant to take part to in relevant deemed they colleagues

terms terms design design factors factors the the the term “non- term data data de novo data data de novo setting where where setting interpretation interpretation a the investigator may may investigator the the data collection collection data De novo term “non-experimental” “non-experimental” term the conditions and no conditions data collection occurs other than other than collectiondata occurs term “RCT” referred to to referred “RCT” term the the setting wherein wherein setting de novo de novo a following aspects: treatment assignment, assignment, aspects: treatment following investigator cannot alter any of any alter cannot investigator authors are aware that several non-identical non-identical several that aware are authors scheme generated was: generated scheme the non-experimental setting) above) the final coding scheme developed was discussed discussed was developed coding scheme final the non-interventional/ non-controlled setting (i.e. setting (i.e. non-controlled non-interventional/ non-RCT setting (i.e. all health data except that that non-RCTexcept data health all (i.e. setting a non-experimental in (i.e. setting The The a the a a category developed here depended on definitions depended on definitions category here developed The The pre-established study protocol) conventional phase III RCT phase setting) conventional a non-interventional/ non-controlled setting), setting), non-controlled non-interventional/ term “RCT” corresponds to several sources in scientific in scientific sources several to corresponds “RCT” term non-interventional/ non-controlled manner. Similarly, Similarly, manner. non-controlled non-interventional/ a non-RCT setting. However, not all data collected in in collected data all not non-RCTHowever, setting. the a non-RCT setting), a a a the non-interventional/non-controlled setting is theoretically theoretically is setting non-interventional/non-controlled study and as such no a setting in which setting setting of setting a the non-experimental setting is theoretically equivalent to that from from that to non-experimental equivalent is theoretically setting the a compiled data. compiled the investigator has no control over any of any over control has no investigator Category in collected 3: Data Category 4: Other of none (i.e. registries) claims databases, (e.g. Category sources A: Data observational RWD studies, (e.g. generate that Category designs B: Study clinical trials) pragmatic Category 1: Data collected in Categorycollected Data 1: in collected Category in collected 2: Data data collected without interference with treatment assignment, and/or patient patient and/or assignment, with treatment without interference collected data and/or selection study population) of monitoring/follow-up, on based collection occurs the Ű Ű Ű Ű Ű Ű Ű Ű Ű of RWDCategories sources Ű Ű Ű Categories of RWD of Categories definitions term “non-interventional” for “non-interventional” term conventional phase III RCT which involves: implementation of inclusion/exclusion inclusion/exclusion of implementation RCT phase III which involves: conventional For category in collected 2 (data For • • It is important to note that categories 1 to 3 above are not mutually exclusive. For For exclusive. not mutually are It 3 above 1 to is important categories that note to For category in collected 3 (data For a the non-RCT in is collected setting of and arms, treatment of subjects different trial subjects, randomisation to criteria for trial subjects for implicit and procedures and follow-up monitoring consistent of interpretation This collection. amongst all authors to ensure consensus. consensus. ensure to authors all amongst RWD sources for subsequent analyses. analyses. subsequent for RWD sources data collected in routine clinical practice. This interpretation of interpretation This practice. clinical in routine collected data “non-interventional/ non-controlled” referred to to referred non-controlled” “non-interventional/ For category in collected 1 (data For (1;2;10;13). literature of or more, with one, interfere able to not be While in this setting. occur not, or may may, intervention in clinical to define (10;14;15), trials exist already definitions of monitoring and follow-up procedures or inclusion/exclusion criteria. or inclusion/exclusion procedures and follow-up monitoring a example, data collected in in collected data example, experimental” referred to to referred experimental” observedor conditions in (10;13;15). literature scientific in sources several to corresponds in collected data to equivalent available from from available all data collected in collected all data

What is RWD? 2 What is RWD? 29 2

total total list of list of and B) and B) second second A the a inclusion inclusion the the Appendix for for Appendix 38 definitions identified identified definitions 38 38 (11%) were category 4 were 38 (11%) term at all and all at term the 4 definition categories (1 to to (1 categories definition 4 the the 2 sources categories ( categories 2 sources 20 interviews included at least 20 interviews least at included Appendix for PRISMA diagrams diagrams PRISMA for Appendix the the the the the grey literature search yielded 66 hits. hits. 66 yielded search literature grey following reasons: document did not did not document reasons: following the the document formats outlined in outlined formats document three stakeholder groups. stakeholder three the the categories which have implications on defining RWD. on defining implications RWD. which have categories discussion sectiondiscussion below. 8 stakeholder groups agreed to participate (see Table iv in iv in Table participate to (see agreed groups 8 stakeholder definition for RWD. Twenty of Twenty for definition RWD. the the a the list of interviews of Eight conducted). a concept of “real-world trials”, rather than RWD. For an overviewan of For RWD. than rather trials”, “real-world of concept number of definitions per definition category was recorded. Additionally, Additionally, category recorded. per definition of definitions number was categories. number and type of sources per category was recorded. per category recorded. and typenumber was of sources the PubMed search yielded 496 hits while hits 496 yielded search PubMed The The the The The the 562 total hits, 509 were excluded due to to due excluded 509 were hits, 562 total summary/abstract (n=1) (see Figures v and vi in summary/abstract Figures (n=1) (see sub-analysis was performed for definitions provided by 3 stakeholder groups which which groups by 3 stakeholder sub-analysis performed was definitions provided for a pharmaceutical industry, regulatory agencies and HTA agencies. Definitions identified Definitionsidentified agencies. regulatoryHTA and agencies industry, pharmaceutical 38 (13%) definitions were category 3 definitions. Four of Four categorywere 3 definitions. definitions 38 (13%) Appendix for for Appendix Twenty stakeholders from from stakeholders Twenty In total, 20 definitions were identified in literature documents and 18 definitions were were 18 definitions and documents in literature identified were 20 definitions In total, For category 2 definitions, it was not always clearly stated what authors and stakeholders authors and stakeholders what clearly stated was not always categoryit definitions, 2 For Each RWD source identified was classified into one of one into was classified identified Each RWD source A the non-interventional/ non-controlled setting but not vice versa. Therefore, there are subtle subtle are there Therefore, versa. but not vice setting non-controlled non-interventional/ RESULTS ANALYSIS included documents). included the This will be elaborated upon in in upon be elaborated will This indicated they cannot provide provide cannot they indicated of Five categorywere category 2 definitions. 38 (24%) of Nine 1 definitions. were (53%) the or 3, 1 to categories in one of fit to too general definitions these either provided definitions; defined had provided in interviews. No definitions were identified in 33 documents nor provided in provided nor 33 documents in identified were definitions No interviews. in provided with familiar be to not interviews;2 interviewee1 stated of document in- and exclusion from PubMed and grey literature searches, respectively). respectively). searches, literature and grey PubMed from in- and exclusion of document iii in Table selected (see were 53 documents Eventually, per stakeholder. 3 representatives 2 included and per stakeholder, 2 representatives focus on RWD use in pharmaceutical drug development, regulation or HTA (n=490), was not not was (n=490), HTA or regulation on RWD development, drug focus pharmaceutical use in one of in not was in English (n=7), published (n=5) or comprised synthesis or evidence analysis solely on data focused criteria (n=6), only Initially, Initially, Each RWD definition identified was classified into one of one into RWDEach classified was identified definition a between differences qualitative 4) created. 4) created. and within, differences to highlight analysed qualitatively category in each definitions were between, created. created. perceived as non-interventional or non-controlled settings. According to some, non- some, to According settings. or non-controlled as non-interventional perceived number of definitions identified per category, see Figure 1. For examples of definitions of definitions examples For 1. Figure see category, per identified of definitions number 2. Table see interviews, and documents per categoryliterature identified from Of were compared both within and between between within and both compared were are directly involved with RWD collection or appraisal to determine drug effectiveness: RWD with effectiveness: drug determine to collection or appraisal directly involved are the

RCT a definition definition citation for for citation the fact that both fact both that the Other the researcher not interfering with not interfering with researcher setting the experimental implementation of any inclusion or or inclusion of any implementation ‐ 3 regulatory stakeholders interviewed, 3 regulatory interviewed, stakeholders Non the the Category

frequency of their mention, see Table 3. Table frequency see mention, of their the setting different data sources and study designs retrieved retrieved and study designs sources data different Definition the interventional/ ‐ controlled ‐ criterion for intervention, stating that RWD that stating intervention, thus should criterion for list of list of a Non a total number of definitions classified under each of each under classified of definitions number total the setting Non

setting where no randomisation of patients occurs (see occurs patients of no randomisation where setting RCT ‐ a Non non-RCT category (i.e. setting 1). Of a population in clinical practice without in clinical population - Overview of 5 RWD data sources cited most in literature documents and interviews were: and interviews documents were: most in literature cited 5 RWD sources data

5 0

25 20 15 10 the Frequency 3 study designs mentioned on more than 5 occasions were: observational (22 studies were: occasions 5 than more on mentioned study designs 3 selection of study population. One stakeholder believed that RWD that selection collected be should believed One stakeholder population. of study The The All 4 pharmaceutical industry 4 pharmaceutical All interviewed stakeholders defined as health data RWD selection of criteria those than stringent on less based albeit study population of categories created. categories Figure 1 Figure (see citations for HTA Agency B and HTA Agency C in Table 2). Another stakeholder focussed focussed stakeholder 2). Another Table Agency C in Agencyand HTA B HTA for (see citations as randomisation on patient in be collected exclusion criteria for selection of patients, while another implied that there might be be might there implied that while another selection patients, of criteria for exclusion a interventional data collection related specifically to specifically collection data interventional related 6 interviews) (5 studies and post-marketing documents, (16 6 interviews), PCT’s documents, 2 interviews). For documents, registries (18 documents, 7 interviews), Electronic Health Records (EHR’s) (16 documents, documents, (16 (EHR’s) 7 interviews), Electronic Health Records documents, (18 registries (6 data 4 interviews), administrative 6 interviews), documents, (12 databases claims Meanwhile, documents). (10 outcomes interviews) 4 patient-reported and documents, the treatment assignment and patient management and follow-up (see citations for Pleil et et Pleil for (see citations follow-up and management and patient assignment treatment Others 2). aspect on another namely of intervention, focussed Table B in Initiative and al. the from trials clinical (PCT’s) pragmatic cited stakeholders other Meanwhile, 2). Table in B Initiative despite of definitions within their RWD, trials simple (LST’s) and large arms. treatment between of patients randomisation involve study designs as well as and interviews, documents from collected in collected

What is RWD? 2

What is RWD? 31 2

Agency A Agency

data.” – Regulatory Regulatory – data.” the of aspect world” “real the

research protocol endanger endanger protocol research a of because conducted

driven way. Any additional procedures that are are that procedures additional Any way. driven in routine care.” (30) care.” routine in

protocol- a in collected not is it that means This under experimental conditions, but data generated generated data but conditions, experimental under

“RWD is data collected from daily clinical practice. practice. clinical daily from collected data is “RWD “With RWD, we mean data that are not collected collected not are that data mean we RWD, “With setting non-experimental a in collected Data

random assignment.” – Initiative B B Initiative – assignment.” random

lack of of lack the imply generally will settings controlled

of healthcare in non-controlled settings. Non- settings. non-controlled in healthcare of

delivery delivery the from generated is that data is “RWD

controlled condition.” – HTA Agency C C Agency HTA – condition.” controlled the

population. So it is something that was not done in in done not was that something is it So population.

drug, or is actually derived from that that from derived actually is or drug, the receiving be observing outcomes” (29) outcomes” observing

[RWD] more closely matches matches closely more [RWD] population who will will who population the medical management of of management medical the patient beyond beyond patient the

step that is not done in in done not is that step a as it of think we So RCT... RCT... a treatment and does not, or cannot, control control cannot, or not, does and treatment

scientific process. process. scientific the about is RWD term the us “To does not, or cannot, control who gets what what gets who control cannot, or not, does

Agency B B Agency (treatment) decision is made where where made is decision (treatment) researcher researcher the

no specific inclusion or exclusion criteria.” – HTA HTA – criteria.” exclusion or inclusion specific no effects based on what happens after after happens what on based effects prescriptive prescriptive a controlled setting controlled

“[RWD is] observational data without blinding and and blinding without data observational is] “[RWD non-interventional/ non- non-interventional/ a in collected Data “In general, real-world data are observations of of observations are data real-world general, “In

Industry A Industry

is more observational in nature.” – Pharmaceutical Pharmaceutical – nature.” in observational more is

RCTs.” (7) RCTs.” are acting as they are in normal clinical practice. It It practice. clinical normal in are they as acting are

conventional (an RCT). So that that So RCT). (an patients patients the and physicians the

decision-making that are not collected in in collected not are that decision-making that is not collected as part of of part as collected not is that strict clinical trial trial clinical strict a

“RWD to us means any health record information information record health any means us to “RWD Data collected in in collected Data non-RCT setting non-RCT a “We settled on on settled “We definition that reflects data used for used data reflects that definition a

Category of Definition of Category Citations from literature documents literature from Citations Citations from stakeholder interviews stakeholder from Citations

4 definition categories created. categories definition 4 the under classified been have which interviews stakeholder and documents literature from definitions of Examples - 2 Table

real world clinical practice…” (2) practice…” clinical world real

benefit or risk is taking place in in place taking is risk or benefit setting closer to to closer setting a

opposed to ‘efficacy’, implying that assessment of of assessment that implying ‘efficacy’, to opposed

stated objective includes includes objective stated term ‘effectiveness’ as as ‘effectiveness’ term the

or prospectively.” – Patient Organisation A Organisation Patient – prospectively.” or few aspects from conventional phase 3 trials. Their Their trials. 3 phase conventional from aspects few a

health professionals. It can be done retrospectively retrospectively done be can It professionals. health to properly randomised trials that differ only in in only differ that trials randomised properly to

setting where data is collected by by collected is data where setting a or time, over studies or NROTs (stand-alone or follow-up of RCTs’ RCTs’ of follow-up or (stand-alone NROTs or studies

pragmatic trial, collecting for example evidence evidence example for collecting trial, pragmatic a of large design spectrum ranging from uncontrolled uncontrolled from ranging spectrum design large a

collected in in collected real world. This can be in in be can This world. real the setting setting the conducting literature searches, appear to include include to appear searches, literature conducting

“RWD are data about effectiveness of treatments treatments of effectiveness about data are “RWD Other “RWT’s are heretofore ill-defined as as ill-defined heretofore are “RWT’s class and, when when and, class a

Category of Definition of Category Citations from literature documents literature from Citations Citations from stakeholder interviews stakeholder from Citations

- continued - 2 Table

What is RWD? 2 What is RWD? 33 2 5 HTA 5 HTA Total 25 22 16 10 10 6 6 5 3 3 3 2 1 1 Total 28 22 7 3 1 1 the remaining 2 2 remaining effectiveness effectiveness sub-analysis the sub-analysis the the result, disparity arises result, a data sources considered considered sources data non-RCT (category setting Interviews 8 6 4 4 - 2 3 3 2 2 - - - - Interviews 6 6 2 - - - the use of data from non-RCT’s non-RCT’s from use of data a value of RWD is contrasted by by of RWDvalue contrasted is the the 12 stakeholders in the 12 stakeholders Literature 17 16 12 6 10 4 3 2 1 1 3 2 2 1 Literature 22 16 5 3 1 1 non-RCT (category setting 1), while non-experimental setting (category Of 3). definition of what of what definition RWD is, a a the value of RWDvalue on in enriching evidence the non-interventional/non-controlled setting (category 2) and 1 was (category setting non-interventional/non-controlled 1 was 2) and a definition. Importantly, only 4 of only 4 of definition. Importantly, a - List - and frequency of occurrence of real-world data sources (RWD) and study types that generate lower degree of consensus on what RWD precisely constitutes. As As RWD constitutes. what on consensus of precisely degree lower Data Sources Data Registries Electronic Health Records (EHR’s) Electronic Records Health Claims databases Claims Administrative databases Administrative Patient-Reported Outcomes (PRO’s) Outcomes Patient-Reported Health SurveysHealth Hospital data Hospital Electronic data health Clinicians Payers Social media Patient charts Patient Pharmacy data Clinical databases Clinical Study Designs Study Observational studies Pragmatic Clinical Tirals (PCT’s) Tirals Clinical Pragmatic Post-marketing studies Post-marketing Supplements to RCT’s to Supplements Drug utilisation studies utilisation Drug Large Simple Trials (LST’s) Trials Simple Large Table3 interviews. stakeholder and documents literature from retrieved RWD DISCUSSION 1 defined 1 defined collected in as data RWD a (16-18), documents produced by regulatory agencies on post-marketing effectiveness and and effectiveness regulatory on post-marketing agencies by produced documents (16-18), industry of RWDof use their to stakeholders referral as in well as (19;20), studies safety on consensus However, (8;21;22). product development Stakeholders’ perception of perception Stakeholders’ of medications has been steadily increasing. This can be observed in guidelines of HTA can be observedThis HTA of in guidelines increasing. been steadily has of medications sections include on conventionally which now agencies defined defined collected in RWD as data amongst stakeholders regarding regarding stakeholders amongst stakeholders interviewed, 2 defined 2 defined collected interviewed, stakeholders in as data RWD as RWD and study designs that generate RWD. as RWD generate that designs and study 1), 2 as data collected in collected 1), 2 as data unable to provide provide to unable had an official, institutional definition for RWD. for definition institutional official, had an other other protocol protocol the discussion discussion intervention intervention the The The perspective of perspective of 3 categories of of 3 categories same category, category, same the highly-controlled, highly-controlled, the the the the spectrum (see Figure 2). 2). spectrum Figure (see the implementation of selection of implementation implementation of additional of additional implementation the the seminal paper by Garrison by seminal paper et al (7) clinical trial study design classify as an as an classify trialclinical study design the a source for RWD for depends on mainly source majority of cases (20 of 38 cases), this perception majority this perception of 38 cases), of cases (20 a qualitative differences between between differences qualitative the the 5 most common types, stark controversy exists on whether on whether exists stark types, controversy 5 most common RWD sources most mentioned, respectively. respectively. mentioned, most RWD sources European Commission, for example, intervention is defined is defined intervention example, for Commission, European RCT lies (least representative of RWD) on and RCT (least representative lies RCT in the the the a spectrum of data exists where on one end, on one end, spectrum where exists of data the previous paragraphs notwithstanding, this would imply that some some imply that this would notwithstanding, paragraphs previous a the registry qualifies as form of additional quality-of-life of additional patients? included surveysform for a study population would not qualify as an intervention, according to to according an intervention, as not qualify would study population previous points in mind, it would seem that from from seem that it would in mind, points previous setting of setting the category of RWD as data from non-interventional/ non-controlled setting categorysetting non-controlled non-interventional/ RWD of from data as the the the the non-experimental setting of EHR’s, where no intervention is implemented by by is implemented no intervention where non-experimental setting of EHR’s, researcher’s control of treatment assignment or or assignment of treatment control researcher’s the study population. This demonstrates, moreover, that some stakeholders may have have may stakeholders some that moreover, demonstrates, This study population. most recurrent example, featured in only 22 of 52 literature documents and 6 of 19 of 19 and 6 documents 52 literature of 22 in only featured example, recurrent most investigator and no extra data is collected other than that from routine clinical practice clinical practice routine from that than other collected is no extra and data investigator Commission’s definition. Commission’s Moreover, different stakeholders cited data sources and study designs interchangeably interchangeably designs study and sources data cited stakeholders different Moreover, Results presented above indicate that observational studies, registries, EHR’s, PCT’s PCT’s EHR’s, observational registries, that studies, indicate above Results presented Bearing Although results demonstrate that RWD is perceived as health data that is not not is that health data as RWD that perceived is demonstrate results Although the (most representative of RWD). Other data sources and study designs such as PCT´s, of RWD). designs study and Othersources data (most representative both ends of observational between fall studies and registries the as RWD. Although this is not theoretically incorrect, it can lead to disparities between disparities between lead to it can incorrect, not theoretically this is Although as RWD. example, For qualify as RWD. of data sources when discussing which stakeholders various or as interventional observational regarded be design on their depending may studies of RWD. representative or less, thus more, stakeholders different by non-interventional whether Similarly, and other literature documents included supplements to RCT’s in their list of RWD their list of in RCT’s types, to supplements included documents literature and other as RWD. them interviews in they do not consider that stated explicitly other stakeholders RWD constitute also weak. is sources on what consensus that argued be it may Therefore, and claims databases were were databases and claims diagnostic or monitoring procedures (14). This implies that that implies This (14). procedures or monitoring diagnostic criteria for the in mentioned points Looking beyond interviews. as RWD; classify RCT’s although to supplements as RWD qualify and study designs determined is subsequently sources such data Whether degree of consensus exists regarding these RWD sources. However, observational studies, observational studies, However, these RWD sources. regarding exists of consensus degree the as RWD definitions, data sources and study designs identified in literature documents and and documents in literature identified and study designs sources RWD data definitions, interviews, stakeholder setting of randomised used for data collection: is data collection purely observational of routine care, or is there or is there observational collection: care, collection data is data purely of routine used for in intervention end an incorrect understanding of which aspects which of understanding an incorrect of is not unanimous. In to addition is not unanimous. of definitions between RWD critical disparities emerge definitions, in namely whether on disagreed, and sometimes (category differed, 2). Stakeholders selection or of follow-up, and monitoring patient assignment, pertains treatment to the intervention. According to to According intervention. collected in collected

What is RWD? 2

What is RWD? 35 2

- Data spectrum in relation to RWD definition categories. definition RWD to relation in spectrum Data - 2 Figure

EHR = Electronic Health Record. Health Electronic = EHR

Trial; PAES: Post-Authorisation Efficacy Study; PASS = Post-Authorisation Safety Studies; Obs. Studies Obs. Studies; Safety Post-Authorisation = PASS Study; Efficacy Post-Authorisation PAES: Trial; = Observational studies; studies; Observational =

Legend: RWD = Real-World Data; RCT = Randomised con Randomised = RCT Data; Real-World = RWD Legend: trolled Clinical Trial; LST = Large Simple Trial; PCT = Pragmatic Clinical Clinical Pragmatic = PCT Trial; Simple Large = LST Trial; Clinical trolled

RWD Sources RWD

Social Media Social Highly-controlled setting of RCT

PAES

PASS Patient Charts Patient

Registries

databases

Claims Claims

Supplements to RCT to Supplements

Obs. Studies Obs.

S PCT LST RCT

Health surveys Health EHR

RWD RWD RWD

Non- Non- Non-

setting controlled setting controlled setting

3- Non-experimental Non-experimental 3- 2- Non-interventional/ Non- Non-interventional/ 2- 1- Non-RCT Non-RCT 1-

Non-controlled setting of routine clinical practice clinical routine of setting Non-controlled

Definition Category Adopted Category Definition

Figure 2 – RWD spectrum in relation to definition c definition to relation in spectrum RWD – 2 Figure ategories. starting types of types of non-RCT choice of of choice a definitions definitions the the the the term Real-World Real-World term the conventional RCT. To To conventional RCT. a RAND corporation were were RAND corporation past years by dedicated dedicated by past years the starting point for discussions starting discussions for point the a types of data and study designs typesdesigns study and data of ISPOR, ABPI and RAND versions RAND and ABPI ISPOR, versions setting of setting significant number of stakeholders of stakeholders number significant the context of drug development, drug drug of drug development, context the a ABPI and ABPI and the the dedicated taskforce, formed formed taskforce, dedicated the a consortium introduced consortium introduced respective institutions through internal rounds rounds internal through institutions respective comprehensive definition was agreed upon by by upon agreed was definition comprehensive the a the non-interventional/non-controlled setting (category setting non-interventional/non-controlled a types of study designs which generate RWD, thereby thereby RWD, generate typeswhich designs study of wider community were able to provide their opinions their opinions provide able to wider community were diagrammatic representation). Therefore, Therefore, diagrammatic representation). a the the domains of information RWD and information inform of can domains the non-experimental setting (category 3). a pharmaceutical industry, regulatory agencies and HTA agencies did did agencies regulatory agencies and HTA industry, pharmaceutical consortium, as well as external procedures of public consultation, consultation, of public consortium, external as procedures as well recent definition developed by IMI-GetReal rounds developed definition underwent internal recent the the ISPOR definition, developed by developed ISPOR definition, seminal examples being provided by ISPOR (7), ABPI (8), RAND (7), ISPOR and IMI- (9) by provided being examples seminal The The wider community to achieve consensus on what RWD constitutes. This is is This RWD what on consensus constitutes. achieve wider community to The The use of RWD in decision-making within the the concept of RWD, of RWD, concept the proposed definition. Eventually, Eventually, definition. proposed categories stakeholders adopt for defining RWD. These categories, set in order of least of least in order set These categories, defining for adopt stakeholders RWD. categories the authors’ knowledge, definitions proposed by proposed definitions knowledge, authors’ Definitions developed by these institutions may provide provide institutions may these by Definitions developed If one were to adopt category 1, all data sources/ study designs other than RCT than other study designs adopt category to sources/ all data 1, would If were one over developed been for definitions have RWD Several the the setting (categorysetting in collected data 1), in collected and data 2), representative of RWD to most representative of RWD, are: all data collected in collected all data are: RWD, of of RWD representative most to representative by interviewed from interviewed from Studies (RWS) on Studies light shed to (10). sources data from these distinguishing amongst to attempt particularly mandates with differing stakeholders important different when discuss RWD developed definitions However, products. of pharmaceutical and HTA regulation and documents in literature cited rarely RAND ABPI, ISPOR, and IMI-GetReal,by were definition own their proposed either documents several Moreover, interviews. stakeholder In (21;23-26). this, to addition or lacked one entirely qualify as RWD; from PCT’s to claims databases and EHR’s. If one were to adopt category to If were one and EHR’s. databases claims to as RWD;qualify PCT’s from assignment, with treatment interfere not do observationalonly 2, protocols whose studies would This selection as RWD qualify or study population would sources. follow-up, patient were if one Finally, and some observational designs. study LST’s PCT’s, exclude effectively qualify would EHR’s and claims databases such as category adopt to sources data 3, only for 2 as RWD see Figure (please GetReal (10). categories for defining RWD has direct implications for defining directfor has RWD implications categories RWD. classify as subsequently that taskforces, not have an official, institutional definition of RWD nor had adopted any of of definition institutional an official, adopted RWD nor had not have point for subsequent ones by ABPI, RAND corporation and IMI-GetRealand RAND ABPI, corporation by ones succinctly and subsequent for point outside collected data to RWD that referred stated Moreover, RWD. which constitute data mentioned above. mentioned the on: on from included elements which stakeholders multiple developed by similar taskforces within within taskforces similar by developed of discussions. of discussions. of review within within of review whereby all stakeholders from from stakeholders all whereby

What is RWD? 2 What is RWD? 37 2 inin- the quality quality concept concept adopted adopted the the the representative representative a aims of this article, the definition of definition a selection of stakeholders the coding analysis were discussed discussed were analysis coding the aim was not to compare compare not to was aim categories created. For example, gene gene example, For created. categories opportunity to invite colleagues they they opportunity colleagues invite to the the the stakeholder’s view on RWD, RWD, on view stakeholder’s consolidated criteria for reporting qualitative reporting qualitative criteria for consolidated a the interviews, and by interviewing more than 1 person interviewing 1 person than and by interviews, more sampling of stakeholders and interview protocols interview and stakeholders of sampling protocols focus of this article was on providing an overview of overview an articlethis of of focus providing on was 20 interviews included at least 2 representatives per per 20 interviews representatives 2 least included at the the the grey literature search and interviews helped ensure that that and interviews helped ensure search literature grey the the full perspective of extent of familiarity and experience of different stakeholders with with stakeholders of familiarityextent of different and experience literature review performed on academic and grey literature, performed literature, review grey and on academic literature the the the authors could triangulate data on definitions of definitions on well- data triangulate based on RWD could authors two degree of variance in length and level of detail provided in different different in provided of detail length and level in of variance degree the The The literature review and stakeholder interviews. This provided multiple sources sources multiple provided This interviews. stakeholder and review literature a non-RCT setting”) may present an inherent limitation when trying to conceptualise when trying limitation conceptualise an inherent to present non-RCT may setting”) term. term. placement of certain data sources within of certainplacement sources data Definitions provided in documents and interviews varied in length and degree of detail detail of and interviewsvariedin documents length and degree in Definitions provided Criteria used for defining RCT’s to create category 1 of collected to create (“Data RWD definitions defining RCT’s used for Criteria Two methods were used to compile data needed to achieve achieve to needed data compile used to were methods Two a comprehensive view of definitions currently used by relevant stakeholders was available. was stakeholders relevant by used currently of definitions view comprehensive per institute. Eventually, 8 of of 8 Eventually, per institute. thus implying that that thus implying stakeholder, and 2 of 20 included 3 representatives. 3 included of 20 and 2 stakeholder, Instead, provided. definitions of example Another trials. not non-interventional also they are Meanwhile, of definition RCT’s. definitions was not analysed in this article, in not analysed was since definitions sample within an organisation should be interviewed. Therefore, it can be argued that that argued it can be Therefore, should be interviewed. an organisation within sample perspectives stakeholder gather to insufficient interviewsstakeholder were conducted selectively stakeholders, sampling by this for account to attempted We comprehensively. approached stakeholders offering explicitly participate to in deemed relevant RWD varied. For example, while some were quite detailed, citing detailed, quite were some while example, RWDFor varied. the Limitations capture to In order in and several data sources, other stakeholders indicated that they were unfamiliar with with unfamiliar they were that indicated other stakeholders sources, data and several the them. between differences of definitions available and qualitative RWD single-arm often as open-label, conducted trials, under do not fall therapy trials, Strengths and compilation practice during data research good ensure to taken were steps Several Within analysis. and exclusion of documents and subsequent data extraction from selected documents extraction data selected and subsequent documents of documents from and exclusion consensus. by resolved 2 authors and all discrepancies by independently conducted were interviews, stakeholder Within were compared to recommendations in recommendations to compared were extracted data of analysis coding Moreover, good quality. ensure to (11) (COREQ) studies and interview documents performed was transcripts 2 by independently literature from RWD for developed categories Finally, consensus. by resolved and all discrepancies authors of results of RWD based on sources and definitions namely which from Moreover, methods. research acknowledged qualitative from 8 diverse groups for for groups diverse 8 from a amongst all authors to ensure consensus. ensure to all authors amongst wide wide a product product types of types of the the authors of this of this authors idea that idea that widest audience, audience, widest The The the the criteria for RCT’s adopted adopted RCT’s criteria for the data compiled from literature literature from compiled data RCT, this perception was not not was perception this RCT, dichotomous attitude amongst amongst attitude dichotomous promise RWD brings, disparities RWD disparities promise brings, a a the collection or use of RWD for drug collection use of RWD or drug for the RCT lies (least representative of RWD) of RWD) RCT representative (least lies greatest value to to value greatest spectrum of data exists where on one on one spectrum where exists of data the spectrum as qualifies on depends RWD a wider community to do so. communitywider to the a the the context of context fact that most documents and stakeholders fact and stakeholders documents most that types of data sources considered as RWDas and considered typessources data of question whether health data originating from from originating data health whether question the the the common understanding amongst stakeholders of of amongst stakeholders understanding common the potential value of RWD value throughout in potential a the types of information domains it may inform, inform, domains it may types information of grounded theory grounded approach, the non-experimental setting of EHR’s, where no intervention is is intervention no where non-experimentalof EHR’s, setting the other hand, other hand, dichotomous representation and have subsequently developed developed subsequently and have representation dichotomous difference between RCT’s and “non-RCT’s”. “non-RCT’s”. and between RCT’s difference a final example relates to PCT’s and LST’s; such trial designs feature feature trial such designs and LST’s; to PCT’s relates example final the the A the investigator and no extra data is collected other than that from routine routine from that other than is collected and no extra data investigator the data spectrum demonstrated in figure 2 to re-assert to 2 in figure spectrumdata demonstrated a significant number of authors and stakeholders do not have an official, an official, do not have stakeholders authors and of number significant perspective of RWD definitions, data sources and study designs identified in identified designs study and sources perspectiveRWDof data definitions, other end other a the highly-controlled, randomised setting of of setting randomised highly-controlled, the the notion of notion of In order to ensure that future work involving involving work future that ensure to In order From From good starting amongst discussions for point CONCLUSION documents and stakeholder interviews. This alludes to to alludes This interviews. stakeholder and documents towards stakeholders relates to open-label extension of RCT’s which conventionally precede long-term post- long-term precede conventionally which extension open-label RCT’s of to relates nor non- RCT’s neither extensionsopen-label Such studies are studies. authorisation studies. interventional measures criteriaand outcome inclusion/exclusion broader implement yet randomisation nor non-interventional RCT’s neither they are such, As practice. clinical for relevant more with In accordance studies. to develop category 1 were directly elucidated from from directly elucidated category develop 1 were to RCT both within spectrumis generated data of non-RCT and settings settings. the and on and defined defined collected in not as data RWD Stakeholders’ acknowledgement of Stakeholders’ Despite RWD. generating study designs lifecycle is increasing. However, despite awareness of awareness despite However, is increasing. lifecycle RWDis, what precisely regarding persist paper do not favour such such do not favour paper clinical practice (most representative of RWD). All stakeholders concede that data generated generated data that RWD). of concede stakeholders All practiceclinical (most representative On is not RWD. RCT’s by definitions. in their stakeholders varyingby adopted categories implemented by by implemented developing towards move should we RWDwhat means, precisely end, end, delivers HTA and regulation drug development, unanimous. Other definitions identified differed and, often, contradicted one another. another. contradicted one and, often, Other differed unanimous. identified definitions Moreover, by ISPOR, ABPI, developed definitions adopted for definition institutional RWD nor have RANDor IMI-GetReal. interviews, stakeholder and documents literature other data sources or study designs within such within such study designs or sources other data data sources which qualify as RWD and study designs which generate RWD. Definitions Definitions RWD. which qualify as RWD which generate study designs and sources data ABPI, RAND as ISPOR, such IMI-GetReal and initiatives provide previous by developed a

What is RWD? 2 What is RWD? 39 2

approximation of of approximation the Member States relating to to relating Member States dictionary of epidemiology. dictionary epidemiology. of the A European Parliament and of and of Parliament European 32-item checklist for interviews checklist interviews 32-item for a European Union 2001;Available Union 2001;Available European Use of Observation Data to Inform of Observation Inform Use to Data the conduct of clinical trials on medicinal conduct of clinical trials on medicinal Council on Council laws, regulations and administrative administrative and regulations laws, the implementation of good clinical practice practice of good clinical implementation The The the and focus groups. Int J Qual Health Int J Qual Health groups. and focus 2007;19(6):349-57. Care Estimates of Treatment Effectiveness Effectiveness Treatment of Estimates Methods for Appraisal: Technology in 2015. Data. Individual Patient Comparative the the of provisions AJ. Wailoo Manca MH, A, R, Alava Faria Support Document Technical DSU NICE 17: MakadyGlossary A. GetReal. of definitions from: 2015;Available terms. of common http://www.imi-getreal.eu/Portals/1/ URL: Documents/Publications/D1.3%20 GetReal%20Glossary.pdf 2001/20/ Directive Commission. European ec of the in Official use. human Journal products for of http://ec.europa.eu/health/ URL: from: files/eudralex/vol-1/dir_2001_20/ dir_2001_20_en.pdf Epidemiological Last JM, International Association. 2001. Press; Univ Oxford ed. 141 voor Leideraad G.O. Delwel Uitkomstenonderzoek. 2008. Nederland. Zorginstituut études post- Les Santé. de Autorité Haute santé de inscriptiontechnologies les sur et médicaux dispositifs, (médicaments, http:// URL: from: actes). 2011;Available tinyurl.com/orw8cw2 Makady IMI-GetRealA. Goettsch Glossary. N, Hummel P, Jonsson A., Willemsen W., IMI-GetReal. 2015. C, editors. Nordon Consolidated J. Craig A, Sainsbury P, Tong research reporting qualitative for criteria (COREQ): theory Grounded A. JM, Strauss Corbin and evaluative canons, Procedures, research: criteria.Qual Sociol 1990;13(1):3-21.

16. 17. 18. 15. 12. 13. 14. 10. 11. Real World Real World Data the efficacy-effectiveness gap: Development and Use of of and Use Development the the practical guide. 2011;Available from: 2011;Available practical guide. A Leveraging of Real-World Data to to Data of Real-World Leveraging ISPOR Real World Data Task Force Force Task Data World ISPOR Real regulator’s perspective on addressing perspective on addressing regulator’s URL: http://www.abpi.org.uk/our-work/library/ URL: guidelines/Pages/real-world-data.aspx Policy Landscape in Europe. Santa Monica, Santa Europe. in Landscape Policy 2014. http://www. CA: RAND Corporation, rand.org/pubs/research_reports/RR544.html. Celine M, Robin E, Horvath V et al. Health and Health V et al. Horvath Robin M, E, Celine Assessing Healthcare: ABPI. Demonstrating Value with Real World World Real with Value Demonstrating ABPI. Data: Medicines. Value Health 2015;18(1):127-30. Health Value Medicines. Erickson P, PJ, Neumann Garrison LP, World Real Using Mullins CD. Marshall D, Decisions: and Payment Coverage for Data The 2007;10(5):326-35. Health Value Report. Makady A. Review of Policies and and Makady A. Review Policies of Data. on Real-World Perspectives http://tinyurl. URL: from: 2015;Available com/p2qq9xh Berger ML, Lipset C, Gutteridge A, Axelsen Axelsen A, Gutteridge C, Lipset ML, Berger Optimizing Madigan D. K, P, Subedi the Improve JM, Alvir Riaz MPP, Ali D, Alemayehu Sanchez EM, Perfetto MullinsMardekian CD, J, of Data, Examination RJ. Willke P, Subedi RJ, Methods Proposed and Issues Analytical Effectiveness Conducting Comparative for J Manag Data”. “Real-World Using Research suppl.):S3-S37. 2011;17(9 Pharm Care Freemantle N, Strack T. Real-world Real-world T. Strack N, Freemantle be should of new medicines effectiveness clinical designed appropriately by evaluated 2010;63(10):1053-8. J Clin Epidemiol trials. A, Breckenridge E, Abadie HG, Eichler H, Leufkens LL, Gustafsson B, Flamion M, Schneider Rowland CK, Bloechl-Daum Bridging B. a Rev Drug Nat variabilityresponse. drug of 2011;10(7):495-506. Discov Schwartz D, Lellouch J. Explanatory and J. Lellouch Schwartz D, J trials. in therapeutical attitudes pragmatic 2009;62(5):499-505. Epidemiol Clin

REFERENCE LIST REFERENCE 4. 3. 2. 1. 9. 8. 7. 6. 5. textbook for Health Health textbook for health decision maker decision maker health A the European Union . 2001. 2001. Union . European van Staa TP, Klungel OH. Background Paper Paper Klungel Background OH. TP, Staa van practice learning and from data Real-life 8.4 from: 2013;Available innovation. advance to http://www.who.int/medicines/areas/ URL: priority_medicines/BP8_4Data.pdf Maccabi Merck. Israel’s and Merck Real- Unique Leverage to Healthcare Health Novel Inform to Database World URL: from: Available 2013; Approaches. http://tinyurl.com/nnuffxn S.J. Walters M.J., Campbell Machin D., Medical Statistics: 2007. ed. Fourth Sciences. perspective. What are we talking we about? are What perspective. 2013;10(3). Health Public Biostat Epidemiol European Commission. DIRECTIVE Commission. 2001/20/ European AND EUROPEANTHE PARLIAMENT OF EC Officialof Journal THE COUNCIL. OF the in Data World Real Using AM. Pleil Evaluations: Pharamcoeconomic OpportunitiesChallenges, Approaches. and URL: http://pharmacy. from: 2013;Available ucsd.edu/faculty/AppliedPEForum/docs/ Andreas_M_Pleil.pdf Real- G. Romio M, Corrao Sturkenboom S, from data world

30. 27. 28. 29. 25. 26. centralised centralised the case for action. case for A 2013;Available from: URL: http://assets1. from: 2013;Available csc.com/health_services/downloads/CSC_ Real_World_Data_Research.pdf Doležal T. Real world data in Czech Republic. Republic. in Czech data Real world T. Doležal http://tinyurl. URL: from: 2008;Available com/o3lvm4o Foltz D, Ferrara L, Volkommer R, Turisco F. F. Turisco R, Volkommer L, Ferrara D, Foltz Research: Data Real-World procedure. 2015;Available from: URL: http:// URL: from: 2015;Available procedure. tinyurl.com/ph7aa8k Sanofi. topics. CSR Main on positions Sanofi http://tinyurl. URL: from: 2013;Available com/qxewnu4 Data Trial Clinical in Leaders Novartis. URL: from: 2014;Available Transparency. https://www.novartis.com/sites/www. novartis.com/files/leaders-in-clinical-trial- data-transparency.pdf European Medicines Agency. Delegated Medicines Agency. European Efficacy Studies. Post-authorisation Act on http://ec.europa.eu/ URL: from: 2013;Available health/files/pharmacovigilance/2012_11_28_ pc_paes.pdf European Medicines Agency. European European Medicines Agency. European Medicines Agency post-authorisation of users for advice procedural

24. 23. 22. 21. 20. 19.

What is RWD? 2

SECTION Policies for Real-World Evidence Use II

CHAPTER

Policies for Use of Real-World Data in Health Technology Assessment (HTA): 3 A comparative study of 6 HTA agencies

A. Makady1,2, R. ten Ham2, A. de Boer2, H. Hillege3, O. Klungel, W. Goettsch1,2 (on behalf of GetReal Workpackage 1)

1 The National Healthcare Institute (ZIN), the Netherlands 2 Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht, the Netherlands 3 Department of Epidemiology, University Medical Centre Groningen, Groningen, the Netherlands

Published in Value in Health. Citations is as follows: Makady A, ten Ham R, de Boer A, Hillege H, Klungel O, Goettsch W. Policies for Use of Real-World Data in Health Technology Assessment (HTA): A Comparative Study of Six HTA Agencies. Value in Health. 2017 Apr 30;20(4):520-32. use of RWD for use of RWD for starting to point a the efficacy of interventions, efficacyinterventions, of the literature review and stakeholder and stakeholder review literature A use of RWD for HTA across Europe, more alignment of policies seems seems of policies alignment more Europe, across use of RWD HTA for the following contexts for RWD use in REA of drugs were reviewed: initial reimbursement initial reimbursement reviewed: RWD for in REA of drugs were use contexts following ABSTRACT Randomised Controlled Trials (RCT)Trials on data robust provide Randomised Controlled discussions, pharmacoeconomic analysis and conditional reimbursement schemes. We We schemes. reimbursement conditional and analysis pharmacoeconomic discussions, conducted 6 interviews. and policy 13 identified publications, academic and 9 documents policies Moreover, contexts. across RWD for differed REA of drugs notably use in Policies discourage might variations Such agencies. HTA between differed rather than effectiveness. Health Technology Assessment (HTA) agencies worldwide are worldwideare agencies (HTA) Assessment Technology Health effectiveness. than rather on of data sources alternative provide (RWD) Data may Real-World whether thus exploring for policies RWD use agencies’ overview an of HTA Presently, of interventions. effectiveness policies of review to This study aimed is lacking. (REA) assessments effectiveness in relative agencies on RWD drugs. use in REA of HTA 6 European TLV RWD on 6 agencies: collect for policies information to conducted interviews were (Italy) and ZIN (Netherlands). AIFA HAS (France), (Germany), IQWiG (U.K.), NICE (Sweden), The facilitate To HTA. this. achieve necessary. Recent EUnetHTA papers and project proposals may provide provide may projectpapers and proposals EUnetHTA Recent necessary.

Policies for RWE use in HTA 3 Policies for RWE use in HTA 47 3 evidence evidence context of of context framework framework the re-assessment collection and collection and the the routine setting setting routine extent to which which to extent European setting. setting. European the IMI-ADAPT SMART SMART IMI-ADAPT the the The The the the broad array of stakeholders stakeholders of array broad a comparator (e.g. placebo or active or active placebo (e.g. comparator growing need for RWD use in HTA RWD need for in HTA use growing a the use of evidence development strategies strategies development evidence of use policy framework for RWD policy for use and good framework selected, homogenous group of patients is is of patients group homogenous selected, introduction of innovative, yet expensive, expensive, yet of innovative, introduction drug gains marketing authorisation, it is drug gains marketing is it authorisation, a a the a use of RCT-generated efficacy data to predict efficacy to data use of RCT-generated platform for multiple-stakeholder for discussions platform the a the evidence base. base. evidence experimental drug or experimental desired results and when provided under under provided and when results desired the the Patient-Centered Outcomes Research Initiative (PCORI) and and (PCORI) Initiative Research Outcomes Patient-Centered the the heterogeneous patient group in routine clinical practice whereby patients patients practice clinical whereby routine in group patient heterogeneous a drug, due to its ability to minimise problems with confounding, information information confounding, with ability its problems minimise due to to drug, a 3-year project enabling 3-year relative effectiveness of an intervention is defined as: “ as: defined is intervention an of effectiveness relative a The The Innovative Medicines Initiative GetReal MedicinesInitiative ConsortiumInnovative (IMI-GetReal). IMI-GetReal is relative effectiveness of drugs, HTA agencies worldwide are currently exploring exploring currently worldwide are agencies HTA of drugs, effectiveness relative possibilities for using Real-World Data (RWD) to supplement and enrich (RWD) supplement Data to Real-World using for possibilities efficacy of Conventionally, data on treatment effects for drugs is collected within is for drugs effects on treatment data Conventionally, Additionally, HTA agencies are exploring exploring are agencies HTA Additionally, Due to limitations associated with with associated limitations Due to Research conducted by IMI-GetReal identified three contexts within which within IMI-GetRealcontexts by conducted RWD is Research three identified 3-year project aiming at investigating policies and methodologies for for methodologies policies and investigating project at aiming 3-year INTRODUCTION pharmaceutical products, Health Technology Assessment (HTA) agencies are seeking are agencies (HTA) Assessment Technology Health products, pharmaceutical clinical routine of drugs in (REA) assessments effectiveness relative for methods robust practice. In light of rising healthcare costs and costs In of rising healthcare light an intervention does more good than harm, when compared to one or more intervention intervention more one or to than harm, good compared when does more an intervention achieving for alternatives of health care practice (i.e. real-world setting).” (1) setting).” real-world (i.e. practice care of health whereby trials (RCT’s), controlled randomised randomly assigned to either either to assigned randomly practices for its integration in in its integration practices for One example, (MAPP’s)3. patients to pathways adaptive of medicine that provide effectiveness research data earlier during drug development in earlier during data drug development research effectiveness provide that project, is project, activities in of MAPP’s implementation to on questions relating highlighted have publications numerous Moreover, comparator) under highly controlled conditions. This study design is ideal to demonstrate demonstrate is ideal to study design This conditions. controlled under highly comparator) the once bias and selection However, bias. administered to to administered present with differing co-morbidities, co-medication and genetic profiles. Consequently, it Consequently, co-medicationco-morbidities, profiles. and genetic differing with present practice (2). in clinical drug effects to RCT’s from results extrapolate to is challenging across Europe to collaborate on developing on developing collaborate to Europe across currently being used for REA of drugs: as supplementary of drugs: REA REA after initial market used for being input for currently and for (PEA) analyses pharmacoeconomic input for as authorisation, decision-making to inform: clinical effectiveness parameters, natural history of disease, history natural of disease, parameters, effectiveness clinical decision-making inform: to on demand and or information quality life, of and health-related treatment to adherence (4-9). specific settings in evaluations health economic for constraints supply the the for REA of drugs. Examples of national and international collaborations exploring these these exploring collaborations international and national of Examples drugs. of REA for include possibilities It and assessment. use of RWD combines in drug development However, (CRS) (8). schemes reimbursement conditional within effectiveness of relative the a following following review of of review a the 4 largest European European 4 largest number of European European of number a the comparison across several several across comparison European network of HTA network of HTA European study authors do not master do not master study authors a Dental and Pharmaceutical Pharmaceutical and Dental search for academic articles for search a the 2016 (date of search), explicitly explicitly of search), (date 2016 the st the appraisal of RWD of appraisal in Italian Medicines Agency (AIFA, Italy) Italian Medicines Agency (AIFA, search strategy). To minimise chances chances minimise To strategy). search the the the Netherlands). HTA agencies within France, France, within agencies Netherlands).HTA European network of HTA (EUnetHTA)), an an (EUnetHTA)), network of HTA European policies of 6 HTA agencies in Europe on RWD in Europe agencies HTA 6 of policies article considers agencies’ policies regarding policies regarding article agencies’ considers harmonisation of HTA activities across Europe activities Europe across of HTA harmonisation the comprehensive overview of RWD policies of HTA overview of RWD of HTA policies comprehensive review of academic publications by HTA affiliates affiliates HTA by publications of academic review the the 2006 and June 21 2006 the a the a st National Institute for Health and Care Excellence Excellence and Care Health Institute for National Appendix for for Appendix the Institute for Quality and Efficiency in Healthcare (IQWiG, Quality and EfficiencyInstitute for (IQWiG, Healthcare in the 6 HTA agencies were searched for guidelines and policy guidelines for searched were agencies 6 HTA time span of 10 years was selected. Articles were included if: included Articles selected. was years time span of 10 were a the the good starting further for point of on harmonisation discussions three contexts mentioned above seems to be lacking. Provided be lacking. Provided to seems above mentioned contexts three a U.K. were selected since they represent they represent selected since were U.K. the similarities and differences between different HTA agencies’ policies policies agencies’ HTA different between similaritiesand differences the three contexts: IRD, PEA and CRS. Documents were included if they were were if they included were PEA and CRS. Documents IRD, contexts: three so co-called “Big Four”); jurisdictions bearing most influence on European jurisdictions Four”); “Big European on bearing most influence so co-called the websites of websites use of RWD in REA of drugs, were published in English, German, French or or German, in English, published French use of RWD were REA of drugs, in the the the United Kingdom),United the PubMed interfacePubMed (see Netherlands were selected due to their pioneering roles, both historically and and historically both pioneering their selected roles, to due Netherlands were National Healthcare Institute (ZIN, Healthcare National use of RWD in the the the the selected agencies. Therefore, this article this review aims to Therefore, the Firstly, Firstly, METHODS initial comparison of policies for RWD use by HTA agencies across across agencies RWDHTA for policies of by use comparison initial Six European HTA agencies were selected for analysis: analysis: selected for agencies were HTA Six European an overview of for for for interest effortsgrowing and recent jurisdictions provide may policies on this topic. on this policies (e.g. as demonstrated by activities by of as demonstrated (e.g. specifically, More of drugs. use in REA for policies as well as requested, or RWDaccepted three contexts: initial reimbursement discussions (IRD), PEA and CRS. ItCRS. and PEA (IRD), discussions important is note reimbursement initial to contexts: three this articlethat provide not aim to does discussed discussed since excluded Italian and were documents (Swedish Dutch Benefits Agency (TLV, Sweden), Sweden), AgencyBenefits (TLV, (NICE, (HAS, France), de Santé Autorité Haute Germany), agencies in all 29 European jurisdictions but rather to present present jurisdictions to rather but European 29 all in agencies jurisdictions in Europe. relevant and Germany, Italy and Germany, policies on several aspects, including health (10-12). Meanwhile HTA agencies in Sweden in Sweden agencies HTA Meanwhile (10-12). aspects, health including on several policies and jurisdictions ( as such projects, HTA in cutting-edge European currently, (EUnetHTA) (13). To ensure that all relevant information on agencies’ policies on RWD policies use on agencies’ information all relevant that ensure To (13). (EUnetHTA) information: retrieve to used 3 methods were collected, of drugs was in REA agencies’ guidelines and policy and guidelines papers, agencies’ on RWD use in REA of drugs and semi-structured interviews with representatives from from RWDon semi-structured and drugs interviewsof REA in use representatives with the within papers Secondly, Dutch. or German, in English, published French published by agency affiliates relating toconducted in was relating REA of drugs RWD use in agency Medline by published affiliates using of missing relevant literature, literature, of missing relevant January between published they were 1

Policies for RWE use in HTA 3 Policies for RWE use in HTA 49 3

6 the manner manner a IMI-GetReal the broader review of of review broader context of highly- of context a the scope of questions posed posed of questions scope the interviewed representatives of all 6 6 of all interviewedrepresentatives effects of health interventions (e.g. (e.g. interventions of health effects the directed content analysis approach for for approach analysis content directed the ” (16) ” . interview to guide discussions (see Figure interview (see Figure discussions guide to validity of stakeholder views, participants validity views, stakeholder of the compiled documents and transcripts reported reported transcripts and documents compiled the the the aims of this research. aims of interviews were conducted as part conducted interviews were of the the standardised questionnaire was sent to all representatives all representatives to sent was questionnaire standardised A appraisal of RWD for REA of drugs within IRD, PEA and CRS (see PEA and CRS (see of RWD IRD, within drugs REA of appraisal for coding analysis of analysis coding the sampling of representatives and interview protocols were compared compared were and interview protocols of representatives sampling freedom to invite colleagues they deemed relevant to take part to in relevant deemed they colleagues invite to freedom the The The the scheme was developed by iterative assessment of included documents documents of included assessment iterative by developed scheme was purpose of this article, we based our definition for for purpose of this article, our definition based on RWD we The The Appendix). In order to increase increase to InAppendix). order consolidated criteria for reporting qualitative studies (COREQ) to ensure good good ensure to (COREQ) studies reporting qualitative for criteria consolidated results from from results the preference for senior HTA assessors and Research & Development senior officers. officers. senior & Development and Research assessors senior HTA for preference standardised coding scheme was developed using MaxQDA 11.0 software to extract 11.0 software to using MaxQDA developed scheme was coding standardised a the interviews. Interviews were conducted, recorded and subsequently transcribed for for transcribed and subsequently recorded conducted, Interviewsinterviews. were authors’ professional network. All representatives were approached by e-mail by approached using were network. representatives All professional authors’ For For The The A Thirdly, semi-structured interviews were conducted with representatives from from representatives with conducted semi-structured interviews were Thirdly, It is important that note to the standardised invitation. invitation. standardised in this manuscript were subsequently verified with verified subsequently in this manuscript were factual ensure correctness. to agencies which reflects how interventions would be used in routine clinical practice or secondaryclinical routine in used be would interventions which reflects how not are but include, collected Data data. collected routinely from derived data research and health-related outcomes patient-reported outcomes, clinical and economic to, limited registries, patient including sources RWD many can be obtained from quality of life. claims databases and electronic records, medical definition: “An umbrella term for data regarding regarding for data term umbrella “An definition: collected in not are that etc) use, resource effectiveness, safety, RWD Instead, be primary can either in collected data RCT’s. research controlled qualitative research (15). Two authors independently performed authors independently abstraction data and Two (15). research qualitative consensus. by resolved were discrepancies Any coding. and interview transcripts, in accordance with and interview in accordance transcripts, data from all compiled documents and transcripts on two aspects; RWD accepted or aspects; two on transcripts and or documents RWD all compiled from accepted data as as well requested, 1). Figure in interviews extended beyond in interviews extended beyond these two languages) and comprised more than an abstract. Articles were excluded if they they if abstract. an than excluded Articles more comprised and languages) two these were agency and PubMed from websites retrieved criteria. Documents inclusion all did not meet regarding disagreements Any authors. two by independently evaluated were searches consensus. by of articles resolved exclusion or inclusion were a HTA agencies. Representatives were selectively based on seniority sampled and function, were Representatives agencies. HTA with agency and/or from websites retrieved was representatives identifying for Information the quality(14). Therefore, policies and perspectivesstakeholder on RWD (8). who agreed to participate 2 weeks prior to participate prior to to weeks 2 who agreed i in provided were the further analysis. to analysis. the interview. Two interviews Two interview. 6 agencies, all agreed to to all agreed 6 agencies, the the ‐making ‐making ‐making 6 agencies, 1 additional colleague was was colleague 1 additional 6 agencies, number and nature of documents varied varied of documents and nature number requested requested requested decision decision decision

the hierarchies hierarchies hierarchies

The The on on on accepted directly accepted directly accepted directly

Accepted/Requested Appraisal Accepted/Requested Appraisal Accepted/Requested Appraisal 9 selected articles, 1 involved affiliates from several from several affiliates 9 selected articles,1 involved •Data •Data •Evidence •Data •Data •Evidence •Data •Data •Evidence •Impact •Impact •Impact •RWD •RWD •RWD •RWD •RWD •RWD the

(IRD)

(CRS)

reimbursement approached representatives to participate to in representatives approached

(PEA) reimbursement

9 agency representatives approached across across approached 9 agency representatives the schemes discussions Pubmed search initially yielded 284 hits; 9 were selected for further selected for and analysis 9 were 284 hits; initially yielded search Pubmed – Coding scheme developed to conduct coding analysis. conduct coding to scheme developed – Coding the Initial search for guidelines and policy and guidelines for on RWD papers search 13 use on agency yielded websites Conditional Pharmacoeconomic analysis In total, 22 documents and 6 interview transcripts were included in included interview and 6 22 documents were In transcripts total, Of The The Figure 1: Coding scheme developed to conduct coding analysis.codingtoconductFigure Codingscheme 1:developed RESULTS Figure 1 Figure participate (response rate= 100%). For 2 of For 100%). participate rate= (response by invited included 1 agencyincluded participant, 3 included 2 agency participants 3 agency and 1 included 1). Table participants (see 275 were excluded because they did not meet all inclusion criteria (see Figure 2 for PRISMA PRISMA for 2 criteria did not meet all inclusion (see Figure because they excluded 275 were on articlediagram selection). Of affiliates to NICE specific were 3 (18-21), affiliates to AIFA specific agencies (17), 4 were HTA Table 1). (25) (see affiliate to an HAS specific was and 1 (22-24) per institute. Some agencies had separate guidelines for IRD and PEA (e.g. TLV, HAS, ZIN), HAS, ZIN), TLV, IRD and PEA (e.g. for guidelines Some had separate agencies per institute. NICE). (e.g. in one document both others combined whereas documents (see Table 1). All 6 agencies had guidance and policy papers available for IRD, IRD, and policy for had guidance 6 agencies 1). All available papers Table (see documents CRS. for PEA and 3 agencies for 5 agencies The The

Policies for RWE use in HTA 3

Policies for RWE use in HTA 51 3

Transcript reference: ‘c’ reference: Transcript

Allgemeine Methoden version 4.2 (23) 4.2 version Methoden Allgemeine IQWiG - participant 1

(20)

collecting evidence of effectiveness and safety safety and effectiveness of evidence collecting

new treatments are funded conditional on on conditional funded are treatments new

quality of evidence when when evidence of quality the improve to How

evidence in HTA (19) HTA in evidence

How real-world data compensate for scarce scarce for compensate data real-world How

of “Colloquial Evidence” at NICE (17) NICE at Evidence” “Colloquial of

Use Use The Making: Decision Informed Evidence

Individual Patient Data (26) Data Patient Individual

Technology Appraisal: Methods for Comparative Comparative for Methods Appraisal: Technology

Estimates of Treatment Effectiveness in in Effectiveness Treatment of Estimates the

Value of Registries (18) Registries of Value the Use of Observational Data to Inform Inform to Data Observational of Use The

Methodological Challenges in Evaluating Evaluating in Challenges Methodological NICE DSU Technical Support Document 17: 17: Document Support Technical DSU NICE

of Pharmaceuticals Across Europe (12) Europe Across Pharmaceuticals of Transcript reference: ‘b’ reference: Transcript 2013 (22) 2013

participants 3 NICE methods of technology appraisal appraisal technology of methods the to Guide Evidence Requirements for Reimbursements Reimbursements for Requirements Evidence

System (33) System

Swedish Pharmaceutical Reimbursement Reimbursement Pharmaceutical Swedish The

2003:2) (28) 2003:2)

Pharmaceutical Benefits Board (LFNAR (LFNAR Board Benefits Pharmaceutical the from

General guidelines for economic evaluations evaluations economic for guidelines General

and pricing for pharmaceutical products (21) products pharmaceutical for pricing and Transcript reference: ‘a’ reference: Transcript

- participant 1 TLV Guide for companies when applying for subsidies subsidies for applying when companies for Guide

& Transcript Reference Transcript & HTA Agency HTA Policy Papers & Guidelines & Papers Policy Academic Publications Academic

Number of Interview Participants Participants Interview of Number

per agency. per

- List of policy documents, guidelines and academic publications retrieved, as well as as well as retrieved, publications academic and guidelines documents, policy of List - 1 Table number of interview participants and transcript reference reference transcript and participants interview of number the

geneeskundige zorg 2015 (32) 2015 zorg geneeskundige

Procedure voorwaardelijke toelating toelating voorwaardelijke Procedure

Leideraad voor Uitkomstenonderzoek (27) Uitkomstenonderzoek voor Leideraad

evaluaties in de gezondheidszorg (30) gezondheidszorg de in evaluaties

Richtlijn voor het uitvoeren van economische economische van uitvoeren het voor Richtlijn

Transcript reference: ‘f’ reference: Transcript of Pharmaceuticals Across Europe (12) Europe Across Pharmaceuticals of praktijk (25) praktijk

2 participants 2 Evidence Requirements for Reimbursements Reimbursements for Requirements Evidence en wetenschap de van stand Beoordeling ZIN

and real-world evidence generation (16) generation evidence real-world and

tool to support health-care decision-making decision-making health-care support to tool a

nationwide Osmed Health-Db database: database: Health-Db Osmed nationwide The

Italian post-marketing registries (14) registries post-marketing Italian The

Agency (AIFA) experience (15) experience (AIFA) Agency

Italian Medicines Medicines Italian the drugs: on research

Feasibility and challenges of independent independent of challenges and Feasibility

clinical trial regulation in Italy (13) Italy in regulation trial clinical

New Perspective and new challenges in in challenges new and Perspective New

Transcript reference: ‘e’ reference: Transcript of Pharmaceuticals Across Europe (12) Europe Across Pharmaceuticals of

2 participants 2 AIFA - Evidence Requirements for Reimbursements Reimbursements for Requirements Evidence

actes) (31) actes)

de santé (médicaments, dispositifs médicaux et et médicaux dispositifs (médicaments, santé de

Les études post-inscription sure les technologies technologies les sure post-inscription études Les

Choices in Methods for Economic Evaluation (29) Evaluation Economic for Methods in Choices

Transcript reference: ‘d’ reference: Transcript Technologies (24) Technologies

- 2 participants 2 HAS General Method for Assessing Health Health Assessing for Method General

& Transcript Reference Transcript & HTA Agency HTA Policy Papers & Guidelines & Papers Policy Academic Publications Academic

Number of Interview Participants Participants Interview of Number

- continued) - 1 Table

Policies for RWE use in HTA 3 Policies for RWE use in HTA 53 3

absence absence the PubMed search. PubMed the drug undergoing REA, which which REA, drug undergoing the applicant should resort to (26, 27, 29, resort should applicant 27, 29, (26, to the suitability of these sources to answering different scientific scientific different answering to suitability sources of these the absence of RCTabsence drug efficacy on evidence 30;b,f). 28, (27, In the - PRISMA diagram of inclusion & exclusion of articles retrieved through articles of through exclusion & inclusion of retrieved diagram - PRISMA assessed drug but only under specific circumstances. For example, RWD may be be example, may RWD For circumstances. specific drug but only under assessed Agencies iterate that RWD may be used to demonstrate treatment effects of of effects treatment demonstrate RWD that to used be may iterate Agencies Figure 2 Figure of RCT data on head-to-head comparisons between treatments, RWD may be drawn RWDof RCT be drawn may on head-to-headtreatments, data between comparisons indirect enable to treatment effectiveness of on estimates information provide to upon 30;b,f). (27, comparisons RCT RWD supplement be used to may on data Finally, is lacking follow-up or long-term on specific subpopulations if data effects treatment 30;b,f).(27, justification explicit an require agencies In above, mentioned all situations the used in questions (27, 28, 31, 32;b,c,f).28, 31, (27, questions All HTA agencies accept all available evidence on evidence all available accept agencies HTA All Initial Discussions Reimbursement 30;a,b,d-f). However, several do provide suggestions for specific RWD sources as well as well as as specific for suggestions RWD sources do provide several 30;a,b,d-f). However, preliminary on guidance implicitly includes RWDimplicitly (26-30;a-f). RWD of nor sources which do not specify Agencies RWD for collectionwhich methodologies

RWD used the evaluation like like evaluation the form of utilities (27, 32, 35) or provide (27, 32, 35) or provide of utilities form biases associated with associated biases use of RWD for epidemiological data, data, RWDof use epidemiological for the the the greater risk to validity of conclusions thus should should thus validity conclusions of risk to greater a clear discussion of clear discussion acceptability and impact acceptability RWD of decision-makingon cases in a summary and RWD or requested on RWD of policies accepted the a health effects over time in over effects health the context of IRD per agency. of IRD per context first context, RWD is directly requested by 5 HTA agencies for various aspectsfor agencies HTA by 5 context, requested first is directlyRWD agency does not conduct PEA). More specifically, agencies recommend that that recommend agencies agency specifically, More does not conduct PEA). exclusion of valuable non-RCT of valuable decision-making from exclusion (27, 30;b,f). evidence th the 6 the the the lower level of quality and reliability than RCT’s. Consequently, agencies iterate that that iterate agencies Consequently, of quality RCT’s. than and reliability level lower With regards to RWD to PEA, in appraisal regards With Several agencies specify that treatment effects used for modelling relative effectiveness effectiveness relative for modelling used effects treatment agencies specify that Several Moreover, 3 agency guidelines iterate that RWD may be used to provide information information RWD provide that be used to 3 agency may iterate guidelines Moreover, with principles of accordance in of evidence hierarchies similar adopt agencies All Agencies differ on differ Agencies 3 presents Table a data on transition probabilities between different disease states in pharmacoeconomic pharmacoeconomic in states disease different between probabilities on transition data to and compliance adherence practice, use in routine resource direct indirect and costs, models (27, 32). models should primarily be based upon results from RCT’s (27, 32, 34, 35;b,d,f). Alternatively, 35;b,d,f). 34, (27, 32, RCT’s from primarilyshould results be based upon Alternatively, 34;b,d,f),(27, 30, effects complementary on treatment RWD evidence provide may be valuate used to adherence to treatment and compliance, can also be collected from RWD sources such as such as RWD from collected be can also sources and compliance, treatment to adherence surveys(34). epidemiological or ad-hoc studies databases, registries, and its consequences on treatment effect estimates (26-30;a-c,e). estimates effect on treatment consequences and its why RWD used and why was of PEA ( of PEA Pharmacoeconomic Analysis Pharmacoeconomic Contrary to on aspects other than treatment effect, such as: epidemiological data (e.g. incidence and and incidence (e.g. data epidemiological as: such effect, aspectson treatment than other 28, 30). (27, data and cost use data resource prevalence), of RWD sources place unanimously evidence-based hierarchies Adopted (26-30). medicines on RWD may be used to confirm or supplement, rather than substitute, findings on causal findings on causal substitute, than rather or supplement, RWD confirm used to be may (27-30;b-d,f).by RCT’s demonstrated effects treatment conclusions on treatment Thus, circumspect than RCT-derived as more regarded generally from RWD are derived effects be can effect this to of quotes decision-making examples by conclusions committees; strictly with associated limitations recognise explicitly 2 agencies 2. However, table in found not should hierarchies such that state and in guidelines hierarchies evidence adopting preclude epidemiological data (e.g. incidence and prevalence), direct and indirect costs, and resource and resource direct indirect and costs, prevalence), and incidence (e.g. data epidemiological databases, claims (e.g. RWD national from collected sources practice be use in routine 32-34;b,e,f). 28, (27, databases) hospital registries, Other aspects of where RCT data is sparse, for example for orphan diseases: several state that non-RCT that state data orphan several diseases: for example RCT where for sparse, is data decision-making for be resorted 30;a,b,f), in these cases (27, to could one states whereas resorting that non-RCT to presents data disparity on this of views agencies’ demonstrating Examples of quotes (28;c). be avoided 2. Table in can be found issue in appraisal

Policies for RWE use in HTA 3 Policies for RWE use in HTA 55 3

RWD very, very, very, very, first line first line a the post-hoc post-hoc the the confirmation of of confirmation a confirmation of of confirmation benefit from such such from benefit registry, no selection registry, expectation you have have expectation you for data data you have had had have you data a a registry with data.” (c) data.” need then would we “… a Quotation B Quotation those accept we course “Of law by forced are We data. but those data accept to conclude to have don’t we the very, high quality. In terms Interms high quality. very, in all patients of having the criteria and no selection that imagine could We bias. accept only then would we registrythese for analyses in but not diseases, veryrare (c) general.” “You can’t really rely on on rely really can’t “You can use You it. as the and assessment on initial the population you have, and and have, you population the of already So it is analysis. subgroup as used (f) conclusions.” previous context of IRD, implying that RWD that implying of IRD, context back-up.” (b) back-up.” robustness robustness bias and and bias a the reevaluation of of reevaluation the the re-evaluation pricing and pricing and most ideal source for for source most ideal the product.” (e) product.” relative treatment treatment relative problems that are are that problems product, this kindproduct, of analysis that we need to need to we that analysis the the the the role in orphan diseases orphan in diseases role “Yes, RWD certainly plays “Yes, a to difficult are RCT’s since In conduct this area. in that may registries patient case, be (f) RWD.” “There is this red flag “There flag red is this use non If you in there. and nonrandomized you evidence, controlled careful, be more to have circumspect about more the those from drawn effect Ideally you studies. one than use more should such of source independent as evidence, Quotation A Quotation the to related kindof that [RWD] of data. For “So, we are used to using using to used are we “So, kind that though of data, know we for for reimbursement of of reimbursement the for enough robust are data the do for pricing and reimbursement pricing and reimbursement of lower quality level (27, 28, 32, 34, 35;b,d,f). Therefore, 35;b,d,f). 32, 34, 28, (27, quality level lower Therefore,

RWD use to inform RWD inform use to effect treatment for estimates orphan diseases Appraisal of RWD vs. of RWDAppraisal vs. RCT treatment for data estimates effect in general Topic Use of RWD generated of RWD Use generated CRS for within decision-making - Examples of interview quotes on RWD use in IRD and CRS. same hierarchies of evidence apply as in evidence of same hierarchies IRD IRD Context for RWD RWD for Context (IRD or CRS) Use CRS Table 2 treatment is largely accepted by HTA agencies. However, for relative treatment effects, effects, treatment relative for However, agencies. HTA by accepted is largely treatment the is conventionally placed on placed is conventionally

valuable non-RCT evidence from decision-making. from evidence non-RCT valuable

*However, agency explicitly recognises limitations associated with strictly adopting evidence hierarchies in guidelines and states that such hierarchies should not preclude preclude not should hierarchies such that states and guidelines in hierarchies evidence adopting strictly with associated limitations recognises explicitly agency *However, exclusion of of exclusion the

data & cost data cost & data

treatment effects treatment prevalence), resource use use resource prevalence),

to evidence for for evidence to (e.g. incidence and and incidence (e.g. circumstances

Yes Yes Yes*; with regards regards with Yes*; epidemiological data data epidemiological Under specific specific Under Yes ZIN

treatment effects treatment

to evidence for for evidence to circumstances

Not mentioned Not Yes Yes; with regards regards with Yes; mentioned Not Under specific specific Under Yes AIFA

treatment effects treatment

to evidence for for evidence to circumstances

Not mentioned Not Yes Yes; with regards regards with Yes; mentioned Not Under specific specific Under Yes HAS

use data use

treatment effects treatment prevalence), resource resource prevalence),

to evidence for for evidence to (e.g. incidence and and incidence (e.g. circumstances

No Yes Yes; with regards regards with Yes; epidemiological data data epidemiological Under specific specific Under Yes IQWiG

data & cost data cost & data

treatment effects treatment prevalence), resource use use resource prevalence),

to evidence for for evidence to (e.g. incidence and and incidence (e.g. circumstances

Yes Yes Yes*; with regards regards with Yes*; epidemiological data data epidemiological Under specific specific Under Yes NICE

treatment effects treatment

to evidence for for evidence to circumstances

Yes Yes Not mentioned Not Yes; with regards regards with Yes; Under specific specific Under Yes TLV

orphan diseases) orphan circumspect evidence adopted evidence parameters treatment effects treatment RWD Accepted RWD Agency HTA

circumstances (e.g. (e.g. circumstances regarded as as regarded Hierarchy of of Hierarchy RWD to inform other other inform to RWD RWD to inform inform to RWD

possible in exceptional exceptional in possible based on RWD RWD on based

effects based on RWD RWD on based effects treatment effects effects treatment

Conclusions on treatment treatment on Conclusions Conclusions on on Conclusions

RWD Accepted/Requested RWD RWD Appraisal Appraisal RWD

context of IRD per agency. per IRD of context the in RWD of appraisal the and requested or accepted RWD on policies of Summary - 3 Table

Policies for RWE use in HTA 3 Policies for RWE use in HTA 57 3

first first RWD the the applicant applicant the same procedure procedure same applicant’s study study applicant’s agencies. Secondly, Secondly, agencies. the contract is drawn up up is drawn contract the fourth agency stated fifth agency recently fifth agencyrecently a 3 agencies. For For 3 agencies. the A a the preference for RWD. However, However, RWD. for preference selection of candidates for CRS CRS selection for candidates of a study protocol by by study protocol protocol(s), protocol(s), the the agency and an applicant to conduct agency conduct to an applicant and study protocol to collect to study protocol study protocol and date for submitting submitting for date and protocol study a the the scientific questions raised during initial duringraised initial questions scientific supplementary role for RWDsupplementary for (32, 37;f). role For the a the preference is made for RWD RCT than for made rather is data preference a CRS scheme for oncologic drugs (39;b). However, it it However, (39;b). drugs oncologic for scheme CRS a applicant develop develop applicant last agency varies; they can be used to inform re-assessmentlast agency inform be used to can they varies; latter two schemes constitute ones as established as those as those as established ones constitute schemes two latter the the the summary RWD and or requested on RWDpolicies of accepted a contract is drawn up between between up drawn is contract preference for RCT for and data preference a a 3 agencies clearly defined criteria for clearly3 agencies criteria defined establishment of establishment purpose for data collection is focused primarily on demonstrating primarily on demonstrating collection data is focused purpose for context of PEA per agency. of PEA per context other 3 agencies (a,b). 3 agencies other last agency, recommendations to set up post-marketing studies are similarly similarly are studies post-marketing up set to recommendations agency, last 6 HTA agencies implement CRS (19, 20, 36, 37;d-f). 36, 20, (19, CRS implement agencies 6 HTA the agencywhich within applicant and the the the the product is nominated for conditional reimbursement on two conditions; that that conditions; two on reimbursement conditional for product is nominated the the the a purposes for RWD collection for CRS differed between between RWD CRS differed purposes for collection for procedure to do so (32, 37;f). do so to procedure agencies request that that request agencies a use of study results for for results use of study drug in national clinical practice and practiceand clinical national in drug second agency, agency, second Table 4 presents 4 presents Table The The Only 1 of Only 1 These principle differences notwithstanding, 2 agencies follow follow agencies 2 notwithstanding, principle differences These conclusions for relative treatment effects based on RWD are considered as being more more being as considered based on effects are RWD treatment relative for conclusions circumspect 35;b-d,f). 34, 28, 32, (27, in appraisal Conditional Reimbursement Schemes Reimbursement Conditional of Three briefly that reimbursement can be conditionally offered to allow an applicant time time applicant an to allow offered conditionally be can reimbursement briefly that RWDmore (38;a). effects on long-term Meanwhile, procure to and outlined by by outlined announced announced unclear whether remains agency, agency, assessment. at initial sparse is highly effectiveness on its data and highly innovative it is Therefore, protocol(s) are reviewed by independent committees to judge their scientific quality and qualityscientific their judge and to committees independent by reviewed are protocol(s) made to are adjustments relevant Once feasibility. the RWD national existing of detailed examples also includes sources guidance This assessment. Thirdly, 37). 36, (32, specific questions answer to be used may that effectiveness, with effectiveness, is highly RWD drug candidate each that implying for collected gaps, such inform to needed on which to applicants guidance case-specific. methodological agencies provide Both answer to order in choose to designs study additional evidence is specified. Furtherto adaptations is specified. evidence additional post-marketing studies that aim to answer questions raised during initial assessment. during assessment. initial raised questions answer to aim that studies post-marketing of cost-effectiveness and/or of effectiveness issues equally to relate questions may These the the for be used may also re- but practice, cost-effectiveness in clinical and/or of effectiveness pricing (19, 20;e). discussions for submissions in presented gaps in evidence Firstly, reimbursement. conditional for by identified systematically are discussions reimbursement initial the between (36;d). For For (36;d). with during assessment initial upon questions raised based

and resource use resource and

(direct & indirect) indirect) & (direct

treatment effects treatment & prevalence), costs costs prevalence), &

for evidence data (e.g. incidence incidence (e.g. data circumstances

Yes Yes Yes; with regards to to regards with Yes; epidemiological epidemiological Under specific specific Under Yes AIFA

compliance

use, adherence and and adherence use,

indirect), resource resource indirect),

costs (direct & & (direct costs

treatment effects treatment & prevalence), prevalence), &

for evidence data (e.g. incidence incidence (e.g. data circumstances

Yes Yes Yes; with regards to to regards with Yes; epidemiological epidemiological Under specific specific Under Yes HAS

N/A N/A N/A N/A N/A N/A IQWiG

and resource use resource and

(direct & indirect) indirect) & (direct

treatment effects treatment & prevalence), costs costs prevalence), &

regards to relative relative to regards data (e.g. incidence incidence (e.g. data circumstances

Yes Yes Yes*; specifically with with specifically Yes*; epidemiological epidemiological Under specific specific Under Yes NICE

and resource use resource and

(direct & indirect) indirect) & (direct

treatment effects treatment & prevalence), costs costs prevalence), &

for evidence data (e.g. incidence incidence (e.g. data circumstances

Yes Yes Yes; with regards to to regards with Yes; epidemiological epidemiological Under specific specific Under Yes TLV

RWD regarded as reliable as regarded RWD as circumspect as treatment effects treatment other parameters other evidence adopted evidence Recommended HTA Agency HTA

parameters based on on based parameters on RWD regarded regarded RWD on

RWD to inform inform to RWD

Hierarchy of of Hierarchy

RWD to inform inform to RWD RWD RWD

Conclusions on other other on Conclusions treatment effects based based effects treatment

Conclusions on on Conclusions

RWD Accepted/Requested RWD RWD Appraisal Appraisal RWD

context of PEA per agency. per PEA of context the in RWD of appraisal the and requested or accepted RWD on policies of Summary - 4 Table

Policies for RWE use in HTA 3

Policies for RWE use in HTA 59 3

and resource use resource and

(direct & indirect) indirect) & (direct

treatment effects treatment & prevalence), costs costs prevalence), &

for evidence data (e.g. incidence incidence (e.g. data circumstances

Yes Yes Yes*; with regards to to regards with Yes*; epidemiological epidemiological Under specific specific Under Yes ZIN

RWD regarded as reliable as regarded RWD as circumspect as treatment effects treatment other parameters other evidence adopted evidence Recommended HTA Agency HTA

parameters based on on based parameters on RWD regarded regarded RWD on

RWD to inform inform to RWD

Hierarchy of of Hierarchy

RWD to inform inform to RWD RWD RWD

Conclusions on other other on Conclusions treatment effects based based effects treatment

Conclusions on on Conclusions

RWD Accepted/Requested RWD RWD Appraisal RWD

- continued - 4 Table scientific quality and qualityscientific and the third agency. third context of CRS per agency. of CRS per context the the basis for decision-makingbasis for during re- impact of RWD on decision-making if agencyIt37). 36, (32, whether unclear is the burden of RWD collection on applicants, of RWD burden collection on applicants, the the the study eventually conducted adhere strictly to strictly adhere conducted to study eventually the studies implemented deliver data of adequate quality quality of adequate data deliver implemented studies the studies would form form would studies the summary on RWD of policies use in a study remain valuable for decision-making. If these conditions are met, met, decision-making. If are these conditions for valuable study remain first 2 agencies, which lay lay which first 2 agencies, the the protocol agreed upon by all parties. This is to ensure that that ensure is to This all parties. by upon agreed protocol third agency third product or indication often actively its own, participates initiates or in, same procedure also applies for CRS implemented by by CRS implemented for applies also same procedure Unlike Unlike All 3 agencies require that that require 3 agencies All Table 5 presents 5 presents Table Similarities and differences in policies for RWD accepted or requested and RWD appraisal and requested appraisal RWD or for policies in accepted RWD Similarities and differences are possible but only after consultation with with after only but possible consultation are the the 37;d,e,f). 36, (18-20, registries 36, 37;d,e,f). (32, during assessment initial identified questions answer to and robustness that require 2 agencies Moreover, the outcomes of outcomes conclusions for treatment effects based on RWD contradict those from RCT-based evidence. based on effects RCT-based contradict those from RWD treatment for conclusions results generated by by generated results within IRD, PEA and CRS are presented in Table 6. Table in presented and CRS are PEA IRD, within assessment (32, 36, 37;d,e,f). However, quotes from interviews varying on light shed from quotes 37;d,e,f). 36, (32, assessment However, decision-making for such studies practice (see from generated of resulted acceptability on guidance no was there 2). Moreover, table

Policies for RWE use in HTA 3

Policies for RWE use in HTA 61 3

upon study protocol. study upon

based on results on based adherence to agreed- to adherence evidence

3- Decision-making Decision-making 3- gap highlighted and and highlighted gap supplementary supplementary

study design study addresses evidence evidence addresses with RWD as as RWD with

2- Consultation on on Consultation 2- delivered sufficiently sufficiently delivered RCT data data RCT as supplementary as

whether data data whether instance instance RCT data with RWD RWD with data RCT evidence gap evidence

Conditional on on Conditional No; in first first in No; No No; in first instance instance first in No; 1- Identification of of Identification 1- Effectiveness Yes ZIN

gap highlighted. gap

addresses evidence evidence addresses

delivered sufficiently sufficiently delivered price re-negotiations price

whether data data whether effectiveness and/or and/or effectiveness

Not mentioned Not Conditional on on Conditional Yes Yes Yes Effectiveness, cost- Effectiveness, Yes AIFA

upon study protocol. study upon

based on results on based adherence to agreed- to adherence

3- Decision-making Decision-making 3- gap highlighted and and highlighted gap

study design study addresses evidence evidence addresses

2- Consultation on on Consultation 2- delivered sufficiently sufficiently delivered

whether data data whether evidence gap evidence cost-effectiveness

Conditional on on Conditional Yes No Yes 1- Identification of of Identification 1- Effectiveness and/or and/or Effectiveness Yes HAS

N/A No IQWiG N/A N/A N/A N/A N/A

N/A No* NICE N/A N/A N/A N/A N/A

N/A No* TLV N/A N/A N/A N/A N/A

decision-making of RWD? of for RWD for implemented? CRS Aims CRS CRS Procedure CRS Preference for RWD for Preference Agency

collection collection

Preference Preference

Impact of RWD on on RWD of Impact CRS CRS HTA HTA

Involvement in in Involvement

context of CRS per agency per CRS of context the in use RWD on policies of Summary - 5 Table

evidence hierarchies. evidence

of principles the from deviance necessitate

small patient populations (e.g. orphan diseases) does not not does diseases) orphan (e.g. populations patient small

circumstance of of circumstance the that stipulates agency 1 Contrastingly,

data available thus could be used for decision-making. decision-making. for used be could thus available data

only source of of source only the be may RWD diseases), orphan (esp.

2 agencies stipulate that in cases where RCT data is sparse sparse is data RCT where cases in that stipulate agencies 2 • No significant differences further. differences significant No •

considering evidence inclusion for decision-making. decision-making. for inclusion evidence considering statistical methods for use of RWD in submissions. in RWD of use for methods statistical

advises against deviating from evidence hierarchies when when hierarchies evidence from deviating against advises document included detailed guidance on on guidance detailed included document the addition, In

non-RCT evidence from decision-making. One agency agency One decision-making. from evidence non-RCT and aggregated data have been used in decision-making. decision-making. in used been have data aggregated and

exclusion of valuable valuable of exclusion the preclude not should hierarchies individual patient-level data (IPD), non-comparative IPD IPD non-comparative (IPD), data patient-level individual

strict evidence hierarchies in guidelines by stating that such such that stating by guidelines in hierarchies evidence strict used in technology appraisals, detailing that comparative comparative that detailing appraisals, technology in used

2 agencies explicitly recognise limitations in adhering to to adhering in limitations recognise explicitly agencies 2 • comprehensive list of RWD RWD of list comprehensive a published recently agency One •

Summary of differences: of Summary Summary of differences: of Summary

evidence (c.f. RCT) (c.f. evidence scientific questions at hand. at questions scientific the given

Lack of clarity on RWD impact in in impact RWD on clarity of Lack • case of conflicting conflicting of case the of which RWD and collection methods should be justifiable justifiable be should methods collection and RWD which of

where RCT are difficult to conduct (e.g. rare diseases) rare (e.g. conduct to difficult are RCT where collected nor methods for collection. However, However, collection. for methods nor collected choice choice the

For some agencies, impact of RWD may be higher in cases cases in higher be may RWD of impact agencies, some For Ű Ű Agencies do not specify which kind of RWD should be be should RWD of kind which specify not do Agencies •

conclusions on treatment effects treatment on conclusions use data. use

RWD can be used to supplement/confirm RCT-based RCT-based supplement/confirm to used be can RWD Ű Ű epidemiological data, natural history of disease or resource resource or disease of history natural data, epidemiological

Other Domains: RWD can be used to provide evidence on on evidence provide to used be can RWD Domains: Other • RWD will be regarded as more circumspect. more as regarded be will RWD

Conclusions regarding causal effects that are based on on based are that effects causal regarding Conclusions Ű Ű explored and documented. and explored

contextual factors: contextual head comparisons. However, biases related to RWD must be be must RWD to related biases However, comparisons. head

Impact of RWD on decision-making differs according to to according differs decision-making on RWD of Impact • effects when RCT evidence is absent on specific head-to- specific on absent is evidence RCT when effects

Treatment Effects: RWD can be used to inform on treatment treatment on inform to used be can RWD Effects: Treatment • lower quality level than RCT data. RCT than level quality lower a at

evidence-based medicine. Hierarchies consistently rank RWD RWD rank consistently Hierarchies medicine. evidence-based RWD also welcome. also RWD

All agencies adopt evidence hierarchies in accordance with with accordance in hierarchies evidence adopt agencies All • All sources of data are welcomed in submissions. Implies Implies submissions. in welcomed are data of sources All • discussions (IRD) discussions

Summary of commonalities: of Summary Summary of commonalities: of Summary Initial reimbursement reimbursement Initial

Context RWD Accepted/Requested RWD RWD Appraisal RWD

three contexts. three the within appraisal RWD and requested or accepted RWD for policies in differences and similarities of Summary - 6 Table

Policies for RWE use in HTA 3

Policies for RWE use in HTA 63 3

No significant differences. significant No

No significant differences. significant No Summary of differences: of Summary

Summary of differences: of Summary

probabilities for models. for probabilities

conclusions on treatment effects treatment on conclusions quality of life, adherence, epidemiological data, transition transition data, epidemiological adherence, life, of quality

RWD can be used to supplement/confirm RCT-based RCT-based supplement/confirm to used be can RWD Ű Ű Other Domains: RWD can be used to provide data on on data provide to used be can RWD Domains: Other •

RWD will be regarded as more circumspect. more as regarded be will RWD use data. use

Conclusions regarding causal effects that are based on on based are that effects causal regarding Conclusions Ű Ű source for costs data (direct, and indirect) and resource resource and indirect) and (direct, data costs for source

contextual factors: contextual Costs & Resource Use Data: national RWD is is RWD national Data: Use Resource & Costs • preferred preferred the

Impact of RWD on decision-making differs according to to according differs decision-making on RWD of Impact • explored and documented. and explored

consistently rank RWD at at RWD rank consistently lower quality level than RCT data. RCT than level quality lower a head comparisons. However, biases related to RWD must be be must RWD to related biases However, comparisons. head

Hierarchies of evidence adopted by HTA agencies agencies HTA by adopted evidence of Hierarchies • effects when RCT evidence is absent on specific head-to- specific on absent is evidence RCT when effects

Treatment Effects: RWD can be used to inform on treatment treatment on inform to used be can RWD Effects: Treatment • largely accepted. largely

RWD use to inform parameters other than treatment effects is is effects treatment than other parameters inform to use RWD • RWD is directly requested by HTA agencies for PEA. for agencies HTA by requested directly is RWD • analysis (PEA) analysis

Summary of commonalities: of Summary Summary of commonalities: of Summary Pharmacoeconomic Pharmacoeconomic

Context RWD Accepted/Requested RWD RWD Appraisal RWD

- continued - 6 Table

burden of data collection on applicant. on collection data of burden

remaining 2 agencies lay lay agencies 2 remaining The registries. indication or product

One agency often actively participates in, or initiates its own own its initiates or in, participates actively often agency One •

this effect, yet 1 does not. does 1 yet effect, this

three agencies: 2 agencies have guidelines to to guidelines have agencies 2 agencies: three the between

degree of guidance available for applicants varies varies applicants for available guidance of degree The •

within each scheme. each within

type of data requested requested data of type the influence discussions) re-pricing

Differing aims of CRS (effectiveness vs. cost-effectiveness vs. vs. cost-effectiveness vs. (effectiveness CRS of aims Differing •

6 HTA agencies implement CRS. implement agencies HTA 6 the of 3 Only •

Summary of differences: of Summary

providing specific information. specific providing

Existing national RWD sources and relevance for for relevance and sources RWD national Existing Ű Ű

No significant differences. significant No and cannot answer. cannot and

Which scientific questions different study designs can can designs study different questions scientific Which Ű Ű Summary of differences: of Summary

Agencies provide practical guidance for applicants on: applicants for guidance practical provide Agencies •

RWD delivery for re-assessment of relative effectiveness. relative of re-assessment for delivery RWD agreed upon by all parties. all by upon agreed

3- Agreement on study protocol and date of collected collected of date and protocol study on Agreement 3- research conducted adheres to to adheres conducted research the That Ű Ű protocol protocol the

scientific quality, feasibility and relevance. and feasibility quality, scientific evidence gaps identified; gaps evidence

2- Assessment of study proposal to collect data for for data collect to proposal study of Assessment 2- research conducted delivers delivers conducted research the That Ű Ű answers to to answers the

reimbursement discussions reimbursement consideration when designing designing when consideration study protocol; study the

1- Identification of evidence gaps during initial initial during gaps evidence of Identification 1- That applicants take practical guidance available into into available guidance practical take applicants That Ű Ű

conditions: similar processes for 2 agencies: 2 for processes similar schemes (CRS) schemes

impact of RWD collected rests on on rests collected RWD of impact The • following following the RWD requested in any scheme is case-specific but follows follows but case-specific is scheme any in requested RWD • reimbursement reimbursement

Summary of commonalities: of Summary Summary of commonalities: of Summary Conditional Conditional

Context RWD Accepted/Requested RWD RWD Appraisal RWD

- continued - 6 Table

Policies for RWE use in HTA 3 Policies for RWE use in HTA 65 3

3 the context of of context differences differences advantages, advantages, candidate for for candidate aims of the the a implementation implementation the the effectiveness and and effectiveness candidate for CRS in CRS in for candidate the a the possibility for CRS, since since CRS, possibility for RWD differently. Another Another RWD differently. a specific dossier will inevitably dossier will inevitably specific the RWD needed for all 3 agencies. RWD 3 agencies. all needed for a various agencies would overlap or or overlap would agencies various the the standard dossier or as dossier or as standard a subtle differences between between differences subtle the 3 contexts analysed. For example, while RWD use for while RWD use for example, For analysed. 3 contexts same contexts. An important example relates to RWD to important An relates example same contexts. the the product lifecycle (8, 40, 41). For instance, in in instance, For 41). 40, (8, productlifecycle the dossier is submitted as dossier is submitted multitude of challenging questions when developing strategies for for strategies when developing questions challenging of multitude the a concept of RWD use for HTA. Conventionally, hierarchies automatically automatically hierarchies Conventionally, of RWD HTA. concept use for the 6 agencies within within 6 agencies the type of insights provided by RCT by validity) (efficacytype provided data and internal insights of high with data manner with which these different assessors would be required to appraise to appraise RWD in required would be assessors different which these with manner effectiveness and pharmacoeconomic components of components pharmacoeconomic and effectiveness 6 agencies differed between between differed 6 agencies different countries; whether questions raised by raised whether questions countries; different Policies for RWD accepted or requested and RWD appraisal for REA of drugs differed differed of drugs REA RWD and for appraisal or requested RWD for accepted Policies Inter-context policyif be an issue may variations Meanwhile, variations between agencies’ policies may present marketing authorization marketing authorization present may policies agencies’ between variations Meanwhile, in documents featured prominently agencies HTA by adopted evidence of Hierarchies the DISCUSSION between between IRD was accepted but not explicitly recommended, its use was recommended by agencies agencies by recommended its use was recommended, explicitly but not accepted IRD was (relative) of REA: parameters on numerous evidence CRS. RWD PEA and provide for may data. cost and data use resource data, epidemiological effects, treatment on whether depend Policies for RWD accepted or requested and RWD appraisal for REA of drugs adopted by by adopted drugs REA of RWD and for appraisal requested or RWD for accepted Policies the downgrade all RWDall exploring without downgrade differ; and consider if one study would suffice to collect to suffice would study if one consider and differ; agencies’ schemes, procedures for conducting CRS, as well as agencies’ involvement in RWD in involvement agencies’ as conducting well as CRS, for procedures schemes, agencies’ collection schemes varied. CRS in use to provide data on treatment effects for IRD in situations where it may be difficult to be difficult it may where for IRD in situations effects on treatment data provide use to others this acceptable, deem some agencies While orphan diseases). (e.g. conduct RCT’s whereby differed CRS policies for Similarly, it. against advise explicitly pharmacoeconomic components of HTA dossiers submitted to an agency are examined agency an examined to submitted dossiers are HTA of components pharmacoeconomic appraise assessors who subsequently different two by offer that agencies in itself factor presents compounding the the the in registry from data, forms of and effectiveness on safety data RWD (long-term different pragmatic from outcomes or patient-reported claims databases from data use resource evidence generation across across generation evidence holders (MAH) with with (MAH) holders through hierarchies such implement agencies Several and interviews assessed. transcripts hierarchies GRADE)Although evidence (42). quality of evidence (e.g. classification for tools are evidence-basedthey in whether well-established debatable is it roots medicine, have to applicable CRS. Bearing these points in mind, one can argue that standardising standardising that argue one can BearingCRS. in mind, points these any single on RWDof policies decision-making be difficult within use for may in practice agency. HTA or claims registries patient (e.g. sources RWD of different and relevance disadvantages do not address hierarchies such evidence importantly, More databases). CRS, MAH’s would need to question whether their product their whether question as qualifies need to would MAH’s CRS, platform platform a probability probability confirmation confirmation a the implementation implementation relevance of using of using relevance other “For stated: review of agency review the a the RWD as the the benefit from such data.” This implies This implies data.” such from benefit product, that kind of data are robust kind robust that product, are data of the the 3 agencies implementing CRS were asked about asked about CRS were implementing 3 agencies issues mentioned above (53-55). above mentioned issues the the latter theoretically yielding outcomes more relevant for for relevant more outcomes yielding theoretically latter the European network for HTA (EUnetHTA) may provide provide may (EUnetHTA) HTA network for European reality of how RWD is used in practice may differ from policies and policies and from RWD differ of how reality practice in may is used low impact on decision-making within that agency, in contrast to to impact contrast in low on decision-making agency, that within The The a the rigid framework of evidence hierarchies due to pragmatic reasons (e.g. (e.g. reasons pragmatic to due hierarchies of evidence framework rigid harmonised set of policies on RWD use for HTA would provide MAH’s with with MAH’s provide would harmonisedRWDHTA on policies of set for use the focus for future research. future for focus mixed-methods approach was used that included included used that was approach mixed-methods A a the lack of harmonisation of policies for RWD use in REA of drugs may discourage RWD discourage for of policies may REA of drugs lack of harmonisation in use expectation you have at initial discussions..”, whereas whereas initial discussions..”, at have expectation you impact RWD of in decision-making contradicting re-assessment, at they displayed increasing trend of new (oncology or orphan) drugs granted conditional marketing marketing conditional or orphan) granted drugs of new (oncology trend increasing ability to plan alternative evidence generation pathways which rely less on RCTon less rely which trials, pathways generation evidence alternative ability plan to re-evaluation of of pricing and reimbursement The The In addition to studying differences in policies for in policies drugs between of use in REA RWD In differences studying addition to the of the Therefore, enough.” all 6 HTA for on RWD gathered information was policies all available that ensure To agencies, Strengths should be should that RWDthat quite has for discussions on aligning RWD policies. EUnetHTA has recently published position papers papers position published has recently RWD EUnetHTA on aligning discussions policies. for (non-RCT)on additional pilot for proposals finalising is and REA for generation evidence some of projects address which will agency asked if their requests or When accepts in practice policies is important.of these law by forced are We those data. accept we course “Of stated: representative one HTA RWD, conclude to have don’t but we data those accept to HTA purposes (49-52). HTA clinical trialsclinical (PCT)). to also refers literature of body increasing An studies; on real-world and more MAH’s from collecting or analysing RWD for HTA purposes (8, 40, 41). Therefore, it may be be it may Therefore, purposes 41). (8, 40, RWD collecting HTA or analysing from for MAH’s on guidance RWDon policies align provide and to Europe within agencies HTA for useful importantespecially is This of light in practicalaspects RWDof collection analysis. and the RCTIII phase than rather outcomes, surrogate or II data on phase based authorisation (46-48). data the data from PCT’s for more generalizable and translatable evidence on real-world outcomes outcomes on real-world evidence and translatable generalizable more for PCT’s from data This can agencyin reflected guidelines. always not was on this topic guidance yet (43-45) agencies some decision-making.Furthermore, in evidence valuable excluding in result abandon may provide even and others for CRS), conduct to or difficult are RCT’s where in situations be necessary it may Therefore, on such aspects guidance 36). 32, (27, 31, methodological be could hierarchies evidence of rigid implementation how consider to agencies HTA for use of effective enable RWD in decision-making to adapted processes. websites, academic literature and stakeholder interviews. This minimised minimised This interviews. stakeholder and academic literature websites, others. When representatives from 2 of from representatives When others. views. One stated: “You can’t really rely on them. You can use You on them. rely really can’t “You One stated: views. the different contexts and agencies, determining if differences extend to extend if differences determining and agencies, contexts different

Policies for RWE use in HTA 3 Policies for RWE use in HTA 67 3

impact of selection of interviewees’ interviewees’ the the interview and by interview and by the the “Big Four” jurisdictions Four” “Big existence of CRS schemes, schemes, of CRS existence the the three contexts assessed: IRD, PEA and CRS. CRS. PEA and IRD, assessed: contexts three PubMed search and stakeholder interviews interviews and stakeholder search PubMed the the other hand, gathering information from several several from information gathering other hand, the agencies’ official position. We attempted to account account to attempted We position. official agencies’ topic on RWDtopic in REA of drugs and use manner by which RWD is appraised for decision-making RWD which by manner for is appraised the implementation of RWD use in CRS schemes is different within within of RWDimplementation is different use in CRS schemes the the the result, valuable information from documents written by TLV or AIFA may may or AIFA TLV by written documents from information valuable result, novelty of novelty impact of excluded information was minimal. Furthermore, TLV published published TLV Furthermore, minimal. was information impact excluded of representative overview of all European policies on RWDpolicies overview use in REA of of all European representative a a use of RWD in REA. On manner for for manner the the agencies selected only represent those vested within vested those selectedagencies represent only the the opportunity of inviting colleagues they deemed relevant to to opportunity relevant deemed they colleagues of inviting lack of harmonisation of policies on RWD use for REA of drugs may present MAH’s MAH’s present on RWD of policies drugs may REA of harmonisation lack of for use information available for analysis varied between agencies. Language capabilities Language capabilities agencies. between varied analysis for available information The The multitude of challenging questions when they consider collecting and using RWD for collecting using RWD and when they consider questions of challenging multitude for involved researchers meant that Swedish and Italian documents were excluded from from excluded and Italian were documents Swedish that meant researchers involved the a agencies and jurisdictions included, this sample was deemed as relevant for an initial initial an for as relevant deemed jurisdictions and agencies was sample this included, The The analysis. As As analysis. agencies examined. agencies

It can be argued that data gathered during only reflect interviews may gathered It data that can be argued The The the CONCLUSION with sources through agency website searches, agency searches, through website sources the ensured that that ensured English several published affiliates 38) and AIFA 33, on REA (26, guidelines English numerous articlesacademic on RWD in Italian use practice (17-21). have been overlooked. Moreover, not all agencies published guidelines that specifically specifically that guidelines not all agencies published Moreover, been overlooked. have on focus opinion, rather than represent than represent rather opinion, participants approached all selective participants, sampling of providing this through for with as well as as well Individual agencies’ policies regarding RWD accepted or requested and appraisal of of and appraisal or requested RWD accepted regarding policies Individual agencies’ RWDdrugs vary REA of for across notably the of the policyon RWD analysis Europe. drugs in in REA of use during provided information Additionally, institute. per interviewingperson one than more policy to from and academic publications that with documents interviews compared was sources. data between alignment ensure and PEA parameters numerous for acceptable being largely agency, given within any varies Moreover, IRD. for effects treatment informing CRS but not for Limitations that mean automatically not does this included, were agencies HTA European 6 Although provided we drugs. of important information being excluded from analysis. Moreover, Moreover, analysis. from of importantbeing excluded information documents for analysis, data abstraction, and coding was conducted independently by two two by independently conducted was abstraction, data and coding analysis, for documents authors. different Additionally, differences are present between each agencies’ policies on RWD use for IRD, on policies for IRD, RWD use agencies’ each between present are differences Additionally, example, For PEA and CRS. and 2 agencies with pioneering roles in cutting-edge European HTA initiatives. However, However, initiatives. HTA cutting-edge in pioneering with and 2 agencies roles European considering starting point for discussions and starting and discussions for point a result, MAH’s may be discouraged to use RWD for HTA. Therefore, HTA HTA Therefore, use RWD HTA. to for be discouraged may MAH’s result, a suitable platform for HTA agencies to achieve this. achieve to agencies HTA for platform suitable agencies within Europe may collaborate to align policies on RWD and provide guidance on on guidance on RWD policies align to and provide collaborate may Europe within agencies practical aspects and papers position published RWD of Recently collection analysis. and provide may EUnetHTA by project proposals future HTA purposes. As As purposes. HTA a

Policies for RWE use in HTA 3 Policies for RWE use in HTA 69 3

Italian 32-Item 32-Item the a Italian Post-Marketing ItalianPost-Marketing The The European Union and international international Union and European Medicines Agency (AIFA) Experience,” Experience,” MedicinesAgency (AIFA) Entela Xoxi et al., “ al., et Xoxi Entela 5, Programming Pharmaceutical Registries,” 57-60. (2012): 1/2 no. of and Challenges “Feasibility S. Bonini al., et Drugs: on IndependentResearch Allison Tong, Peter Sainsbury, and Jonathan Jonathan and Sainsbury, Peter Tong, Allison Reporting for Criteria “Consolidated Craig, (COREQ): Research Qualitative Shannon, E. Hsieh and Sarah Hsiu Fang Content “Three Qualitative to Approaches 15, Health Research Qualitative Analysis,” 1277-1288. (2005): 9 no. glossary Updated “GetReal. A. Makady, 2015. “ terms. common of of definitions https://www.imi-getreal. from Available eu/Portals/1/Documents/Publications/ D1.3%20GetReal%20Glossary%20-%20 Update.pdf. “EVIDENCE Oyinlola Oyebode et al., FOR REIMBURSEMENT REQUIREMENTS ACROSS OF PHARMACEUTICALS Journal International of EUROPE,” 31, Care in Health Assessment Technology 59-67. (2015): 1-2 no. and New Perspective “New Tomino, Carlo Regulation Trial in Clinical Challenges di superiore dell’Istituto Annali in Italy,” 19-21. 1 (2011): 47, no. sanitá Peter Debaere, EU Coordination in International Coordination EU Debaere, Peter 2015 in Gx Forums Institutions: & Process Policy Macmillan,2015). Palgrave ed.: Verlin Knudand Katie Erik Jorgensen Laatikainen, Routledge on handbook the power: performance, policy, institutions: 2012). Routledge, JA3 (2016-2020) “EUnetHTA EUnetHTA, http:// from Available 2016. Partners,” Lead eunethta.eu/activities/joint-action-3/ jointaction31/eunethta-joint-action-3- 2016-2020#involved_organisations. Groups,” Checklist Interviews for Focus and Quality Journal in Health International for 349-357. (2007): 6 no. 19, Care

18. 19. 20. 15. 16. 17. 13. 14. 11. 12.

Regulator’s Regulator’s Evolution of of Evolution a the growing Need,” 2015. 2015. Need,” growing A European Parliament and of and of Parliament European Next Step in Next Step the the Health Decision Maker Perspective. Health Decision Maker Perspective. Council,” 2015. Available from http:// from Available 2015. Council,” ISPOR Real World Data Task Force Report,” Report,” Force Task Data World ISPOR Real Efficacy-Effectiveness Gap: Drug Approval,” Clin Pharmacol Ther 91, 91, Ther Pharmacol Clin Drug Approval,” 426-437. (2012): 3 no. Anna Vassall et al., “Incorporating Demand Demand “Incorporating et al., Vassall Anna Economic into and Supply Constraints and LowGÇÉIncome in Evaluations Health MiddleGÇÉIncome Countries,” 95-115. S1 (2016): 25, no. Economics Indicators: Leading Composite “OECD OECD, and Component Points Turning Reference http:// from Available 2016. Series,” www.oecd.org/std/leading-indicators/ indicatorsreference leading oecdcomposite series.htm. component and turning points L. Annemans, M. Aristides, and M. Kubin, and M. Kubin, Aristides, M. L. Annemans, Data: “Real-Life and “Review of Policies A. Makady, 2015. Data,” Real-World on Perspectives http://tinyurl.com/p2qq9xh. from Available Vicki Fung et al., “Using Medicare Data Data Medicare “Using et al., Vicki Fung Research Effectiveness Comparative for J Manag Am Opportunities and Challenges,” 488. (2011): 7 no. 17, Care What Are We Talking About?,” Epidemiology, Epidemiology, About?,” Talking We Are What (2013). 3 no. 10, Health Public and Biostatistics World Real “Using Garrison et al., P. Louis Decisions: and Payment Coverage for Data The (2007): 326-335. 5 10, no. in Health Value Perspective on Addressing Variability of of Variability Addressing on Perspective Drug Reviews Nature Drug Response,” 495-506. 7 (2011): 10, no. Discovery Licensing: “Adaptive et al., H. G. Eichler Taking Miriam and Sturkenboom, Romio, Silvana From Data “Real-World Corrao, Giovanni the the ec.europa.eu/health/files/eudralex/vol-1/ dir_2001_20/dir_2001_20_en.pdf. “Bridging Eichler et al., Hans Georg the European Commission, “Directive 2001/20/ “Directive Commission, European ec of Available from http://www.ispor.org/news/ from Available articles/oct07/rld.asp.

REFERENCE LIST REFERENCE 7. 6. 5. 4. 3. 2. 1. 10. 9. 8. Academy Academy Current DrugCurrent the joint meeting meeting joint Association of of Association a Pharmaceutical Pharmaceutical The The the Cancer Drugs Fund: Fund: Drugs Cancer the The The new model,” Available from from Available new model,” the Swedish Pharmaceutical Pharmaceutical Swedish Academy of Medical Sciences, “Real “Real of Medical Sciences, Academy British Pharmaceutical Industry,” 2016. 2016. Industry,” British Pharmaceutical The The Summary evidence: world of 2015 by held on 17 September of Medical and Sciences the https://www.acmedsci. from Available ac.uk/viewFile/56cab22108cf9.pdf. Zorginstituut Nederland. Richtlijn voor Richtlijn voor Nederland. Zorginstituut evaluaties economische van het uitvoeren 17-11-2015. in de gezondheidszorg. Nederland. Zorginstituut post- études “Les de Santé, Autorité Haute santé de inscriptiontechnologies les sur et médicaux dispositifs, (médicaments, http://tinyurl. from Available 2015. actes),” com/orw8cw2. Procedure Nederland. Zorginstituut geneeskundige zorg toelating voorwaardelijke Nederland. Zorginstituut 19-6-2015. 2015. och läkemedelsförmånsverket. Tandvårds- The Tandvårds- 2007. System. Reimbursement och läkemedelsförmånsverket. NHS “ England, “ al., et Mestre-Ferrandiz Jorge College voor Zorgverzekeringen, “Leideraad “Leideraad Zorgverzekeringen, voor College Uitkomstenonderzoek,”. voor läkemedelsförmånsverket. och Tandvårds- economic for guidelines General from evaluations och Tandvårds- 24-4-2003. Benefits Board. Läkemedelsförmånsverket. in Choices de Santé. Autorité Haute 2012. Evaluation. Economic Methods for Santé. de Autorité Haute Data to Inform Estimates of Treatment Treatment of Estimates Inform to Data Appraisal: Technology in Effectiveness Individual Patient Comparative Methods for Health Institute for National 2015. Data. Excellence. and Care Transition to to Transition https://www.england.nhs.uk/ourwork/ cancer/cdf/#. US and Responding to Paradigm: Development of Comparative Evidence Demands for European Office Effectiveness,” and Relative Effectiveness (2014). of Health Economics

40. 41. 36. 37. 38. 39. 34. 35. 33. 32.

Tool To To Tool A Nationwide Nationwide methods of The The the Use of Observational Use The The Quality of Evidence When New New When Quality Evidence of Zorginstituut Nederland. Beoordeling Beoordeling Nederland. Zorginstituut praktijk. en 2015. de wetenschap stand van Nederland. Zorginstituut Care and Health Institute for National Excellence. Haute Autorité de Santé, “General Method Method “General Santé, de Autorité Haute 2015. Technologies,” Health Assessing for http://www.has-sante. from Available fr/portail/upload/docs/application/pdf/ general_method_eval_techno.pdf. pharmaceutical products.,”. pharmaceutical Care and Health Institute for National Guide to Excellence. Wirtschaftlichkeit Institut und für Qualität Allgemeine Gesundheitswesen. im Methoden. 22-4-0015. Tandvårds- och läkemedelsförmånsverket, och läkemedelsförmånsverket, Tandvårds- when applying companies for “Guide and pricing subsidies for for Institute National 2013. appraisal. technology 3-7-2015. Excellence. Care Health and for David Glynn et al., “How to Improve Improve to “How Glynn et al., David the on Conditional Funded Are Treatments and Effectiveness of Collecting Evidence & Journal servicesof health research Safety,” 71-72. 2 (2016): policy 21, no. “Études MaryseLapeyre-Mestre et al., Quels Pharmaco-Épidémiologiques: Niveaux Atteindre?,” Les Comment Et De Preuve 241-246. 4 (2013): no. 68, Th+®rapie Support Decision-Making Healthcare And in Value Generation,” Evidence Real-World A693. 7 (2015): 18, no. Health Data Real-World “How George, Elisabeth HTA,” in Evidence Scarce for Compensate und Fortbildung Zeitschrift für Evidenz, (2016). im Gesundheitswesen Qualit+ñt Jayne Taylor et al., “METHODOLOGICAL “METHODOLOGICAL et al., Taylor Jayne VALUE THE IN EVALUATING CHALLENGES Journal International of OF REGISTERS,” 30, Care in Health Assessment Technology 28-33. (2014): 01 no. European Journal of Clinical Investigation Journal Investigation of Clinical European 69-86. 1 (2009): no. 40, “ DegliL. al., et Esposti Database. Osmed Health-Db

31. 30. 29. 28. 27. 26. 25. 24. 23. 22. 21.

Policies for RWE use in HTA 3 Policies for RWE use in HTA 71 3

Development and and Development practical guide,” 2015. 2015. practicalguide,” the A appropriate study design for primary for study design appropriate Leveraging of Real-World Data Data Real-World of Leveraging the case for action,” 2015. Available from from Available 2015. action,” case for research arising from HTA reports,” Available Available reports,” HTA arising from research http://eunethta.eu/sites/5026. from fedimbo.belgium.be/files/EUnetHTA%20 Position%20Paper%20on%20study%20 design_0.pdf. EUnetHTA, “Position paper on how to best best to on how paper “Position EUnetHTA, recommendations research formulate HTA primary arising from for research http://eunethta. from Available reports,” eu/sites/5026.fedimbo.belgium.be/files/ EUnetHTA%20Position%20Paper%20 on%20research%20recommendations.pdf. decide to paper on how “Position EUnetHTA, on Additional for Pilot Protocol “Core EUnetHTA, http:// from Available Generation,” Evidence www.eunethta.eu/sites/5026.fedimbo. belgium.be/files/news-attachments/ eunethta_core_protocol_pilot_for_aeg.pdf. ABPI, “Demonstrating Value with Real Real with Value “Demonstrating ABPI, Data: World http://www.abpi.org.uk/ from Available our-work/library/guidelines/Pages/real- world-data.aspx. Research: Data “Real-World al., et Foltz D. A http://assets1.csc.com/health_services/ downloads/CSC_Real_World_Data_ Research.pdf. “Optimizing et al., Berger L. Marc the Improve to in Health 18, Value of Medicines,” Use 127-130. (2015): 1 no. Designed Clinical Trials,” Journal of Clinical Journal of Clinical Trials,” Clinical Designed 1053-1058. (2010): 10 no. 63, Epidemiology

54. 55. 53. 51. 52. 50.

Conditional Conditional PRECIS-2 Tool: PRECIS-2 Tool: the The The International Society International the Effectiveness of Patient-Centered Patient-Centered of Effectiveness journal of Nick Freemantle and Thomas Strack, “Real- Strack, Thomas and Nick Freemantle New of Medicines Effectiveness World Appropriately by BeShould Evaluated A. Kaaniche, A. Troubat, and A. Sherwood, A. Sherwood, and Troubat, A. Kaaniche, A. Marketing Authorisation “Does Conditional France, in Access Influence Market health: in Value Germany?,” and England, the J. Hoekman et al., “Use of “Use Hoekman J. et al., for Pathway Marketing Authorization Clinical Medicines Europe,” in Oncology 98, Therapeutics & Pharmacology 534-541. (2015): 5 no. Anthony Delitto, “Pragmatic Clinical Trials: Trials: Clinical “Pragmatic Delitto, Anthony Another or Just Implementation Opportunity, 137. 2 (2016): no. 96, Therapy Physical Fad?,” “ Kirsty et al., Loudon (2015). Purpose,” for Fit Are That Trials Designing “Does Conditional I. Lipska al., et in Drugs New Oncology for Approval in Health Differences to Lead Europe Decisions?,” Assessment Technology 98, Therapeutics & Pharmacology Clinical Geonnotti K Wang W Peikes D, “Using “Using D, W Peikes Wang Geonnotti K Pragmatic Test to Trials Clinical the MedicalModelsHome Real-World in Research Healthcare Agency for Settings,” 2013). and Quality, 489-491. (2015): 5 no. Gordon H. Guyatt et al., “GRADE: an Emerging an Emerging “GRADE: et al., Guyatt H. Gordon on Rating and QualityConsensus of Evidence BMJ : British Recommendations,” of Strength 924-926. (2008): 7650 MedicalJournal no. 336, for Pharmacoeconomics and Outcomes and Outcomes Pharmacoeconomics for A559. 7 (2015): 18, no. Research

49. 48. 45. 44. 43. 42. 47. 46.

SECTION Real-World Evidence Use in Practice III

CHAPTER

Using Real-World Data in Health Technology Assessment (HTA) Practice: 4 A comparative study of 5 HTA agencies

A. Makady1,2 , A. van Veelen2, Jonsson P.3, Moseley O.4, A. D’Andon5, A. de Boer2, H. Hillege6, O. Klungel2, W. Goettsch1,2.

1. The National Healthcare Institute (ZIN), Eekholt 4, 1112 XH, Diemen, the Netherlands. 2. Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Universiteitsweg 99, 3584 CE, Utrecht, the Netherlands. 3. The National Institute for Health and Care Excellence (NICE), Level 1A, City Tower, Piccadilly Plaza, Manchester, M1 4BT, Manchester, the United Kingdom. 4. The Scottish Medicines Consortium (SMC), Healthcare Improvement Scotland (HIS), Delta House (8th floor), 50 West Nile Street, G1 2NP, Glasgow, Scotland. 5. La Haute Autorité de Santé (HAS), 5 avenue du Stade de France, 93218 Saint-Denis La Plaine Cedex, Paris, France. 6. Department of Epidemiology, University Medical Centre Groningen, Broerstraat 5 9712 CP, Groningen, the Netherlands.

Published in PharmacoEconomics. Citation is as follows: Makady A, van Veelen A, Jonsson P, Moseley O, D’Andon A, de Boer A, Hillege H, Klungel O, Goettsch W. Using Real-World Data in Health Technology Assessment (HTA) Practice: A Comparative Study of Five HTA Agencies. PharmacoEconomics. 2017 Dec 6:1-0. standardized data-extraction used standardized was form A Netherlands (ZIN). the ABSTRACT Conclusion It estimate used to mostly was RWD REAs. than Inin CEAs higher was inclusion general, Differences in CEAs. effectiveness predict long-term to or in REAs prevalence melanoma time RWD for use over trends no visible However, use of RWD. agencies’ between emerged observed. were Results 25. in present were CEAs 52 reports REAs; All contained retrieved. reports were Fifty-two RWD melanoma prevalence. estimate RWD to (54%); mainly of 52 REAs 28 included in was and/ effectiveness long-term extrapolate to mainly in 22 of 25 (88%) of CEAs; included was regarding RWD agencies between emerged Differences drugs. costs drug-related or identify SMC and NICE, whereas on prevalence RWD evidence cited for and IQWiG ZIN REAs; use in for RWD use in REAs trend No visible RWD effectiveness. drug cited HAS additionally use for observed. was time over and CEAs Methods websites from retrieved reports were and 31.12.2016 01.01.2011 between published HTA (HAS), 5 jurisdictions: (SMC), Scotland England(NICE), representing of agencies France and (IQWiG) Germany to extract information on RWD inclusion for both REAs and CEAs. extract RWD on both REAs to for information inclusion Background randomised from evidence based on conventionally are decisions Reimbursement external validity validity. low but internal high often which have trials (RCTs) controlled effectiveness relative complimentary for evidence provide (RWD)data may Real-world examines This study (CEAs). assessments and cost-effectiveness (REAs) assessments melanoma drugs of (HTA) Assessment Technology in Health RWDwhether is incorporated in time. across and agencies between RWD use differences agencies, HTA European by

Use of RWE in HTA practice 4 Use of RWE in HTA practice 77 4 more more a standard standard remaining remaining respective respective the the the setting of RCTs (14, (14, of RCTs setting the treatment of melanoma of melanoma treatment the purposes of this article, we purposes of this article, we the routine setting of health care practice(9, practice(9, care health setting of routine large heterogeneity of patients in clinical in clinical of patients heterogeneity large good fit to demonstrate causality (13-15). causality (13-15). to demonstrate good fit a efficacy–effectiveness gap (13). Therefore, Therefore, efficacy–effectiveness(13). gap lack of head-to-headlack of in RCTs. comparisons extent to which an intervention does more does more which an intervention to extent the the the the the treatment of metastatic (or non-operable) melanoma (or non-operable) of metastatic melanoma treatment relationship between relative effects and effects relative between relationship the the intervention versus its comparators (11). its comparators versus intervention external validity of RCTs is relatively low (14-17). Consequently, Consequently, (14-17). low is relatively external validity of RCTs the the most serious and fatal form of skinof Its form seriousmost fatal cancer(1). and been has incidence market representing 4 novel mechanisms of action, thereby increasing increasing of action,mechanisms thereby 4 novel market representing European Commission(8). However, each European jurisdiction decides European each However, Commission(8). European the the the marketing authorisation holders of drugs. For For marketing holders of drugs. authorisation desired results and when provided under under provided and when results desired the Regulatory approval of new therapeutics in Europe is centralized, with decisions being being decisions with is centralized, Europe in of new therapeutics Regulatory approval Evidence on drug effectiveness informing HTA submissions is conventionally derived derived conventionally submissions is HTA informing effectiveness drug on Evidence Real-world data (RWD), defined here as data collected outside as data (RWD), data defined here Real-world high degree of internal validity, making them validity, of internal high degree INTRODUCTION increasing, largely caused by increased exposure to ultra-violet radiation (1-3). Primary (1-3). radiation ultra-violet to exposure increased by caused largely increasing, after metastasis, tumour However, excision. surgical by most often removed tumours are becomes and pharmacotherapy is often feasible, no longer excision surgical Melanoma is option (1, 4). According to literature, before 2011, dacarbazine was dacarbazine2011, was before literature, to According 4). (1, option good than harm, when compared to one or more alternative interventions for achieving achieving for interventions alternative more one or to good than harm, when compared the chemotherapeutic of choice for for of choice chemotherapeutic for drugs multiple 2011, (henceforth 6). Since melanoma)(5, entered have and on assessments based conventionally and pricing, reimbursement on drug nationally Assessment Technology Health national by conducted evidence available of appraisals in sometimes (REAs), assessments effectiveness relative involve These agencies. (HTA) submitted evidence based on (CEAs), assessments with cost-effectiveness combination by examine that as assessments REAs define issued by issued by treatment options substantially (1, 7). (1, 7). substantially options treatment generalizable picture of treatment effects in clinical practice (18). In contrast, using RWD for using for practice (18). clinical contrast, in RWD effects In picture treatment of generalizable 10). Meanwhile, CEAs examine examine CEAs Meanwhile, 10). costs of implementing implementing of costs from randomised clinical trials (RCTs) (12). Due to their design characteristics, RCTs have have RCTs characteristics, design their Due (12). to clinical trials (RCTs) randomised from a extrapolation of drug efficacy to drug effectiveness in clinical practice is difficult. This practice effectiveness clinical difficult. is in efficacy of drug extrapolation to drug as to referred discrepancy is frequently However, due patient randomisation, inclusion and exclusion criteria and regulated regulated criteria and and exclusion inclusion randomisation, patient due However, protocols, follow-up to benefit additional potential their and drugs melanoma in advances recent despite rely that and CEAs REAs of results in interpreting challenges still face agencies HTA patients, factorsas such due to RCTs from on evidence RCTto practice compared and populations drugs existing or novel of estimates effectiveness inform be used to theoretically could 15), supporting thereby RCT practice, in clinical numerous RWD from derived can be evidence. observational electronic and studies health records including disease registries, sources, allocation, treatment non-randomised characteristicsspecific of Due15). (14, to (e.g. RWD provide it may populations), patient and broader follow-up patient longer groups groups majority appraisal appraisal the the the effectiveness and and effectiveness the Netherlands. However, due due However, Netherlands. study by Kleijnen et al (8). Kleijnen (8). by al study et the the context of use (i.e. whether for REAs REAs whether for of use (i.e. context the study proceeded with 5 agencies. 5 agencies. with study proceeded latter criterion ensured that that ensured criterion latter use of RWD in HTA practice. Specifically, it it Specifically, practice. use of RWD in HTA the The The the subsequent question remains whether and how how and whether remains question subsequent A melanoma indication, publication dates between between dates publication indication, melanoma a of December 2016 and availability of at least 3 reports, reports, least 3 of at and availability 2016 December of st potential known and unknown confounders in known unknown and potential confounders new report. 31 quality of data present or may complicate research across across research complicate or may quality present of data a the the the lack of interoperability across RWD sources with different database database different with RWD across sources lack of interoperability number of issues cited here such as propensity scoring techniques techniques scoring propensity such as here of issues cited number a the standardized data-extraction open-ended standardized containing and closed questions form authors’ inability to read Polish reports, Polish inability read to authors’ a of January of and 2011 st 1 Currently, RWD is used in drug development to examine natural history of diseases, history natural examine of diseases, to RWD development used in drug is Currently, Data extraction from compiled reports was performed independently by AM and AvV reports extractionData compiled performed and AvV was AM from by independently HTA reports on 7 new melanoma drugs (ipilimumab, vemurafenib, dabrafenib, dabrafenib, vemurafenib, reports (ipilimumab, on 7 new melanoma drugs HTA the retrospective, comparative analysis of HTA reports (henceforth reports) on melanoma of HTA analysis comparative retrospective, METHODS or CEAs) (23). This study aims to examine examine study aims to This (23). or CEAs) and instrumental variable techniques (to address confounding) or multiple imputation imputation multiple or confounding) address (to techniques variable and instrumental own their with come techniques these (19-21).However, data) missing address (to methods 21). (19, limitations and assumptions one should combine RWD with RCT data for REA and CEA for HTA purposes (22).In brief, purposes (22).In brief, RWD one should combine HTA with RCTfor and CEA REA for data on insights much-needed supply RWDalthough potentially may delineating clinical treatment pathways, determining costs and resource use associated use associated resource and costs determining pathways, treatment clinical delineating (23).Previous comparators with associated outcomes health examining and treatments with RWD on policies decision in that and appraisal assessment has demonstrated research on depend and agencies making vary HTA between cost-effectiveness of new drugs in practice, its incorporation into analyses and subsequent subsequent and analyses into its incorporation practice, in drugs of new cost-effectiveness clear-cut. is not makingdecision HTA for whether RWD melanoma drugs and of examines and CEAs REAs included in is in Europe. agencies 5 HTA purposes by of RWD its intended for using of included agencies had conducted assessments for each drug. Each resubmission or or resubmission Each drug. each for assessments had conducted agencies of included as categorized was addendum published by 3 different agencies, per drug. drug. per agencies, 3 different by published infrastructures may affect affect may infrastructures Methods used were comparable to those presented in presented those to comparable Methodswere used different datasets, respectively (18). Some statistical methods have been developed in an in an developed been respectivelyhave methods (18). statistical Some datasets, different address to attempt of patients being compared (18). Moreover, other practical other aspects in data such as missing Moreover, (18). compared being of patients and RWD sources decision making presents new methodological and analytical challenges. For example, due due example, For challenges. analytical and makingdecision methodological new presents may effects treatment in estimated confounding allocation, treatment non-randomized to in an imbalance to due occur to drugs was performed. Six HTA agencies representing 6 European jurisdictions were selected selected jurisdictions were 6 European representing agencies performed.drugs was Six HTA SMC - Scotland; NICE - England; they make full reports since inclusion, available: publicly for - ZIN and – Poland; – Germany; IQWiG AOTMiT HAS - France; A cobimetinib, trametinib, nivolumab and pembrolizumab) were retrieved from agency from retrieved were and pembrolizumab) nivolumab trametinib, cobimetinib, Inclusion were: criteria websites. the

Use of RWE in HTA practice 4 Use of RWE in HTA practice 79 4

third, third, inter- reports a copy of of copy the a validity of validityof panel of 5 of 5 panel the Fleiss’ kappa Fleiss’ a parameter(s) parameter(s) the the the score of 1 indicates perfect indicates of 1 score source as cited in as cited source a the agencies included, included, agencies remaining reports were equally equally reportsremaining were dataset, whether data for specific specific for data whether dataset, the the the authors are aware of overlap between between of overlap aware are authors panel members. Panel members then members then Panel members. panel frequency thereof were recorded. It is recorded. were frequency thereof frequency thereof were recorded. Then, Then, recorded. were frequency thereof reason for inclusion (i.e. (i.e. inclusion for reason nature of of nature the the the the final check.final the the a negative opinion on validity opinion of RWD negative use a intended parameter (henceforth RWD parameter appraisal) intended positive opinion on validity of RWD use and source. on validity opinion of RWDpositive source. and use neutral opinion on validityopinion on of RWDneutral use and source. the a a 5 respective agencies was consulted (see Appendix (see Appendix consulted was agencies 5 respective the data extracted from reports of HTA agencies and results of of results and agencies reports extracteddata from HTA of authors registered registered authors The The the results of analyses. Panel members subsequently received received subsequently members Panel analyses. of results source of RWD. Subsequently, agencies’ appraisals of appraisals agencies’ Subsequently, of RWD. source score of 0 indicates poor agreement and and agreement poor indicates of 0 score the a sources chosen for chosen for sources the data extraction form was missing and where to find it in reports, as well as reports, as well find it in as to extraction where missing and data was form the the assessment. assessment. and source. and source. identifying statement Neutral: neutral. or negative as positive, cannot clearly be identified that Unknown: statement despite appraisal in inclusion RWD regarding statement no Not identified: the Positive: statement identifying identifying statement Positive: identifying statement Negative: frequency of RWD inclusion in REAs and CEAs was recorded separately. Subsequently, Subsequently, separately. frequency recorded RWD of was and CEAs in REAs inclusion analyses mentioned below were mailed to to mailed were below mentioned analyses source(s) of RWD of and used per parameter source(s) parameter(s) for which RWD and used for was parameter(s) modified dataset and analyses results for results analyses and dataset modified following algorithm: following definitions of registries and observational and registries of definitions (14). studies • • • To verify whether data extracted from reports extracted data whether from RWD on verify RWD inclusion, scoring appraisal To • • ANALYSIS method, whereby whereby method, To measure agreement within data extraction and scoring performed by AM & AvV, extraction data within performedand scoring & AvV, AM agreement by measure To rater reliability (IRR) was calculated twice in 2 different rounds. In each round, authors authors round, In each rounds. twice in 2 different calculated (IRR) was reliability rater reports reports 4 randomly-selected 2 for extracted Appendix (see independently from data using compared extraction were questions closed for Authors’ per round). by extraction open-ended compared for was questions Authors’ (24). agreement established, was IRR Once researcher. independent and results of analyses accurately reflectpractice in accurately of analyses and results divided amongst both authors. amongst both divided senior assessors representing representing senior assessors 3 for panel members). members). panel 3 for the RWD and use the was examined by identifying corresponding statements in reports and scoring them using in reports using them scoring and statements corresponding identifying by examined was the of questions on their feedback indicated if, for example, reports were missing from from missing reports were example, for if, indicated the e.g. “SEER registry data” was recorded as “registry”, whereas “MELODY observational study” observational study” “MELODY whereas “registry”, as recorded was registry“SEER data” e.g. However, study”. “observational as recorded was it informed) and it informed) the The The the (DEF; see Appendix 1). Inclusion of RWD in REAs and CEAs was examined separately. When When Inclusion 1). separately. Appendix see examined of RWD was and CEAs (DEF; REAs in examined: RWD 2 aspects included, were was important that note to

st 31 the model’s model’s frequency frequency the the RWD included distribution of distribution of REAs and CEAs and CEAs REAs the end of The The the 65 reports, 52 were 65 reports, were 52 detailed summarydetailed of the first round to 0.80 in in to 0.80 round first the a the of January and 2011 majority of st 1 The The the detailed summarydetailed of a new drug. new drug. the 7 drugs on agencies’ websites. Of websites. 7 drugs on agencies’ the general analysis mentioned above, potential variation in RWD variation use potential above, mentioned analysis general new drug beyond RCT trial duration to estimate its long-term RCT long-term its beyond drug new estimate to trial duration agencies was as follows: ZIN (n=2), HAS (n=8), NICE (n=10), SMC (n=13) SMC (n=13) (n=10), NICE HAS (n=8), ZIN (n=2), follows: as was agencies the the inclusion of all 52 reports (see Appendix 4 for full list). full list). reports of all 52 inclusion for 4 Appendix (see the 5 agencies was examined by comparing RWD inclusion in REAs and CEAs RWD comparing and CEAs by examined in REAs inclusion was 5 agencies the the IRR was calculated twice and improved from 0.60 in 0.60 from twice and improved calculated IRR was second round, corresponding to substantial agreement between AM & AvV (24). (24). & AvV AM between agreement substantial to corresponding round, second studies used to provide RWD on effectiveness and safety, see Table S1 in Appendix 5. Appendix in Table S1 see RWD safety, and on effectiveness provide used to studies effectiveness (7/28) and safety (6/28) of (6/28) and safety (7/28) effectiveness Finally, an analysis of RWD inclusion in REAs and CEAs combined for all compiled compiled all for combined of RWD CEAs analysis and an REAs in inclusion Finally, In to addition The The melanoma estimate used to mainly RWD was and REAs (54%) 28/52 included in was RWD was included in 22/25 (88%) CEAs and was primarily used to extrapolate extrapolate primarilyRWD and was to used (88%) CEAs in 22/25 included was specific time-point beyond trial duration (e.g. 10 or 15 years), while national statistics data data statistics years), while national 10 or 15 time-pointspecific (e.g. trial duration beyond RESULTS indicated for melanoma. All 52 were published between between published were 52 All melanoma. for indicated for estimation of melanoma prevalence/incidence originated from registries. Additionally, Additionally, registries. from originated prevalence/incidence of melanoma estimation for observational were from data studies and claims databases databases, statistics national observational from mainly derived was or safety RWD effectiveness used. for included For phase I/II studies. non-randomized and/or studies 65 reports were identified for identified 65 reports were reports per publication year was performed to examine potential changes in RWD changes performed was reports potential examine inclusion year to per publication amongst amongst agency. per time. over of December 2016. NICE, HAS, and IQWiG published at least one report for all 7 drugs, all 7 drugs, one report least at published for and IQWiG HAS, NICE, 2016. of December for allowing prevalence and/or incidence (28/28 REAs). Additionally, RWD was used to estimate RWD estimate used to was Additionally, REAs). (28/28 and/or incidence prevalence the reports across included. were the For 1. Table see of RWD and RWD use per parameter source, and IQWiG (n=19). All reports included REAs. However, IQWiG and HAS reports and did not IQWiG However, reports All (n=19). REAs. included and IQWiG in reports and ZIN. It SMC located NICE, of is important were CEAs 25 In total, CEAs. include ZIN reports as part that assessments initial note entailed to reimbursement of conditional research outcomes as such CEAs, and included sections such, they REAs As beyond schemes. only this study for RWD prospective collection. However, for proposals effectiveness of effectiveness the was used to extrapolate overall survival from that point forwards until until survival forwards point that from overall extrapolate used to was effectiveness (21/22 CEAs). Additionally, RWD was included to estimate costs associated associated costs estimate to was included RWD Additionally, CEAs). (21/22 effectiveness using quality utilities and determine use (8/22) of life resource estimate with drugs (12/22), derived effectiveness long-term included RWD that estimate CEAs to All (4/22). information In further reports, some this was registries. supported from RWDdata national by from survival until overall extrapolate used to registry was In data case, that databases. statistics a time horizon. Costs were estimated using data from claims databases, observational studies claims databases, from data using estimated were Costs time horizon.

Use of RWE in HTA practice 4 Use of RWE in HTA practice 81 4 Source Registry (n=21) statistics National (n=12) database database Claims (n=10) Observational study (n=4) Cost-of-illness study (n=1) Observational study (n=7) database Claims (n=4) Registry (n=1) Quality-of-lifestudy (n=3) Registry (n=1) Frequency 21 12 8 4 Cost-effectiveness assessment Cost-effectiveness

Reason Reason Long-term effectiveness Costs use Resource Quality-of- data Life for inclusion for

Source Registry (n=22) statistics National (n=9) database Observational study (n=6) (n=2) Claims database Observational study (n=6) Non-randomized / II trial I (n=6) Phase Registry (n=1) Non-randomized / II trial I (n=4) Phase Observational study (n=3) outcome of RWD appraisal in REAs and CEAs. For 16 of 49 (33%) 49 (33%) of 16 For RWD of CEAs. and REAs in outcome appraisal the negative appraisal of RWD decision- primarily appraisal was by caused in REAs negative The The Frequency 29 7 6

Relative effectiveness assessment effectiveness Relative - Parameters for whichReal-World Data is included (RWD) and RWD sources used per parameter Figure 1 shows 1 shows Figure Prevalence/ incidence Effectiveness Safety for inclusion for Reason Reason (including frequency). Table 1 or cost-of-illness studies. Data sources used for resource use and quality-of-life parameters quality-of-life and use parameters resource for used sources Data or cost-of-illness studies. 1. Table in presented are and 27 of 58 (32%) parameters for which RWD was used in REAs and CEAs respectively, respectively, RWD which CEAs and REAs in used for was parameters (32%) 58 of 27 and identified statements appraisal Meanwhile, identified. be could statements no appraisal unknown mostly was and (51%) (25/49 outcome appraisal that indicated or CEAs in REAs parameters, (16%) (12%) and 9/58 (6/49 or negative respectively) (31%) parameters, 18/58 respectively). No Yes (n=2)

inclusion of of inclusion (n=2) =

0% 0%

20% 40% 40% (n The The 28) (n=22/25)

2016 5% 2% (n=8) ZIN 16% 31% 47%

nature of RWDof its and nature CEA

= in

the HAS (n 15) RWD 2015 (n=13) ZIN 4% 4% CEAs) 23% 43% 26%

= +

(n=19) IQWiG (n 5 agencies. For example, NICE reports NICE example, For 5 agencies. 14) REAs

inclusion of RWD in CEAs exceeded 75% in 75% in of RWDinclusion exceeded CEAs in the of

statistical methods applied for extrapolation extrapolation for applied methods statistical The The 6) 2014

SMC = the

(n=13) 52 reports were initial assessment reports assessment initial within 52 reports were (n number

(n=10) SMC = (n=28/52)

0% 5%

50% 20% 25% the (n 2013 2% 2%

51% 33% 12% REA NICE

in 0% 77% 47% 38% 0%

Year low reliability of RWD reliability observationallow use from to studies 22%22%56% 6% 12% 6% 0% 0% 58% 9% 9% 45% 0% 0% 100% (n NICE (n=10) estimation of effectiveness and/or safety more frequently. In In frequently. more and/or safety effectiveness of estimation 13) the RWD the validity of RWD use and sources chosen when included in REAs and CEAs and validity in REAs included when chosen of RWD sources use and 0% 90% 80% 70% 60% 50% 40% 30% 20% 10% 1) 2012

the 100% 0% = 90% 80% 70% 60% 50% 40% 30% 20% 10%

data 100%

(n effectiveness life

‐ Use

0% 90% 80% 70% 60% 50% 40% 30% 20% 10% of 2011 ‐ current study only 2 of study only current 100% term ‐ included included identified - Appraisal of - Appraisal inclusion of RWD over time in REAs and CEAs combined varied per year, ranging ranging year, per varied combined and CEAs in REAs time of RWDinclusion over

the 0% 90% 80% 70% 60% 50% 40% 30% 20% 10%

Not Long Costs

Resource Quality Safety Effectiveness Prevalence/incidence Not Unknown Not Positive Negative Neutral negative appraisal of RWD in CEAs was primarily due to decision-makers’ uncertainties of RWD primarily decision-makers’ appraisal was due to in CEAs negative 100% In The The Inclusion of RWD in REAs differed between between Inclusion of RWD differed in REAs Percentage Percentage Percentage Percentage Figure 1 Figure all years (2011-2016), displaying no visible variation in trend. variation no visible displaying (2011-2016), all years However, ZIN. by those published namely (CRS), schemes reimbursement conditional makers’ perceptions of perceptions makers’ regarding extrapolations of long-term effectiveness. However, in some reports in some was it However, effectiveness. of long-term extrapolations regarding uncertainties these whether discern to pertainedsolely difficult to estimate clinical effectiveness due to biases associated with observational data. Similarly, observational with Similarly, associated biases to data. due effectiveness clinical estimate the RWD in REAs and CEAs over time is shown separately in Figures S1 and S2 in Appendix 5. No 5. No S1 and S2 in Appendix in Figures separately is shown time RWD over and CEAs in REAs RWD for visible in REAs. inclusion was trend from 1/1 reports (100%) in 2011 to 17/28 reports (61%) in 2016 (Figure 4). reports 1/1 17/28 reportsfrom to in 2011 (100%) 2016 (Figure (61%) in cited RWD in 10/10 (100%) REAs, while SMC reports cited RWD in 3/10 (33%) (Figure 2). ZIN 2). ZIN SMC reports while RWD RWDcited cited REAs, (100%) in 10/10 (33%) (Figure 3/10 in SMC and NICE, while melanoma prevalence, RWD estimating cited mainly for and IQWiG RWD HAS cited for use found in was above inclusion CEAs; inclusion in RWD were differences no notable contrast, RWD pertained mainly CEAs in ZIN cited drug to However, 3). (Figure all 3 agencies 75% for reports SMC and NICE in that pertainedmainly quality-of-life and whereas data, costs to estimates. use resource and effectiveness long-term associated biases or in combination with biases or in combination associated effects. of long-term Figure S1 - Inclusion of RWD in CEAs across the 3 agencies and reasons for inclusion per agency. Figure 3 - Inclusion of RWD in REAs and CEAs (combined) over time. Figure 2 - Inclusion of RWD in REAs across the 5 HTA agencies and the reasons for inclusion per agency. Figure 2 - Inclusion of RWD in REAs across the 5 HTA agencies and the reasons Figure 1 - Outcome of appraisal of RWD when included in REAs and CEAs. Figure 1 - Outcome of appraisal of RWD

Use of RWE in HTA practice 4 Use of RWE in HTA practice 83 4 No Yes No Yes (n=2) (n=2)

(n=2) = (n=2)

= 0% 0%

0% 0%

20% 40% 40% 20% 40% 40% (n (n 28) 28) (n=22/25) (n=22/25)

2016 5% 2% 2016 (n=8) ZIN 5% 2% (n=8) ZIN 16% 31% 47%

16% 31% 47%

CEA CEA

= = in in

HAS

HAS (n (n 15) 15) RWD RWD 2015 2015 (n=13) ZIN (n=13) ZIN 4% 4% 4% 4% CEAs) 23% 43% 26% CEAs)

23% 43% 26%

= + = +

(n=19) IQWiG

(n=19) IQWiG (n (n 14) 14) REAs REAs

of of

6) 2014 6) 2014

SMC = SMC =

(n=13)

3 agencies and reasons for inclusion per agency per inclusion for reasons and 3 agencies (n=13) (n (n reasons for inclusion per agency inclusion for reasons number number

the (n=10) SMC = (n=28/52) (n=10) SMC =

(n=28/52) 0% 5%

0% 5%

50% 20% 25% 50% 20% 25% the (n 2013 (n 2% 2%

2013 2% 2%

12% 51% 33% 12% 51% 33% REA NICE REA

= NICE

in 0% 77% 47% 38% 0% in

Year 0% 77% 47% 38% 0% 22%22%56% 6% 12% 6% 0% 0% 58% 9% 9% 45% 0% 0% 100%

(n Year discussion sectiondiscussion below. NICE 22%22%56% 6% 12% 6% 0% 0% 58% 9% 9% 45% 0% 0% 100% (n NICE (n=10) (n=10) 13) 13) the RWD RWD 0% 0% 90% 80% 70% 60% 50% 40% 30% 20% 10% 90% 80% 70% 60% 50% 40% 30% 20% 10% 1) 2012 1) 2012

100% 0%

100% 0% = 90% 80% 70% 60% 50% 40% 30% 20% 10% =

90% 80% 70% 60% 50% 40% 30% 20% 10%

data 100% data 100%

(n (n effectiveness life effectiveness

life

‐ Use ‐ Use 0%

90% 80% 70% 60% 50% 40% 30% 20% 10% of 2011 90% 80% 70% 60% 50% 40% 30% 20% 10% of 2011 ‐ ‐ 100% 100% term term ‐ ‐ included identified included included identified included

- Inclusion of RWDand in REAs - Inclusion of RWD across in CEAs

0% 0% 90% 80% 70% 60% 50% 40% 30% 20% 10%

90% 80% 70% 60% 50% 40% 30% 20% 10%

respective re-assessment reports have not yet been published. We will return to to return will We been published. re-assessment not yet respective reports have Not Long Costs

Positive Negative Neutral Unknown Not Quality Resource Safety Effectiveness Prevalence/incidence Not 100% Not Long Costs

Positive Negative Neutral Unknown Not Quality Resource Safety Effectiveness Prevalence/incidence Not 100% implications of this in of implications Percentage Percentage Percentage Percentage Percentage Percentage Percentage Percentage the the Figure 2 Figure Figure 3 Figure Figure S1 - Inclusion of RWD in CEAs across the 3 agencies and reasons for inclusion per agency. Figure S1 - Inclusion of RWD in CEAs across the 3 agencies and reasons for Figure 3 - Inclusion of RWD in REAs and CEAs (combined) over time. Figure 3 - Inclusion of RWD in REAs and Figure 2 - Inclusion of RWD in REAs across the 5 HTA agencies and the reasons for inclusion per agency. per for inclusion reasons and the agencies the 5 HTA across in REAs of RWD 2 - Inclusion Figure Figure 1 - Outcome of appraisal of RWD when included in REAs and CEAs. of appraisal of RWD when included Figure 1 - Outcome Figure S1 - Inclusion of RWD in CEAs across the 3 agencies and reasons for inclusion per agency. Figure S1 - Inclusion of RWD in CEAs across the 3 agencies and reasons for Figure 3 - Inclusion of RWD in REAs and CEAs (combined) over time. Figure 3 - Inclusion of RWD in REAs and Figure 2 - Inclusion of RWD in REAs across the 5 HTA agencies and the reasons for inclusion per agency. per for inclusion reasons and the agencies the 5 HTA across in REAs of RWD 2 - Inclusion Figure Figure 1 - Outcome of appraisal of RWD when included in REAs and CEAs. of appraisal of RWD when included Figure 1 - Outcome No Yes (n=2)

statistical statistical (n=2) =

0% 0%

40% 20% 40% the (n review examined examined review 28) (n=22/25)

previous review of of review previous varying number of 2016 5% 2% (n=8) ZIN a The The 16% 31% 47%

CEA

the = in

HAS (n 15) RWD 2015 (n=13) ZIN 4% 4% CEAs) 26% 43% 23%

= +

(n=19) IQWiG fact that not all agencies conducted fact conducted not all agencies that (n 14) the REAs

nature of appraisal statements was mostly mostly was statements appraisal of nature of

6) 2014 the

SMC =

(n=13) (n number

(n=10) SMC = (n=28/52)

0% 5%

25% 20% 50% (n 2013 2% 2%

12% 51% 33% REA NICE

= in 0% 77% 47% 38% 0%

Year 22%22%56% 6% 12% 6% 0% 0% 58% 9% 9% 45% 0% 0% 100% (n NICE (n=10) 13) extent with which RWD was included and its appraisal in HTA which RWD with extent in HTA its appraisal and included was RWD 0% the 90% 80% 70% 60% 50% 40% 30% 20% 10% 1) 2012

100% 0% = 90% 80% 70% 60% 50% 40% 30% 20% 10%

data 100%

(n effectiveness life

‐ Use 0%

90% 80% 70% 60% 50% 40% 30% 20% 10% of 2011 ‐ 100% term ‐ findings summarised above coincide well with results from from results with well coincide summarisedfindings above identified included included - Inclusion of RWD in REAs and CEAs (combined) over time - Inclusion over of RWD(combined) and CEAs in REAs

90% 80% 70% 60% 50% 40% 30% 20% 10%

Positive Negative Neutral Unknown Not Safety Effectiveness Prevalence/incidence Not Quality Resource Costs Long Not 100% The The Inclusion of RWD in REAs varied somewhat between agencies. In contrast, little variation variation little InInclusion contrast, of RWD agencies. between somewhat varied in REAs Percentage Percentage Percentage Percentage DISCUSSION policies on RWD use amongst 6 HTA agencies (4 of which were included in this study) thus thus study) this in included which were of (4 agencies RWD on policies 6 HTA use amongst RWD current (23). with policies use in practice is in line that indicating CEAs. Therefore, interpretation of differences in RWD use between agencies and across in and across agencies use between RWD differences of interpretation Therefore, CEAs. with caution. be made time must in RWD inclusion in CEAs was observed. Analysis of differences in in both RWD inclusion of differences in RWD observed. Analysis was in CEAs inclusion However, 2011 and 2016. between trends no identifiable time revealed over and CEAs REAs by complicated time were across agencies and between analyses unknown or negative. Reasons for negative appraisals were manifold, often relating often relating manifold, were appraisals negative Reasonsfor unknown or negative. RWD with as biases associated of as well awareness decision-makers’ to estimates. it in effectiveness incorporate used to approaches as as well reportstotal per agency and per year, reports of 7 melanoma drugs from 5 different agencies. Results demonstrate an overall overall an Results demonstrate agencies. reports 5 different of 7 melanoma drugs from common is more inclusion whereby in and CEAs, REAs between inclusion RWD difference and/ RWD (54%). prevalence melanoma REAs (88%) than in CEAs informed mainly included of RWD used Sources costs in CEAs. and effectiveness and long-term in REAs or incidence observational national studies, registries, included: and varied those parameters inform to often not on RWD were appraisal Statements databases. and claims databases statistics identified, When and CEAs. in REAs found This study examined study examined This Figure 4 Figure Figure S1 - Inclusion of RWD in CEAs across the 3 agencies and reasons for inclusion per agency. per for inclusion and reasons agencies the 3 across RWD in CEAs of S1 - Inclusion Figure Figure 3 - Inclusion of RWD in REAs and CEAs (combined) over time. of RWD in REAs and CEAs (combined) Figure 3 - Inclusion Figure 2 - Inclusion of RWD in REAs across the 5 HTA agencies and the reasons for inclusion per agency. Figure 2 - Inclusion of RWD in REAs Figure 1 - Outcome of appraisal of RWD when included in REAs and CEAs.

Use of RWE in HTA practice 4 Use of RWE in HTA practice 85 4

lack lack ideal ideal the the setting of routine of routine setting the rate of RWD inclusion was of RWD was inclusion rate relatively minor role in HTA, in HTA, minor role relatively a the different agencies, and differed and differed agencies, different possible reason could be be could reason possible broader patient population than than population patient broader A a the drug, there might be insufficient time time insufficient be might there drug, results of this study are pragmatic clinical clinical pragmatic study are of this results design of such trials is fraught with many many with trialssuch of fraught is design a the Patient-Centred Outcomes Research Institute Institute Research Outcomes Patient-Centred the the absence of guidance on systematic approaches approaches on systematic guidance of absence generalizability of results for different settings such settings such different for generalizability results of advantages of PCTs use in HTA decision-making may decision-making may in HTA use of PCTs advantages the the The The role for RWD in HTA practice. For example, RWD inclusion example, For practice. RWD HTA for in role time of initial HTA assessments. Since these assessments assessments these Since assessments. HTA time of initial International Society for Pharmacoeconomic and Outcomes and Outcomes Society Pharmacoeconomic International for the context analysed. For example, agencies’ policies iterate that that iterate policies agencies’ example, For analysed. context the product of these ongoing collaborations will be deemed useful will be deemed useful product of these ongoing collaborations the question why RWD currently plays RWD plays why question currently the the the Innovative Medicines Initiative GetReal MedicinesInitiative ConsortiumInnovative (IMI-GetReal) (26). the preferred source for relative effectiveness estimates in CEAs. CEAs. in estimates effectiveness relative for source preferred past ten years, RWD use in drug development and healthcare decision-making healthcare and RWD has development drug in use years, ten past inclusion, analysis and interpretation of RWD for HTA purposes. Moreover, HTA HTA Moreover, purposes. RWD of HTA for interpretation and analysis inclusion, the broader set of outcome measures than RCTs, are embedded in embedded are than RCTs, measures of outcome set broader the a preferred source for data on effectiveness estimates of drugs. Consequently, use RWD Consequently, of drugs. estimates effectiveness on data for source preferred European Network of HTA (EUnetHTA) (27). Further dialogue amongst HTA agencies is is agencies amongst HTA dialogue Further (27). (EUnetHTA) Network of HTA European the In These results raise raise results These One potential source of RWD in found not source One potential multitude of initiatives have explored possibilities for incorporating RWD incorporating in decision- for possibilities explored have of initiatives multitude markedly on depending policies on RWD use in REAs, CEAs and conditional reimbursement schemes (CRS), (CRS), schemes reimbursement conditional and CEAs on RWD policies in REAs, use between somewhat differed policies that concluding RWD use is welcome in REAs to provide incidence or prevalence data but that RCT’s remain remain RCT’s that but data or prevalence incidence provide to REAs in RWD is welcome use the policies Meanwhile, in REAs. appraised to be negatively likely more is effectiveness for specific demanded for and even accepted, RWD that is largely CEAs in inclusion iterate RCT’s that also iterate policies use). However, resource and costs treatment (e.g. parameters remain Research (ISPOR) Task Force on RWD (15), Force Task (ISPOR) Research gained increasing attention both in scientific literature and grey literature (25). Moreover, (25). Moreover, literature grey and literature in scientific both attention increasing gained a making. Examples include making. include Examples lowest in 2016. 2016. in lowest effectiveness. drug to relating parameters for especially RWD at of robust available of soon after regulatorytake place approval or observational collect marketing RWD holders to authorisation registries for through be factor Another could studies. for (PCORI) and (PCORI) little attention, increased despite that be argued it may this study, from Based on findings to regards with changed has In fact, time. in reports proportionally did not increase over agencies have only recently begun collaborating on strengthening understanding of of understanding on strengthening collaborating begun only recently have agencies RWD further generating developing and for methods analytic study designs appropriate as IMI-GetReal such initiatives and through sources of RWD synthesis different from for the necessary that ensure to by decision-makers. by represent may they that imply of PCTs elements design Several trials (PCTs). balance between RCTs and RWD: often they include RCTs between balance RCTs, clinical practice and may include initial randomization followed by cross-over between arms arms between cross-over by followed randomization include initial practice and may clinical (14, 28). analyses on interim based seem straightforward at first sight. However, However, sight. first at straightforward seem strategic choices that may impact may that choices strategic value value authors authors study is study is a The The incremental incremental The The authors also also authors respective re-respective the the issues associated issues associated the the 52 reports were initial initial reports52 were choice of comparators of comparators choice case with PCTs whereby whereby PCTs with case counter-argument is that that is counter-argument lifetime horizon in health in health horizon lifetime the the need for RWD. For example, example, For RWD. need for the a a other hand, other hand, re-assessment of drugs (e.g. to to re-assessment of drugs (e.g. the the intervention within routine clinical clinical routine within intervention the use of scope of this study, we recommend recommend we of this study, scope the need for RWD to provide data on (long- data RWD provide need for to the the other hand, hand, other the clinical population. Meanwhile, guidelines guidelines Meanwhile, population. clinical the broad spectrum of design choices that make make that spectrum choices broad of design question whether current modelling methods modelling methods whether current question need for RWD, quantitative research is required is required research quantitative RWD, for need reality that it is neither ethical nor feasible to to nor feasible ethical it is neither that reality a the critical role in critical role internal and external and generalizabilityinternal of the the current study only 2 of 2 of only study current a the the the implementation of PCTs in practice is also associated with with in practice associated also is PCTs of implementation valuable source of RWD whose potential for decision makingdecision RWDof for potential whose source valuable The The a heterogeneous clinical population remains crucial for HTA purposes. purposes. HTA crucial for remains population clinical heterogeneous predictive validity of outputs from health economic models and and models economic health validity from of outputs predictive robust answer to to answer robust a a balance between between balance the impact that different effectiveness estimates can have on have can estimates effectiveness impact different that the considerable number of stakeholders define RWD strictly as data generated define stakeholders of number strictlyRWD considerable generated data as the a efficacy-effectiveness gap, potentially supplanting supplanting potentially gap, efficacy-effectiveness selection and of participating centres clinical hospitals/ the underlying distributions used to randomly sample effectiveness parameters in PSA PSA in parameters sample effectiveness underlying randomly distributions to used Theoretically, CRS provide an ideal context for incorporating RWD in HTA. RWD incorporating in HTA. for context an ideal CRS provide Theoretically, With regards to pharmacoeconomic analysis for CEA, one could argue that quantitative quantitative that argue one could CEA, for analysis pharmacoeconomic to regards With the pre-specified study protocol details such aspects of researcher intervention. intervention. such aspects details researcher pre-specified of study protocol and sensitivity analyses could supplant supplant sensitivityand could analyses term) effectiveness in effectiveness term) provide to In order conduct long-term RCTs, one could argue that that argue one could conduct RCTs, long-term sensitivity analyses (34). Although this is beyond is beyond this sensitivity Although (34). analyses play would in CRSs of RWD generated impact). or budget estimates ICER cost-effectiveness efficacy previous confirm estimates, that indicated agencies 3 by implemented CRS policies for research, previous to According abide collection data and analysis provided this context, within RWD accepted largely is pre-defined by (23). conditions In to bring to light light bring to to topic. this pursuits on future economic analyses (31-33) and given and given (31-33) analyses economic assessment reports within CRS, namely those published by ZIN. However, reports ZIN. However, assessment by those published namely CRS, within not HAS reports were examined Moreover, published. been yet not reportsassessment have are based on numerous assumptions and RCT data which may arguably also not be and RCT assumptions also not be based on numerous arguably are which may data in of drug effectiveness representative the require models increasingly economic health for techniques such as bootstrapping and probabilistic sensitivity analyses (PSA) may help help may sensitivity (PSA) analyses and probabilistic as bootstrapping such techniques on shed light On 31). (11, (ICER) ratio cost-effectiveness with as methods for modelling and sensitivity analyses may address some of some address sensitivity and may modelling analyses for methods in HTA should be further be should explored. in HTA difficult to achieve and that PCTs include include and that to achieve difficult PCTs offer may PCTs that believe such studies more or less representative of RWD (28). On representative or less such studies more and outcome measures (28). (28). measures and outcome of operationalization as such challenges numerous practice, data management across sites and monitoring across sites (28, 29). Moreover, 29). Moreover, (28, sites across and monitoring sites across management data practice, has RWD;as qualify research previous PCTs that agree unanimously stakeholders all not that shown inclusion/exclusion assignment, treatment on researchers by intervention any without often is not This (30). protocols monitoring and patient criteria are aware that that aware are a

Use of RWE in HTA practice 4 Use of RWE in HTA practice 87 4 of st final final 31 authors authors the analyses, analyses, the The The the results accurately accurately results chances of missing missing of chances 5 agencies and over and over 5 agencies the the the possible underestimation of of underestimation possible a more detailed evidence package evidence detailed that more the potential role of RWD role reports in melanoma potential consequence of inter-author differences in in differences inter-author of consequence the a issue of RWD use in HTA extends beyond HTA in in HTA extends beyond RWDissue of use in HTA Polish HTA agency (AOTMiT) be achieved not could HTA Polish the the study examined different disease areas and included reports included and areas disease different study examined varying number of reports published per agency, per year, varying per year, number of reports agency, per published 2 authors responsible for data extraction and scoring was extraction data was and scoring for responsible 2 authors randomly selected set of reports. In doing so, authors minimised minimised authors selected reports.of set In randomly so, doing the the a the decision makers. This may lead to to lead may This makers. decision total number of reports number total and including until published up the the authors recognize that that recognize authors study by Wilk et al. on RWD use by AOTMiT (35), which reported increasing use use reported which (35), increasing on RWD AOTMiT et al. by use Wilk by study authors’ inability to read Polish reports. Inclusion of AOTMiT’s reports may have reports have may reports. Inclusion AOTMiT’s of Polish inability read to authors’ the a different time period, its results are not easily comparable to those presented here. here. to those presented comparable easily not are results its time period, different IRR between IRR between comparison of RWDcomparison RWD inclusion and between appraisal the a study included all 52 reports from 5 HTA agencies’ websites in websites agencies’ reports52 all study included 5 HTA from probability that results reached were were reached results that probability inclusion of reportsinclusion by published role of RWD in decision making. In an attempt to address these shortcomings, authors authors shortcomings, these RWDof address role to making.decision Inin attempt an focus of future research once re-assessment once reportspublished. are research future of focus The The Findings generated by this study were presented to an HTA panel, consisting of 5 senior senior of 5 consisting panel, an HTA to presented this study were by generated Findings The The more informative overview of RWD use in HTA practice across Europe. Europe. practice across overview of RWD use in HTA informative more is considered by by is considered the and procedural differences in practice between agencies. For example, almost ten times ten times almost example, For agencies. in practice between differences and procedural not all agencies included ZIN. Furthermore, for than IQWiG for retrieved reportsmore were included ZIN and SMC NICE, only process; as part conduct CEAs automatically of their HTA some evidence that indicated member (PJ) one panel their reports. in Moreover, CEAs in mentioned explicitly not is CEAs and REAs for NICE RWD), (including by assessed guidance document. However, it is provided in it is provided However, document. guidance time was complicated by by complicated time was Moreover, Moreover, Europe. Therefore, future research should aim to include HTA agencies from outside Europe Europe outside from agencies HTA include to aim should research future Therefore, Europe. (PBAC)). and Australia (CADTH) Canada (e.g. identified identified in practice. However, since since However, in practice. within within due to to due provided insights on RWD use by an HTA agency within Eastern Europe, thus arguably arguably thus agency Europe, within Eastern RWD on insights HTA an use by provided a assessors representing all 5 agencies included, to verify whether whether verify to included, agencies 5 all representing assessors extraction and scoring. December 2016. Inclusion 2016. December minimised 5 agencies all all reports of for Limitations The The The Strengths part of CRS implemented in France. As such, such, As part France. in of CRS implemented represent practice within their agency thus improving their plausibility. their practice agency within their thus improving represent the measured twiceon based measured relevant information. information. relevant corresponding to to corresponding within CRS could not be assessed. To our knowledge, work is ongoing within ZIN and HAS HAS and ZIN within ongoing is work knowledge, our To assessed. be not could CRS within on RWD study no similar provided Therefore, re-assessto RWD. using drugs melanoma performed, has been agencies be should this HTA CRS across within appraisal and inclusion the main main previous previous The The focus on this this on focus the the other hand, hand, other the case study team had hypothesized hypothesized had case study team 5 agencies; some citing RWD only for RWDciting some agencies; 5 only for IMI-GetReal metastatic on study case the the the (long-term) value of drugs in clinical practice for practice for clinical of drugs in value (long-term) the treatment of metastatic melanoma in in melanoma of metastatic treatment the considerable number of new, yet expensive, drugs that have have drugs that expensive, yet of new, number considerable the use of RWD to demonstrate use of RWD demonstrate to the Future research should aim to explore RWD inclusion and appraisal within CRS RWD CRS within appraisal explore and inclusion to aim should research Future Inclusion of RWD in REAs differed between between InclusionRWD of differed in REAs This study represents spin-off work from spin-off work from represents study This previously-performed policy review. CONCLUSIONS implemented by different HTA agencies, which provide an ideal context for RWD use in HTA for in HTA use context RWD ideal an which provide agencies, HTA different by implemented disease indications. multiple and across practice, prevalence and/or incidence and others for drug effectiveness and safety. Meanwhile, no no Meanwhile, and safety. effectiveness drug for and others incidence and/or prevalence results these However found. time was RWD total in over inclusion trend distinguishable and agencies between in practices differences to caution, owing with interpreted be should varyingof reports numbers per year. published reason for inclusion in REA was prevalence and/or incidence of melanoma and in CEA CEA in and melanoma of incidence and/or prevalence was REA in inclusion for reason If reports, drugs. of new included in was effectiveness RWD long-term extrapolating for appraisal When identified, often not identified. were its appraisal regarding statements from findings with correspond results These mostly unknown was or negative. outcome a disease area could arguably hinder generalizability of results to others whereby RWD use whereby others to generalizabilityhinder results of arguably could area disease RWD inclusion investigate aim to should therefore research Future be relevant. also may thus multiple disease areas, reports or across areas disease in other HTA in and its appraisal practice. HTA broader to generalizabilityof results increasing In general, RWD was more often included in CEAs than in REAs of HTA reports. HTA of REAs in than RWD often in CEAs included Inmore was general, recently become available for for available become recently melanoma (4). Given (4). Given melanoma respectively: included all melanoma reports published per agency, explicitly distinguished distinguished respectively: explicitly reports melanoma all included agency, per published were statements appraisal where cases all registered in analyses, and CEAs REAs between agencies. for all evidence published considered and only not identified, HTA purposes in this indication would be pertinent. would purposes On in this indication HTA years based largely on (short-term) largely based efficacyyears data, that that

Use of RWE in HTA practice 4 Use of RWE in HTA practice 89 4

et al. et al. A ISPOR Real The The In W. G, A. W, Jonsson P et Jonsson P et W, G, A. W. In efficacy-effectiveness gap: EUnetHTA partner countries. partnercountries. EUnetHTA the the IMI-GetReal 2016. regulator’s perspective on addressing perspective on addressing regulator’s synopsis of methodological guidelines guidelines of methodological synopsis World Data Task Force Report. Value in in Value Report. Force Task Data World 10: 326-335. 2007; Health RH, Klungel Groenwold OH, Schmidt AF, confounding for Consortium Adjusting G. in early going postlaunch settings: models. regression logistic beyond 133-142. 27: 2016; Epidemiology Myrtveit Analyzing UH. Olsson E, Stensrud I, empirical An data: sets with missing data and methods imputation of evaluation Garrison LP, Neumann PJ, Erickson P et EricksonP et PJ, Neumann Garrison LP, Coverage for Data World Real Using al. Decisions: and Payment Infrastructure Mardekian J. D, Alemayehu Secondary in for Sources Data Requirements Journal Research. Effectiveness Comparative Pharmacyof Managed S16-S21. 17: 2011; Care Real-world T. Strack N, Freemantle medicines should of new effectiveness designed appropriately by evaluated be Journal of Clinical trials. clinical 1053-1058. 63: 2010; Epidemiology al.et JM Alvir Riaz MPP, Ali D, Alemayehu Issues and Analytical of Data, Examination Conducting Comparative Methods for Proposed Data”. “Real-World Using Research Effectiveness S3-S37. 2011; 17: Pharm J Manag Care BM et Psaty Klungel OH, Martens EP, effects intended assess Methods to al. in observational treatment of drug Journal of clinical reviewed. are studies 1223-1231. 57: 2004; epidemiology Eichler HG, Abadie E, Breckenridge Breckenridge E, HG, Abadie Eichler Bridging a Reviews Nature variability of drug response. 495-506. 10: Discovery 2011; Drug al. et N Hummel W, MakadyGoettschA, glossary Updated GetReal. of of definitions terms. common (eds). al. economic evaluations? Results from Resultsfrom evaluations? economic a used in 59-76. 34: 2016; Pharmacoeconomics

19. 20. 21. 16. 17. 18. 14. 15. 13.

economic evaluation of of evaluation economic European view on health on health view European the Systematic Literature Review. Review. Literature Systematic a A clinical breakthrough? Anti-Cancer Anti-Cancer breakthrough? clinical a health care programmes. Oxford university university Oxford programmes. care health press,2015. A, Ghabri Gerber-Grote S et E, Heintz Is there al. Kleijnen S, George E, Goulden S et Goulden Kleijnenet S E, George S, assessment effectiveness Relative al. similarities and of pharmaceuticals: in Value jurisdictions. in 29 differences Claxton K et Sculpher MJ, MF, Drummond Methods for al. HLPF. High Level Pharmaceutical Forum Forum Pharmaceutical High Level HLPF. and Conclusions Final 2005-2008. In. 2008. Recommendations. 15: 954-960. 2012; health Rubio-Rodríguez D, Blanco SDD, Pérez Pérez SDD, Blanco Rubio-Rodríguez D, Cost-Effectiveness C. M, Rubio-Terrés Advanced for Treatments Drug of Melanoma: Relative al. et TL KleijnenLipskaS, Alves I, oncology of assessments effectiveness pricing and reimbursement for medicines of Annals countries. European in decisions 27: 1768-1775. 2016; Oncology Julia F, Thomas L, Dumontet C, Dalle S. Dalle S. C, L, Dumontet Thomas Julia F, melanoma: in metastatic therapies Targeted toward Medicinal in ChemistryAgents (Formerly Medicinal Chemistry-Anti-Cancer Current 10: 661-665. 2010; Agents) Makady A, Kalf R, W. G, Lees M. Deliverable Deliverable M. Lees G, W. MakadyKalf A, R, Review: Study Case WP1 Metastatic D1.6 Melanoma. In.2016. for therapy systemic Agarwala SS. Current Expert melanoma. metastatic of review 9: 587-595. 2009; therapy anticancer Kumar P, Clark Medcine.2012. Clinical M. P, Kumar risk Azoury C, Lange R. Epidemiology, and early detection prevention, factors, North of Clinics Surgical of melanoma. 945-962. 94: America 2014; Liu Y, Sheikh MS. Melanoma: molecular Y, Liu therapeutic and pathogenesis Molecular and cellular management. 228. 6: 2014; pharmacology PharmacoEconomics 2017; 1-15. 2017; PharmacoEconomics

REFERENCE LIST REFERENCE 7. 5. 6. 4. 2. 3. 1. 12. 11. 10. 9. 8.

Review A pragmatic, pragmatic, methods of of methods a the In. Zorginstituut In.Zorginstituut Re-Analysis of Cost- the Hospital Elder Life Program to prevent prevent to Program Life Hospital Elder Excellence NIfHaC. Guide to Excellence In. Institute National appraisal. technology 2013. Excellence and Care Health for HH. Predictive Karnon Afzali J, and Validation Wierzbicka Skrzekowska-Baran N, I, Wilk Evidence World of Real Adoption al. N et public on decision-makingin processes Journal of of drugs in Poland. funding 2. 2015; Research & Outcomes Policy Health Heim N, van Stel HF, Ettema RG et al. Ettema HF, Stel van N, Heim in executing HELP! Problems MakadyBoer A, de Hillege A, H et al. (RWD)? Data Real-World Is What and of Definitions on Literature Based 2017. in Health Value Interviews.Stakeholder Briggs Claxton AH, K, Decision Sculpher MJ. evaluation. health economic for modelling E,2006. Economic in Health Handbooks het Nederland RichtlijnZ. voor evaluaties economische van uitvoeren gezondheidszorg. de in Nederland2015. Tread? to We Dare Do Effectiveness: 1111-1112. 35: 2017; PharmacoEconomics randomized, stepped wedge trial onwedge stepped randomized, the 220. 2017; 18: Trials deliriumolder patients. in

34. 35. 31. 32. 33. 30. 29. measurement of of measurement Journal of Clinical Journal of Clinical 2016. The The series. series. the Policies for Use of Real-World Data Data of Real-World Use for Policies Comparative Study of Six HTA Agencies. Agencies. of Six HTA Study Comparative et al. et al. Epidemiology 2017. Epidemiology Zuidgeest M, Goetz I, Groenwold R et al. R et al. M, Goetz I, Groenwold Zuidgeest evidence–an world trials real and Pragmatic to introduction Makady A, W. G. Review of Policies and and Review G. of Policies W. Makady A, In. Data. IMI- Real-World on Perspectives GetReal 2015. In.IMI-GetReal. 2017. objectives. Overall cycle approach 5 - Life Package Work EUnetHTA. In. Generation. 2017. Evidence improve to Landis JR, Koch GG. Landis Makady Boer de R, Ham ten A, A (HTA): Assessment Technology in Health A 520-532. 20: Health 2017; in Value data. categorical for observer agreement Biometrics 159-174. 1977; likelihood-based methods. IEEE Transactions Transactions IEEE methods. likelihood-based Softwareon 999-1013. 27: 2001; Engineering al. et E-M Didden T, Debray Hummel N, guidance, 4 Methodological Package Work case illustrative and recommendations (network) meta-analysis for studies predict real-world to and modelling participant using individual effectiveness data. aggregate and/or

28. 26. 27. 25. 24. 23. 22.

Use of RWE in HTA practice 4

CHAPTER

Implementing Managed Entry Agreements in Practice: The Dutch reality check 5

A. Makady1,2 , A. van Veelen2, A. de Boer2, H. Hillege3, O. Klungel2, W. Goettsch1,2

1 The National Healthcare Institute (ZIN), Diemen, the Netherlands 2 Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht, the Netherlands 3 Department of Epidemiology, University Medical Centre Groningen, Groningen, the Netherlands

Currently under review in Health Policy. CF the Netherlands Netherlands full procedure. full procedure. the the CF framework, focusing on on CF framework, focusing the envisioned 4 years period for 11/12 drugs. Outcomes Outcomes drugs. 11/12 period for 4 years envisioned design and implementation of CF negatively affected its its affected of CF negatively and implementation design the the 4-year coverage with evidence development (CED) framework framework (CED) development evidence with coverage 4-year a standardised data extraction form, researchers independently extracted information extracted independently information extraction data researchers standardised form, a procedure extended beyond extended beyond procedure ABSTRACT The The After drugs. for 5/12 re-assessment, data scientific insufficient provided conducted research necessitating 6 with drugs, 10/12 for advised was reimbursement of continuation discontinue to advice Notably, generation. evidence for conditions additional yet practice. healthcare Dutch in been implemented has not yet drugs 2/12 for reimbursement for expensive hospital drugs. This study aims to evaluate evaluate to study aims This drugs. hospital expensive for between 2006 and 2012; and 2012; 2006 between procedures. (HTA) Assessment Technology Health Background in applied was hospital drugs expensive (CF) of financing Conditional Conclusions However, drugs. to access but conditional quick for an option CF provided Theoretically, to aspectsnumerous related Results underwent 12 of which CF, for chosen were drugs Forty-nine learnings from incorporate should aim to schemes CED Future in practice. value on procedural, methodological and decision-making aspects from HTA reports of drugs decision-making and methodological aspects HTA from on procedural, CF. selected for Methods Using example to increase their impact practice. increase in healthcare to example

Use of RWE in HTA practice 5 Use of RWE in HTA practice 95 5 then then well- a proven a the development development the outcomes research research outcomes Healthcare Insurance Insurance Healthcare arrangements between between arrangements the treatment of early breast of early breast treatment the the (1).Three different categories of of categories different (1).Three advent of innovative, yet expensive expensive yet of innovative, advent use of MEAs as policy tools to address as policy address use of MEAs to tools the issue of unequal access to to access issue of unequal re-assessment (T=4) comprising of drugs following components: therapeutic value, value, therapeutic components: following a The The the basic healthcare package, respectively. healthcare basic the budget impact above €2.5 million/year, million/year, €2.5 impactbudget above the a so-called “ZIP code healthcare” phenomenon and public public and phenomenon healthcare” code “ZIP so-called value of information analysis)(6). Inclusion of drugs in CF Inclusion analysis)(6). drugs in CF of information of value Dutch National Healthcare Authority (NZa) was asked by asked by (NZa) was Authority Healthcare National Dutch a period of 3 years, which was eventually extended to 4 years. years. 4 extended to eventually was which period of 3 years, the a the selected expensive or orphan drugs were funded for 80% and and 80% for funded orphanor were drugs selectedexpensive flexible policy framework that incorporates both early access to to both early access policy incorporates flexible that framework a the coverage with evidence development (CED) framework (1;5). framework (CED) development evidence with coverage a Netherlands encountered Netherlands encountered national Health Technology Assessment (HTA) authority, was responsible responsible was authority, (HTA) Assessment Technology Health national the the 4-year period, ZIN would conduct would ZIN period, 4-year the National Healthcare Institute (ZIN; formerly known Institute formerly (ZIN; as Healthcare National implementation of CF and issuing eventual advice on reimbursement on behalf of of behalf on reimbursement on advice eventual issuing and CF of implementation drug or medical technology under specified conditions” conditions” under specified technology or medical drug proposed outcomes research to collect real-world evidence (RWE) on AU and CE in CE in and (RWE) evidence on AU collect real-world to research outcomes proposed MEA, specifically specifically MEA, NZa. According to ZIN guidelines (6;7), drugs nominated for CF would be included in included in be CF would for drugs nominated (6;7), ZIN guidelines to NZa. According following elements: therapeutic value, appropriate use, cost-effectiveness and budget budget and cost-effectiveness use, appropriate value, therapeutic elements: following Dutch MinistryDutch policy 2012 2006 and two between implement of Health to frameworks hospital setting, respectively. These policy frameworks were linked to to linked policy were These frameworks respectively. setting, hospital The The In 2005, After After the a a INTRODUCTION Board (CVZ)), (CVZ)), Board outcry (4). To address this, this, address To outcry (4). the within and orphan drugs drugs administered expensive (CF) of financing the conditional for the of for impact. Finally, an appraisal of all available evidence at T=4 would be performed T=4 would advise to at evidence of all available an appraisal impact. Finally, MEAs are defined based on issues they address: (i) managing budget impact, (ii) managing impact, budget (i) managing managing (ii) address: they based on issues defined are MEAs to utilization managing and (iii) cost-effectiveness, clinical and/or uncertainty to relating design their with associated are challenges numerous performanceoptimize (3).However, (2;3). debate of ongoing topics to leading and implementation, the an initial assessment (T=0an initial assessment comprising years) the this dilemma has increased globally (2). MEAs can be described as “ be described can as MEAs (2). globally increased has dilemma this or reimbursement coverage -ensure that to or providers access and payers manufacturers drug of for therapy as adjuvant trastuzumab expensive yet innovative, in hospitals between significantly varied (4). Access with HER2+ over-expression cancer to leading provinces different cost-effectiveness, budget impact analysis, and assessment of assessment and budget impact analysis, cost-effectiveness, met: 3 criteria if were warranted only was proposal (preferably including including (preferably proposal drugs and additional evidence generation (1). (1). generation evidence additional drugs and drugs, it is suggested that managed entry agreements (MEAs) may provide healthcare healthcare provide may (MEAs) entry managed that suggested it is agreements drugs, with and insurers payers and treatments, comparator available to in comparison value therapeutic additional use appropriate uncertaintiesregarding to address research for outcomes proposal defined marketingauthorisation Subsequently, practice. routine (CE) in cost-effectiveness and (AU) implement would societies and clinician clinicians hospitals, with in collaboration holders, the practice throughout routine administering Hospitals through drug expenditures of their 100% In an era of rising healthcare expenditures due to due to expenditures In of rising healthcare era an research research 4 criteria 4 criteria the the last available last available assessment of of assessment the the procedure. procedure. Scientific AdvisoryScientific the The The Netherlands has been Netherlands has been three-pronged approach three-pronged approach a the trade name, active ingredient, active ingredient, name, trade the taxonomy of MEAs and recommend and recommend MEAs of taxonomy the CF scheme. CF scheme. authors used used authors the healthcare system (7). (7). system healthcare the CF scheme, documents listing notifications of of listing notifications documents CF scheme, use of an alternative, tailored approach. tailored an alternative, use of the combination of combination appraisal of evidence at T=4 based on T=4 based at of evidence appraisal the the a trade name, active ingredient, registered indication indication registered active ingredient, name, trade T=0 and T=4 reports for finalized drugs were present. were drugs finalized reportsT=4 for and T=0 the the the CF scheme, CF scheme, ZIN website (www.zorginstituutnederland.nl). This period period This (www.zorginstituutnederland.nl). ZIN website process chartprocess of implementation of CF to date by reviewing HTA reports. In reports. In HTA reviewing by date of CF to implementation the a the the Insured Package Advisory Committee (ACP; hereafter Appraisal Appraisal hereafter (ACP; Advisory Package Insured Committee full procedure (hereafter finalized drugs) equalled 4 years. 4 equalled drugs) finalized (hereafter procedure full authors opted for for opted authors particular context. Therefore, in order to best address address best to order in Therefore, particular context. time span between published T=0 and T=4 reports for drugs that that drugs T=4 reports for T=0 and published between time span the relevant committees were consulted throughout throughout consulted were committees relevant date of CF scheme implementation (01.01.2006) and (01.01.2006) implementation of CF scheme date a the the assessment were compiled. Duplicate entries for each drug were removed removed each drug were for entries Duplicate compiled. were assessment the the the the authors endeavour to provide empirical insights to inform ongoing discussions ongoing discussions inform to empirical insights provide to endeavour authors fact that such frameworks aim to classify to aim fact such frameworks that the different documents based on documents different the reimbursement of drugs based on 4 criteria: necessity, clinical effectiveness, cost- effectiveness, clinical criteria: on 4 drugs based necessity, of reimbursement implementation of MEAs in practice. MEAs of implementation underwent Whether T=0 and T=4 assessments were conducted for all CF drugs. If not, whether If whether not, all CF drugs. for conducted were T=4 assessments T=0 and Whether communicated. transparently were assessments T=4 not conducting for reasons Whether of components all Whether Whether Whether the authors are aware of other MEA analysis frameworks proposed in literature (1;5;8) but but (1;5;8) in literature proposed frameworks other MEA analysis of aware are authors implementation of CF as per ZIN guidelines, specifically:ZIN guidelines, as per of CF implementation To subsequently evaluate evaluate subsequently To • Procedural aspects related to whether due procedure had been followed in in followed had been procedure due whether to aspects related Procedural • • • To our knowledge, no systematic evaluation of CF in of evaluation no systematic knowledge, our To the the METHODS best practices for their design, rather than to retrospectively analyse their implementation their implementation analyse retrospectively to than rather design, their best practices for within thoroughly hand, at question based on procedural, methodological and decision-making aspects outlined below. and decision-making methodological aspects below. outlined on procedural, based The to refer Committee (WAR; hereafter Assessment Committee) was responsible for for responsible was Committee) Assessment hereafter (WAR; Committee effectiveness and implementability within effectiveness registered indication and status of assessment. and status indication registered the Committee). See Figure 1 for 1 for See Figure Committee). was conducted by by conducted was A. Procedural aspects A. Procedural report assessments per year and announcements of assessment statuses were compiled compiled were statuses of assessment and announcements report per year assessments from 2017 to 2006 from corresponds to to corresponds evidence at T=0 and T=4. Meanwhile, Meanwhile, T=4. T=0 and at evidence under registered all assessments document, notifications each For (15.05.2017). document drug each For collected. CF were of and status To generate an overview an of all drugs in generate To from from on conducted since its inclusion stopped in 2012. Therefore, this article aims to evaluate this article evaluate aims to Therefore, in 2012. stopped its inclusion since conducted with gained experiences doing so, doing so, on

Use of RWE in HTA practice 5

Use of RWE in HTA practice 97 5

opinion on continued reimbursement, the Appraisal Committee was consulted. was Committee Appraisal the reimbursement, continued on opinion

† It is important to note that in some cases, the final appraisal of evidence in relation to the 4 package criteria was performed by the Assessment Commi Assessment the by performed was criteria package 4 the to relation in evidence of appraisal final the cases, some in that note to important is It † ttee, rather than the Appraisal Committee. This occurred for for occurred This Committee. Appraisal the than rather ttee,

drugs whereby appraisal was relatively straight-forward (i.e. evidence at T=4 on all 4 criteria indicated a positive opinion on continued reimbursement). However, in cases where evidence may have led to a to led have may evidence where cases in However, reimbursement). continued on opinion positive a indicated criteria 4 all on T=4 at evidence (i.e. straight-forward relatively was appraisal whereby drugs negative negative

and third components. third and

research inform the second second the inform research

• • Implementability Note: results from outcomes outcomes from results Note: the Dutch Ministry of Health of Ministry Dutch the

• • Cost-effectiveness This advice is addressed to to addressed is advice This

• • • • Budget impact analysis impact Budget Clinical effectiveness Clinical

• • assessment should be continued. be should Necessity

• • Cost-effectiveness Cost-effectiveness reimbursement package package reimbursement reimbursement package: reimbursement

from the national statutory statutory national the from assessment assessment national statutory statutory national

• • reimbursement of the drug drug the of reimbursement Appropriate use use Appropriate following 4 criteria for the the for criteria 4 following

• • advice on whether whether on advice Therapeutic value Therapeutic appraises evidence on the the on evidence appraises

ZIN officially publishes its its publishes officially ZIN components: The Appraisal Commiee † † Commiee Appraisal The

assesses the following following the assesses

Final Advice Final Appraisal (T=4) Appraisal Assessment Committee re- Committee Assessment

Re-Assessment (T=4) Re-Assessment

practice.

and CE from Dutch clinical clinical Dutch from CE and

Aim: to collect data on AU AU on data collect to Aim:

(academic) hospitals. (academic) Re-submission made Re-submission

clinician societies and and societies clinician

collaboration with clinicians, clinicians, with collaboration

authorisation holder, in in holder, authorisation

Conducted by the marketing marketing the by Conducted

Outcomes Research Study Research Outcomes

included, etc.) included,

outcome measures measures outcome CF scheme.

proposal

patient population studied, studied, population patient

outcomes research research outcomes

regarding the proposal (e.g. (e.g. proposal the regarding the

and well-defined well-defined and

made by the committee committee the by made

cost-effectiveness (CE) (CE) cost-effectiveness

Note: Recommendations are are Recommendations Note:

appropriate use (AU), (AU), use appropriate

• • Uncertainties on on Uncertainties

proposal proposal

Initial submission Initial therapeutic value therapeutic

• •

Outcomes research research Outcomes

• • Proven added added Proven

• • Budget impact analysis impact Budget

million/year

• •

Therapeutic value Therapeutic

• • Budget impact >€2.5 >€2.5 impact Budget

components:

Inclusion criteria: Inclusion

addresses the following following the addresses chart of – Process

Assessment Committee Committee Assessment

Selection of Candidates of Selection Initial Assessment (T=0) Assessment Initial

Figure 1 Figure Figure 1- Process chart for conditional financing (CF) as implemented by ZIN in the Netherlands. the in ZIN by implemented as (CF) financing conditional for chart Process 1- Figure

CF final final the specific specific the evidence evidence proposed proposed questions questions appendix. appendix. respective respective Assessment Assessment the committee’s committee’s the the the the the the The The list of list core aims of core a the to C in C in to evidence collected and its its and collected evidence A purposes of this analysis, purposes analysis, of this Assessment Committee on on Committee Assessment the the report whereby report whereby outcomes research implemented. research outcomes a the specific element of element specific the committees consulted. committees a Assessment Committee on Committee Assessment the Assessment Committee on AU and CE based based CE and on AU Committee Assessment evidence submitted at T=4 was analysed in in analysed T=4 was at submitted evidence the report components. For For report components. the quantity of critical commentary was recorded quantity critical of commentary recorded was standardised data extraction form was used to extraction to used data was standardised form the the appraisal of evidence presented at T=4 and T=4 at presented of evidence appraisal a assessment of evidence at T=0 and T=4, specifically: and T=0 at evidence of assessment The The recorded instance in in instance recorded the a the Appraisal Committee’s advice on reimbursement at T=4 in relation T=4 in relation at on reimbursement advice Committee’s Appraisal the scientific qualityscientific of Assessment Committee at T=4 deemed T=4 at Committee Assessment the the latter three points, T=0 and T=4 reports for finalized drugs were collected collected were drugs finalized reportsT=4 for T=0 and points, three latter 4 package criteria. following information from reports: date of publication of T=0 and T=4 reports, reports, T=4 T=0 and of reports:of publication from date information following the nature of conclusions made by made by conclusions of nature nature of of nature final advice published by ZIN on reimbursement of finalized drugs. drugs. finalized of reimbursement ZIN on by published advice final ZIN website. Subsequently, Subsequently, ZIN website. the the The The T=4. at submitted on evidence The to The The outcomes research at T=0 were incorporated in incorporated were T=0 at research outcomes Whether recommendations made by made by recommendations Whether Whether Whether analysis to be of sufficient scientific quality to provide conclusions for conclusions provide quality to scientific be of sufficient to analysis formulated T=0. at the standardised data extraction form was used to retrieve information from from information retrieve extraction used to data was standardised form outcomes research proposals (T=0), proposals (T=4) use assessments research outcomes appropriate cost- and • • • • • Quantity of critical commentary provided by Quantity by of critical commentary provided A critical comment was defined as defined was critical comment To address address To from from retrieve Decision-making to aspects related C. Decision-making aspects reimbursement advice, specifically: advice, reimbursement B. Methodological aspects Methodological B. to aspects Methodological related components of T=0 and T=4 reports present and T=4 reports present and T=0 of components Committee provided an objective an critique on provided Committee assessed. being effectiveness assessments (T=4) of finalized drugs. For assessments effectiveness drugs. of finalized (T=4) the a respective statements in T=4 reports. reports. T=4 in statements respective Specific attention was given to these aspects because they represent to these aspectsrepresent because they given was attention Specific scheme (i.e. to prospectively design studies to collect RWE on AU and CE) in comparison to to and CE) in comparison collect RWE studies to design on AU prospectively to scheme (i.e. performed ZIN. by HTA conventional T=4 report components. T=0 and per section and drug finalized per of Tables see which commentary collected, was for elements proposals research made on outcomes of recommendations on implementation Analysis T=4. at those T=0 to performed at was critical comparing comments by on conclusions

Use of RWE in HTA practice 5 Use of RWE in HTA practice 99 5

list of list of a T=0 and and T=0 outcome outcome status of of status following following drugs were drugs were the the the the the CF scheme. Information Information CF scheme. the Assessment Committee’s conclusions on on conclusions Committee’s Assessment CF scheme in comparison to conventional HTA HTA conventional to comparison in CF scheme the flowchart of drugs in CF and Table 1 for 1 Table flowchartCF and in of drugs expected budget impact at T=4 was below €2.5 €2.5 below was T=4 expected impact budget at the a submission file but later deemed unusable due to an to an unusable due deemed file but later submission the the elapsed time period between publication of time periodpublication elapsed between public domain for 12/25 drugs, which have been finalized finalized been have which drugs, 12/25 domain for public the core aims of aims core the Appendix for for Appendix the the expected added therapeutic value at T=4 was diminished (e.g. due due (e.g. diminished T=4 was at value expected added therapeutic in A the Appraisal Committee’s advice on reimbursement and official ZIN advice advice official ZIN and on reimbursement advice Committee’s Appraisal the availability of report components for T=0 and T=4 reports of finalized drugs T=4 reportsdrugs of finalized and T=0 of report for availability components standardised data extraction form was used to retrieve information on information retrieve extraction to used data was standardised form drugs were excluded because because excluded drugs were remaining 13/25 drugs, authors were obliged to retrieve information from assessors assessors from information retrieve obliged to authors were drugs, 13/25 remaining 2 authors. For 11/12 finalized drugs, drugs, finalized 11/12 For All data extraction steps and analyses mentioned above for procedural, methodological methodological procedural, for above extraction mentioned data All analyses and steps The The A RESULTS finalized drugs. finalized and decision-making aspects were conducted independently by 2 authors (AM and AvV). and (AM 2 authors and decision-making by independently conducted aspects were amongst consensus by resolved extracted were in data and analyses discrepancies Any the on reimbursement. on reimbursement. AU & CE, & CE, AU aspects from T=4 reportT=4 components: aspects from varied (Table D- Appendix). For all 12 T=0 reports, therapeutic value assessments and and assessments value T=0 reports, therapeutic 12 all D- For Appendix). (Table varied none of Contrastingly, present. were assessments cost-effectiveness T=4 reports extended beyond 4 years; ranging from 3.99 years (trastuzumab) to 7.58 years years 7.58 to (trastuzumab) years 3.99 from ranging T=4 reports years; 4 extended beyond 2). drug (Table per years of 5.93 an average with (natalizumab), guideline T=0 despite (VoI) at of information analyses value contained proposals research one for WG) that communication, (internal mentioned it was However, recommendations. included in was analysis VoI drug with subsequent publication of official ZIN advice. For information on For information ZIN advice. of official publication with subsequent could only be retrieved in be retrieved only could the due to (e.g. pending re-assessments or (5/13) ongoing drugs, are 13 these For ZIN. within datasets; inadequate supplement collection extra data to for extended deadlines allowing See8/13). Figure transferred to an alternative national scheme on orphan drugs (“monitoring of orphan of orphan scheme on orphan drugs (“monitoring national an alternative to transferred in remained drugs Twenty-five (2/24). products”(9)) disease performed ZIN. by or million/year products; or 22/24), comparator of equally effective emergence to incorrect choice of comparator treatment. For T=4 reports, therapeutic value assessments, assessments, value T=4 reports, therapeutic For treatment. of comparator choice incorrect reports in all 12 present were impact and budget assessments cost-effectiveness analyses reports. in 11/12 present were use assessments and appropriate A. Procedural aspects A. Procedural T=0 assessments. after excluded of which 24 were CF, for nominated drugs were Forty-nine The Specific attention was provided to decision-making on AU, CE & reimbursement advice advice reimbursement & CE AU, to decision-making on provided was attention Specific represent they because

remaining remaining the deficiency). deficiency). A A 7,07 6,98 5,31 6,54 6,11 3,99 7,58 7,51 5,06 4,77 4,77 5,50 Duration of of Duration for procedure financing conditional treatment of early breast cancer cancer of early breast treatment

the appraisal of evidence at T=4 in relation to to T=4 in relation at of evidence appraisal 18-07-2016 23-03-2015 13-08-2012 30-06-2014 02-07-2012 30-06-2014 14-07-2014 30-06-2014 18-03-2013 27-02-2012 27-02-2012 23-01-2012 Date of completion and and of completion Date of publication T=4 re-assessment the Indication disease type II). storage (glycogen disease Pompe (alpha-galactosidase disease Fabry´s (alpha-galactosidase disease Fabry´s nocturnal hemoglobinuria (PNH). Paroxysmal arthritis activerheumatoid to after failure Severe, blocker. TNF/alpha 1 least at to respond sclerosis multiple remitting Highly activerelapsing (RRMS). for therapy Adjuvant with increased HER2+ expression. HER2+ with increased Add-on therapy for severe, persistent allergic asthma. allergic persistent severe, for Add-on therapy aspergillosis. invasive Serious, age-related degeneration. macular Wet, keratosis. Actinic (NSCLC). cancer lung Metastatic cell non-small

Assessment Committee was consulted for all T=0 and T=4 T=4 T=0 and all for consulted was Committee Assessment the 22-06-2009 28-04-2008 23-04-2007 17-12-2007 23-05-2006 03-07-2010 18-12-2006 25-09-2006 25-02-2008 21-05-2007 21-05-2007 24-07-2006 Date of completion & & of completion Date of publication T=0 assessment Active Ingredient Active beta agalsidase eculizumab alglucosidase alpha alpha alglucosidase alpha agalsidase rituximab natalizumab trastuzumab omalizumab voriconazol ranibizumab aminolevulinate methyl pemetrexed positive opinion on continued reimbursement). However, for for However, reimbursement). continued on opinion positive a - T=0 and - T=0 report T=4 publication dates and total elapsed time per finalized drug. – List of finalized drugs’ trade names, active ingredients and indications. Assessment Committee also performed Committee Assessment 4 package criteria for 5/12 drugs (Table E – Appendix). This occurred for drugs whereby whereby drugs for occurred This E – Appendix). 4 package (Table drugs 5/12 criteria for

As per ZIN guidelines, per ZIN guidelines, As

pemetrexed methyl aminolevulinate methyl ranibizumab voriconazol omalizumab trastuzumab natalizumab rituximab eculizumab agalsidase beta agalsidase agalsidase alpha agalsidase alglucosidase alpha alpha alglucosidase Finalized drug Finalized Fabrazyme® Soliris® Trade Name Trade Myozyme® Replagal® MabThera Tysabri® Herceptin® Xolair® Vfend® Lucentis® Metvix® Alimta® Table 2 Table 1 assessments and for conclusions on AU and CE at T=4. However, contrary to guidelines, contrary guidelines, to T=4. However, CE at and on AU conclusions and for assessments the the T=4 on all 4 package criteria at evidence (i.e. straight-forward relatively was appraisal indicated

Use of RWE in HTA practice 5 100 Use of RWE in HTA practice 5 101

least least the number of of number the Appraisal Committee Committee Appraisal last drug (eculizumab), last drug (eculizumab), the broader patient population population patient broader selected outcome measures measures selected outcome the a the absence of an outcomes research research outcomes of an absence presented model structure (15/123; structure model (15/123; presented Assessment Committee addressing addressing Committee Assessment Assessment Committee concluded concluded Committee Assessment the the the the outcomes research proposal were fully fully were proposal research outcomes negative advice, advice, negative a the drug was administered to to administered drug was the studied patient population (9/58; 16%). Finally, critical commentary Finally, 16%). (9/58; population patient studied the total number of critical comments for T=0 and T=4 combined varied considerably considerably varied T=4 combined T=0 and for critical of number comments total majority of critical comments were posed at T=4, of which 58/249 (23%) related to to related (23%) 58/249 of which T=4, posed at majority were of critical comments Assessment Committee concluded that evidence submitted at re-assessment at (T=4) submitted evidence that concluded Committee Assessment proposed cost-effectiveness model (14/68; 21%) and 21%) and model (14/68; cost-effectiveness proposed total number of critical comments made by made by critical of number comments total components outcomes research proposals (T=0), proposals (T=4) use assessments research outcomes appropriate components committee concluded that that concluded committee committee concluded that evidence submitted at re-assessment and its analysis was of re-assessment of at was submitted and its analysis evidence that concluded committee most (54/249;19%). The The Recommendations made at T=0 on at made Recommendations The The The The the and its analysis was of sufficient scientific quality to assess AU in Dutch clinical practice practice Dutch clinical in AU qualityassess to scientific of sufficient was and its analysis Meanwhile, drugs. of (25%) for 3/12 drugs and inadequate 9/12 (75%) of finalized for the (58%) of finalized 7/12 for practice Dutch clinical quality to assess CE in scientific sufficient of drugs. (42%) 5/12 for and inadequate drugs 7/12 drugs where evidence may have led to led to have may evidence where drugs 7/12 was consulted. was implemented in studies conducted for 5/12 finalized drugs. For 6/12 finalized drugs, drugs, finalized For 6/12 drugs. finalized 5/12 for in studies conducted implemented only partially Moreover, were implemented. recommendations For 8/9 drugs with sufficient evidence on AU, on evidence with sufficient drugs 8/9 For C. Decision-making aspects recommendations that were not incorporated varied. For example, for rituximab 6/7 rituximab 6/7 for example, For varied. not incorporated were that recommendations for (18%) 2/11 only to compared not incorporated, were made recommendations of (86%) Due to C – Appendix). (Figure methylaminolevulinate this drug. for not be conducted could analysis this trastuzumab, for proposal practice; for clinical in used appropriately they were that The The B. Methodological aspects Methodological B. number of comments at T=0 and T=4 combined (2/249; 0.01%), whereas rituximab incurred rituximab incurred whereas 0.01%), (2/249; T=4 combined T=0 and at of comments number the the between finalized drugs (Figure B - Appendix). For example, pemetrexed incurred incurred pemetrexed example, For Appendix). - B (Figure drugs finalized between 12%) and clinical effectiveness outcomes measured (14/123; 11%). (14/123; measured outcomes effectiveness and clinical 12%) provided at T=4 on cost-effectiveness assessment was mostly directed at costs outcomes costs outcomes at mostly directed was assessment on cost-effectiveness T=4 at provided 16%), (20/123; gathered was information which for appropriate use assessments and 123/249 (49%) related to cost-effectiveness assessments. assessments. cost-effectiveness to related (49%) and 123/249 assessments use appropriate quality at directed mostly was use assessment on appropriate T=4 at Commentary provided included measures outcome effectiveness clinical 21%), (12/58; collected information of life and 17%) (10/58; the assessments 68/249 and cost-effectiveness total, varied. In drugs for all finalized (T=4) mostly directed T=0 and were at proposals research outcomes to related comments (27%) at 19%). (13/68; effect clinical for 4 the form of of form case-by- evidence evidence Appraisal Appraisal a committee committee drug being drug being the Assessment Assessment Assessment Assessment the the the the the the novel comparator comparator novel a final drug (rituximab), (rituximab), final drug last drug (pemetrexed), last drug (pemetrexed), the basic healthcare package. package. healthcare basic the ICERs presented were below below were presented ICERs the the pragmatic solution in pragmatic a drug belonged to and on drug belonged to threshold value of €80,000/QALY and and of €80,000/QALY value threshold near future. near future. final drug (natalizumab), final drug (natalizumab), the the the risk for incurringhigh ICER’s. risk for the low probability (10-40%) of (10-40%) probability low the the evidence submitted. For For submitted. evidence separate initiative (Round Table on Multiple Sclerosis) Table (Round initiative separate the a ICERs presented were not substantiated by by substantiated not were presented ICERs need for exceptional financing of orphan drugs outside of orphan financing drugs outside exceptional need for ICERS were above above ICERS were committee argued for for argued committee committee stated that it could not reach conclusions on AU AU on conclusions reach not could it that stated committee committee concluded that that concluded committee the the the decision on discontinuation of reimbursement until further until of reimbursement on discontinuation decision the the the submitted evidence was insufficient. was evidence submitted the assessment and appraisal of evidence at re-assessment at of evidence (T=4) and appraisal assessment by Appraisal Committee appraised evidence at T=4 for 7/12 drugs. For For drugs. 7/12 T=4 for at evidence appraised Committee Appraisal threshold, impending expiry of its patent and emergence of generic of generic expiry impending emergence and of its patent threshold, the Appraisal Committee advised to discontinue reimbursement. reimbursement. discontinue to advised Committee Appraisal 4 orphan drugs (alglucosidase alpha, agalsidase alpha and agalsidase and agalsidase alpha agalsidase alpha, 4 orphan drugs (alglucosidase the the committee advised clinician societies to update clinical guidelines to to clinical guidelines update to societies advised clinician committee the 7 drugs with sufficient evidence on CE were indicated for orphan diseases, for diseases, orphan indicated were on CE evidence sufficient 7 drugs with the the the other hand, for 4/5 drugs with inadequate evidence on CE, CE, on evidence inadequate with 4/5 drugs for other hand, the establishment of necessary patient registries to monitor real-world outcomes outcomes real-world monitor to necessary of establishment registries patient Assessment Committee went on to appraise all evidence at T=4 in relation to to relation in T=4 at evidence all appraise to on went Committee Assessment the the healthcare system) for 5/12 drugs; for 4/5 drugs, continued reimbursement from from reimbursement continued 4/5 drugs, for drugs; 5/12 for system) healthcare committee concluded that additional data collection was unnecessary due to collectionunnecessary data was additional that to due concluded committee basic healthcare package, tailored policies on development and pricing of orphan and pricing of orphan on development policies package, tailored basic healthcare committee concluded that despite despite that concluded committee respective committees, ZIN issued their final advice to continue reimbursement for for reimbursement continue to advice issued their final ZIN committees, respective basic healthcare package was advised. For package For basic healthcare advised. was

threshold and substantiated by by and substantiated threshold Four of of Four Based upon Based upon On The The For 3 of For the Pay-for-Performance scheme to avoid its exclusion from from its exclusion avoid scheme to Pay-for-Performance than intended. Meanwhile, Meanwhile, intended. than cost-effective at cost-effective the for drugs for which for drugs for whereby high incremental cost-effectiveness ratios (ICERs) led to (ICERs) led ratios cost-effectiveness incremental high whereby Committee. For 2/7 drugs, 2/7 drugs, For Committee. with methylaminolevulinate, treatment who qualify for patients clearly criteria specify for using patients amongst compliance low to due (e.g. over-prescription avoiding thereby and drugs (eculizumab 2/7 For non-response). apparent leading to treatments comparator ranibizumab), delegating further discussions in relation to other societal considerations to to considerations further other societal delegating to in relation discussions products would improve its cost-effectiveness in its cost-effectiveness improve products would Committee concluded that that concluded Committee treatment, both factors thereby minimising minimising both factors thereby treatment, the Committee concluding that that concluding Committee the Meanwhile, thus no conclusions could be reached on their CE in practice. For For their CE in practice. on be reached could thus no conclusions diminished added therapeutic value and costs which are comparable to to comparable which are and costs value added therapeutic diminished the a advised to postpone postpone to advised from available becomes evidence Meanwhile, (10). case basis. included conditions beta), 5/7 drugs, continued reimbursement was advised based on additional conditions. Such Such conditions. additional on advised based was reimbursement continued drugs, 5/7 NZa framework on which based varied conditions package criteria (necessity, clinical effectiveness, cost-effectiveness and implementability implementability and cost-effectiveness effectiveness, clinical package criteria (necessity, in the the drugs, and bundling of clinical expertise to ensure AU. Conditions for expensive drugs varied drugs varied expensive for Conditions expertiseand bundling of clinical AU. ensure to omalizumab, For per case.

Use of RWE in HTA practice 5 102 Use of RWE in HTA practice 5 103

time time finalized finalized the evidence evidence the the use of voriconazol use of voriconazol the use of tailored approaches approaches use of tailored envisioned 4-year period. period. 4-year envisioned The The the scheme, of which 12 underwent of which 12 underwent scheme, Assessment Committee concluded concluded Committee Assessment myriad of factors, including including of factors, myriad than rather T=0, at raised questions varying percentage of unaddressed of unaddressed a varying percentage drugs. Critical commentary provided commentary Critical drugs. provided the a the outcomes research proposal were fully fully were proposal research outcomes the the the finalized drugs, reimbursement was continued continued was reimbursement drugs, finalized outcomes research proposal (T=0), proposal research outcomes appropriate remaining drugs are not directly apparent from from not directly apparent are drugs remaining the envisioned 4-year time window. Although reasons reasons Although window. time 4-year envisioned committees. committees. reimbursement of 6/12 drugs were similar to, albeit albeit to, similar were drugs 6/12 of reimbursement more intuitive design. intuitive more the the the the a the implemented outcome research studies to reach grounded grounded reach to studies research outcome implemented assessment of mortality of assessment with outcomes remaining drugs. At T=4, At drugs. remaining the 4-year period is applicable to all indications for which for all indications period to is applicable 4-year the third of research questions defined at T=0, insufficient evidence evidence insufficient T=0, at defined questions of research third the a Appendix. a CED framework, were fit for its envisioned purpose. For example, only only example, For purpose. envisioned for its fit CED framework, were a the detailed summarydetailed see of all decision-making drug, per aspects describedabove Assessment Committee on on Committee Assessment 49 drugs nominated for CF, 25 remained in in 25 remained CF, for 49 drugs nominated a Assessment Committee for almost half of almost for Committee Assessment full procedure. Only 1 drug was completed within within completed 1 drug was Only full procedure. extracted data for this study, they may relate to to relate they may study, this extracted for data the Importantly, for for Importantly, In light of results summarised above, one may question whether some design aspects design some whether question of may one summarised above, Inresults of light For For the the third of research questions defined at T=0. Eventually, based on advice of its committees, committees, of its advice based on Eventually, T=0. at defined questions of research third fixed 4-year window, may provide provide may window, 4-year fixed DISCUSSION for serious, invasive aspergillosis (an acute, life-threatening condition) requires shorter requires condition) life-threatening (an acute, aspergillosis invasive serious, for lung cancer. cell non-small for pemetrexed for than follow-up drugs were approved; approved; drugs were time-frames answer required to determining for a needed to set up registries required for data collection, to compile and evaluate data data and evaluate compile collection, data to for required up registries set to needed assess and appraise to subsequently and studies, outcome from generated up in setting invested were resources extensive In (11;12). Italy instance, generated for one (13). years Moreover, many over data necessary collect fit-for-purpose to infrastructures whether wonder may Of was generated through through generated was 4/12 (33%) finalized drugs, continue reimbursement based on additional conditions conditions on additional based reimbursement continue drugs, finalized (33%) 4/12 drugs 2/12 (17%) for reimbursement and discontinue drugs, (50%) finalized 6/12 for CF, an example of example an CF, (Table 3). Additional conditions for for conditions Additional 3). (Table by those cited than extensive, more with drugs, finalized of half less than for implemented quality to answer of insufficient was research outcomes through generated evidence that a with additional conditions, of which 6 10 drugs, for reimbursement continue ZIN advised to 2. for reimbursement discontinue and to within processed been drug had 1 use assessment (T=4)use assessment assessment and cost-effectiveness per considerably varied (T=4) on provided Recommendations drug. finalized for recommendations Tables F and G in F and G in Tables the the conclusions at T=4. Moreover, for half of half for Moreover, T=4. at conclusions Published T=0 and T=4 reports did not consistently include all necessary components. necessary all include T=4 reports components. did not consistently and T=0 Published conducted was T=4 at evidence of appraisal in guidelines, outlined procedures Contrary to by of processing timely to failure for by

and stop-decisions for treatment with this drug. this with treatment for stop-decisions and

Consider establishing an independent committee to advise clinicians in practice on start- start- on practice in clinicians advise to committee independent an establishing Consider •

implementation of start- and stop-criteria. and start- of implementation the for system transparent

factors for clinical effectiveness, develop start- & stop-criteria and develop develop and stop-criteria & start- develop effectiveness, clinical for factors more more a

(European) study to investigate predictive predictive investigate to study (European) a up set to parties necessary the Demand •

dose modification). dose

Discuss with clinicians if, and how, costs per QALY can be reduced (e.g. through through (e.g. reduced be can QALY per costs how, and if, clinicians with Discuss •

marketing authorisation holder (MAH). holder authorisation marketing the with negotiations price Negotiate •

orphan diseases. orphan

new framework specific to drugs for for drugs to specific framework new the to drug the of reimbursement the Transfer •

separate financial framework for drugs for orphan diseases. orphan for drugs for framework financial separate a Develop • package based on certain conditions. certain on based package

basic healthcare package. healthcare basic the from drug the of reimbursement continue Temporarily • Keep drug in basic healthcare healthcare basic in drug Keep alpha agalsidase

and stop-decisions for treatment with this drug. this with treatment for stop-decisions and

Consider establishing an independent committee to advise clinicians in practice on start- start- on practice in clinicians advise to committee independent an establishing Consider •

transparent system for for system transparent implementation of start- and stop-criteria. and start- of implementation the

factors for clinical effectiveness, develop start- & stop-criteria and develop develop and stop-criteria & start- develop effectiveness, clinical for factors more more a

necessary parties to set up up set to parties necessary the Demand • (European) study to investigate predictive predictive investigate to study (European) a

dose modification). dose

Discuss with clinicians if, and how, costs per QALY can be reduced (e.g. through through (e.g. reduced be can QALY per costs how, and if, clinicians with Discuss •

Negotiate price negotiations with with negotiations price Negotiate • marketing authorisation holder (MAH). holder authorisation marketing the

orphan diseases. orphan

new framework specific to drugs for for drugs to specific framework new the to drug the of reimbursement the Transfer •

separate financial framework for drugs for orphan diseases. orphan for drugs for framework financial separate a Develop • package based on certain conditions. certain on based package

Temporarily continue reimbursement of of reimbursement continue Temporarily • basic healthcare package. healthcare basic the from drug the alglucosidase alpha alpha alglucosidase Keep drug in basic healthcare healthcare basic in drug Keep

basic healthcare package healthcare basic the Extra conditions specified conditions Extra Finalized drug Finalized

ZIN advice on reimbursement from from reimbursement on advice ZIN

- Summary of ZIN advice on reimbursement for all finalized drugs. finalized all for reimbursement on advice ZIN of Summary - 3 Table

Use of RWE in HTA practice 5 104

Use of RWE in HTA practice 5 105

basic healthcare package. healthcare basic the from drug this of removal the

from individual hospitals or no refunds to hospitals based on outcomes), ZIN will advise for for advise will ZIN outcomes), on based hospitals to refunds no or hospitals individual from

case of defaults on PfPO agreements (e.g. due to lack of cooperation cooperation of lack to due (e.g. agreements PfPO on defaults of case the In prescribed.

sign Pay-for-Performance (PfPO) agreements with all hospitals whereby whereby hospitals all with agreements (PfPO) Pay-for-Performance sign conditions. certain on based package drug will be be will drug the

marketing authorisation holder (MAH) should should (MAH) holder authorisation marketing the reimbursement, continued guarantee To Keep drug in basic healthcare healthcare basic in drug Keep omalizumab

healthcare package healthcare

basic in drug Keep trastuzumab N/A

package based on certain conditions. certain on based package [separate] Round Table on Multiple Sclerosis are presented. are Sclerosis Multiple on Table Round [separate] the from results until

ZIN postpones its final decision for removal of this drug from from drug this of removal for decision final its postpones ZIN basic healthcare package package healthcare basic the Keep drug in basic healthcare healthcare basic in drug Keep natalizumab

healthcare package healthcare

basic in drug Keep rituximab N/A

healthcare package. healthcare

basic from drug Remove eculizumab N/A

and stop-decisions for treatment with this drug. this with treatment for stop-decisions and

Consider establishing an independent committee to advise clinicians in practice on start- start- on practice in clinicians advise to committee independent an establishing Consider •

transparent system for for system transparent implementation of start- and stop-criteria. and start- of implementation the

factors for clinical effectiveness, develop start- & stop-criteria and develop develop and stop-criteria & start- develop effectiveness, clinical for factors more more a

necessary parties to set up up set to parties necessary the Demand • (European) study to investigate predictive predictive investigate to study (European) a

dose modification). dose

Discuss with clinicians if, and how, costs per QALY can be reduced (e.g. through through (e.g. reduced be can QALY per costs how, and if, clinicians with Discuss •

Negotiate price negotiations with with negotiations price Negotiate • marketing authorisation holder (MAH). holder authorisation marketing the

orphan diseases. orphan

new framework specific to drugs for for drugs to specific framework new the to drug the of reimbursement the Transfer •

separate financial framework for drugs for orphan diseases. orphan for drugs for framework financial separate a Develop • package based on certain conditions. certain on based package

Temporarily continue reimbursement of of reimbursement continue Temporarily • basic healthcare package. healthcare basic the from drug the agalsidase beta agalsidase Keep drug in basic healthcare healthcare basic in drug Keep

basic healthcare package healthcare basic the Extra conditions specified conditions Extra Finalized drug Finalized

ZIN advice on reimbursement from from reimbursement on advice ZIN

- continued - 3 Table

healthcare package healthcare

pemetrexed N/A basic in drug Keep

implementation of such criteria becomes feasible in practice. in feasible becomes criteria such of implementation the

criteria for treatment with methylaminolevulinate thus ensuring that that ensuring thus methylaminolevulinate with treatment for criteria the specify and conditions. certain on based package aminolevulinate

clinicians’ societies to update update to societies clinicians’ the requests ZIN clinical guideline, in order to clarify clarify to order in guideline, clinical the Keep drug in basic healthcare healthcare basic in drug Keep methyl methyl

healthcare package. healthcare

ranibizumab Remove drug from basic basic from drug Remove N/A

healthcare package healthcare

voriconazol Keep drug in basic basic in drug Keep

N/A

basic healthcare package healthcare basic the Extra conditions specified conditions Extra Finalized drug Finalized

ZIN advice on reimbursement from from reimbursement on advice ZIN

- continued - 3 Table

Use of RWE in HTA practice 5 106 Use of RWE in HTA practice 5 107 Pay- start a advice advice context the mid-term mid-term marketing marketing the the responsible responsible the the the Pay-for-Performance Pay-for-Performance implementation of CF CF of implementation a the lack of full incorporation of of incorporation of full lack basic healthcare package by package healthcare basic by appropriate use of voriconazol of voriconazol use appropriate majority of finalized drugs had majoritydrugs had of finalized the the Ministry of Health, the the feasibility and intrinsic value of data of data value and intrinsic feasibility the diversity of stakeholders active within active of stakeholders within diversity finding are not directly apparent from from not directly apparent finding are the the authors’ knowledge, it was not specified in in specified not it was knowledge, authors’ a strategy for for strategy priori timelines projected. Secondly Secondly projected. timelines a the the modification of clinical guidelines as per ZIN advice. guidelines as per ZIN advice. of clinical modification complexity of interactions between their mandates and and mandates their between interactions of complexity CF procedure for drugs, thereby avoiding waste of valuable of valuable waste avoiding thereby drugs, for CF procedure absence of an of absence the the CED scheme, only to follow up with up with follow only to CED scheme, the Dutch healthcare setting for their respective tasks. For example, example, For tasks. their respective for setting healthcare Dutch the a proposed outcomes research contributed to this. Two safeguards safeguards Two this. to contributed research outcomes proposed the the actual healthcare setting. To To actual setting. healthcare potential reasons behind such behind reasons potential CF scheme (6). By then, T=0 assessments for for T=0 assessments (6). ByCF scheme then, the the basic healthcare package would or should be implemented by by package implemented be or should healthcare basic would emergence of innovative, yet expensive medications is occurring rapidly. Moreover, Moreover, is occurring rapidly. medications expensive yet of innovative, emergence the the conduct of VoI analyses at T=0 to highlight highlight T=0 to at analyses VoI conduct of Dutch healthcare setting, setting, healthcare Dutch Ministry of Health to date, and it remains unknownMinistry if and it remains date, to of Health data extracted, literature alludes to numerous reasons such as challenges with analysing analysing with such as challenges reasons numerous to alludes literature extracted, data The The Another shortcoming Another is strategies on how CF outputs would and should be implemented in practice by in practice by implemented be and should would outputs CF on how priori strategies notable trend amongst novel oncology treatments relates to conditional marketing based marketing based conditional to relates treatments oncology novel amongst trend notable the a within phase I/II studies) or efficacy(e.g. on safety evidence conclusive on less based on yet further evidence generation to address remaining uncertainties. Once again, again, uncertainties. Once remaining address further to on yet based generation evidence although the It (14;15). RWEthat be also may generated interpreting and recommendations on recommendations and participating organisations per as hospitals patient holder, authorisation proposed in ZIN guidelines may have prevented such shortcomings in hindsight. Firstly, such shortcomings Firstly, in hindsight. prevented have may in ZIN guidelines proposed the collection for specific parameters within within specific parameters collection for T=4 T=0 and between results and interim progress research reporting of outcomes continuation, regarding decisions timely to more led have T=3) may T=1 & at (specifically of termination or adjustment all to and discouraging be burdensome it would However, Committee. Appraisal of ZIN’s parties first implement to time and money for all stakeholders involved. Unfortunately, both recommendations (VoI recommendations both Unfortunately, involved. stakeholders all for money and time after 2 years than more 2008, in December published reporting)and interim were date of date schemes pay-for-performance implementing Moreover, (2;18). each drug scheme for parties, responsible from costs retrieving to relating incurs other practical considerations provided Therefore, in Italyas experienced (13). to establishing paid been should have attention more interests, differing stakeholders’ a stakeholders. different already been completed. Nevertheless, both design aspects may be essential for future future for aspects be essential Nevertheless, design both may been completed. already (16). literature (particularlyin previous of MEAs design iterated as has been CEDs), outputs in outputs external in stakeholders the ZIN, by jointly initiated scheme was for-Performance it is known that ranibizumab has not been removed from from removed has not been it is known ranibizumab that guidelines beforehand how advice officially issued by ZIN on reimbursement of CF drugs drugs CF of reimbursement on ZIN by issued officially advice how beforehand guidelines from has been improved through through improved has been Previous experiences in Germany allude to difficulties associated with removing medicines medicines removing with associated difficulties to allude Germany in experiences Previous treatment in choice packages reimbursement physicians´ limiting or national from whereby story of omalizumab, one successful is that Contrastingly, (17). prescription novel novel a research research quantity quantity the the more prominent role role prominent more experiences gained by by gained experiences a taxonomy of MEAs and and MEAs of taxonomy the re-assessment of submitted re-assessment of submitted examined study population study population examined the authors for this study is for authors the the authors examined examined authors means to determine experiences experiences determine means to the a the Assessment Committee’s conclusions on on conclusions Committee’s Assessment most. the Assessment Committee on appropriate use use on appropriate Committee Assessment use of an alternative, tailored approach. approach. tailored an alternative, use of the Netherlands and elsewhere. generalizability of the the the authors. the authors were able to discern which aspects influenced which aspects discern able to influenced were authors the successes, failures, strengths and weaknesses of established and weaknesses of established strengths failures, successes, the evidence submitted for AU and CE, as well as its final conclusions conclusions final its as as well CE, and AU for submitted evidence the particular context. Therefore, in order to best address best address to in order Therefore, particular context. finalized drugs, we noted that 9 critical comments were provided provided were comments 9 critical that we noted drugs, finalized a the authors opted for for opted authors focus of future research, in order to avoid repeating historical mistakes mistakes historical repeating avoid to in order research, future of focus the authors examined both examined authors the fact that such frameworks aim to classify classify aim to fact such frameworks that the qualitative nature of comments provided have not been separately not been separately have provided of comments nature qualitative the design and implementation of MEAs, particularly CEDs, remains complicated particularly complicated remains of MEAs, CEDs, and implementation design the assessment of methodological aspects, methodological of assessment the the authors are aware of other MEA analysis frameworks proposed in literature (1;5;8) (1;5;8) literature in proposed frameworks MEA analysis of other aware are authors Dutch clinical population, such comments may have had have may comments such population, clinical Dutch final appraisal of evidence compared to other comments. In an attempt to address to address attempt In an comments. to other compared of evidence final appraisal the The The evaluation scheme developed and implemented by by and implemented scheme developed evaluation increasing dependence on MEAs for both HTA and regulatory and decision-making (20). both HTA for on MEAs dependence increasing Committee’s conclusions on AU and CE and on AU conclusions Committee’s topic of ongoing research by by research of ongoing topic wide array of stakeholders involved in implementing CF. In doing so, numerous findings findings numerous In so, doing CF. implementing in involved of stakeholders wide array

scientific qualityscientific of In Finally, this study presents an analysis of reports an analysis as this study presents Finally, the the

in this limitation, to the In CE. and so, doing on AU but refer to to but refer one. one. their analyse retrospectively to than rather design, their practices for best recommend within implementation the on patient populations examined in outcomes research studies. Bearing in mind that Bearing in mind that studies. research in outcomes examined populations on patient on hinge on AU questions research could be brought to light which may not be part of HTA reports analysed. This is currently is currently This reports analysed. not be part which may light to HTA of be brought could the gained in implementing CF. However, this research question additionally warrants alternative alternative warrants additionally question research this However, CF. implementing in gained on information interviews) stakeholder gather to (e.g. methods the question at hand, hand, at question by of critical commentary provided assessments and cost-effectiveness assessments. Although this provided insights as to to as insights this provided Although assessments. and cost-effectiveness assessments during been most controversial have may which elements evidence, in appropriate example, For impact their determine appraisal. to on evidence addressed of use assessments Limitations The However, However, of accelerated/conditional approval pathways (19). Consequently, HTA agencies and payers and payers agencies HTA (19). Consequently, pathways approval of accelerated/conditional uncertainties more with submissions aspects on as long-term such encounter increasingly global an increasing Meanwhile, practice. in clinical and effectiveness outcomes health approach an iterative whereby (MAPP’s), patients to pathways adaptive in medicines interest reasserts lifetime, their products throughout for is adopted generation evidence to the MEAs should be should MEAs (2;13;17;18;20). One may argue that without systematic evaluations of established MEAs, MEAs, of established evaluations systematic without that argue One may (2;13;17;18;20). potential this counter To ones. as previous similar caveats suffer to likely are schemes novel risk, knowledge regarding when setting up new schemes within new schemes setting up when

Use of RWE in HTA practice 5 108 Use of RWE in HTA practice 5 109

future. the variety of shortcomings related to procedural, procedural, to variety of shortcomings related a design of better schemes in of better design the design and implementation of future MEA schemes. schemes. MEA future of implementation and design valuable MEA framework, guaranteeing patient access to to access patient MEA framework, guaranteeing valuable the a continuing onslaught of innovative, yet expensive drugs and HTA and HTA drugs expensive yet of innovative, onslaught continuing the This study illustrates an attempt to systematically evaluate CF in order to inform ongoing ongoing inform to order in CF evaluate systematically to an attempt illustrates study This CONCLUSION agencies´ and payers’ increasing reliance on MEAs, further research on experiences gained gained further on experiences MEAs, on research reliance increasing and payers’ agencies´ inform is critical to MEAs with other However, provided provided However, international discussions on discussions international innovative agreements while simultaneously obliging responsible parties responsible collect obliging while simultaneously RWE to agreements innovative informing uncertainties, thereby to address cost-effectiveness use and on appropriate decision-making re-assessment. at However, practice. in value its decision-making and affected methodological aspects have may in implemented MEAs on literature in available been echoed Such shortcomings have jurisdictions.other In principle, CF may provide provide Inmay CF principle,

“cost-sharing “cost-sharing German Medical PHAROS registry. PHAROSregistry. the the the American journal of American journal of framework for coverage coverage for framework The The A practical guide for using using practical guide for A United States: trends, barriers, barriers, trends, States: United current performance-basedcurrent schemes cost effectiveness of new cancer drugs: drugs: cancer new of effectiveness cost the case of oxaliplatin for treatment of stage stage of treatment for case of oxaliplatin novel measure for cost-containment of drug cost-containment for measure novel Association. Opinion on on Opinion Association. initiatives” and “risk-sharing agreements” agreements” “risk-sharing and initiatives” manufacturers pharmaceutical between 2017 2008.[cited and Hospital. and Health http://www.akdae. from: Available April 14]; de/Stellungnahmen/Weitere/20080508.pdf III colon cancer. Value Health 2015;18:84-90. Health Value cancer. III colon Lipska et al. N, Hoekman I, McAuslane J, new oncology for approval Does conditional in health differences lead to drugs in Europe Clin decisions? assessment technology Ther Pharmacol 2015;98:489-91. Blommestein HM, Franken MG, Uyl-de HM, Franken Blommestein CA. Groot Uyl-deC, Gils Mohseninejadvan L, Groot registries of patient Evaluation CA, et al. supporting decisions: reimbursement the Sculpher S, Claxton M, K, et al. Walker development, with evidence Coverage patient or risk sharing, only in research, scheme? access 2012;15:570-9. Health Value decisions. of Drug Commission Bajaj PS, et al. JJ, Carlson Garrison LP, Jr sector agreements risk-sharing Private in and prospects. 2015;21:632-40. care managed Alemayehu D, Mardekian J. Infrastructure Infrastructure Mardekian J. D, Alemayehu Secondary for Sources Data Requirements J Research. Effectiveness in Comparative 2011;17:S16-S21. Pharm Manag Care Doal. L, et Gozzo V, A, Drago Navarria the FeeGÇØ: in Italy work?GÇ£Success really A Health 2015;18:131-6. Value expenditure. about decisions inform registry to data the learnedlessons from and Proposed Methods for Conducting Conducting Methods for and Proposed Research Effectiveness Comparative ManagJ Care Data”. “Real-World Using 2011;17:S3-S37. Pharm Pharmacoecon 2015;33:551-60. Pharmacoecon

18. 19. 15. 16. 17. 13. 14. 12.

use of of use the ISPOR good practices task force. Value Value practices force. ISPOR good task taxonomy and examination of performance- of and examination taxonomy Zorginstituut Nederland. Fingolimod (Gilenya): (Gilenya): Fingolimod Nederland. Zorginstituut Summary8-7-2016. of Recommendations. et JM, Alvir Riaz MPP, Ali D, Alemayehu Issues Analytical of Data, Examination al. McCabe CJ, Stafinski T, Edlin R, et al. Access Access Edlin R, et al. T, Stafinski CJ, McCabe schemes. development with evidence 2010;28:143-52. Pharmacoecon Pakketbeheer P. AJ,Link Pasman 26-10-2015. Weesgeneesmiddelen. Nederland. Zorginstituut College voor Zorgverzekeringen. Leidraad Leidraad Zorgverzekeringen. voor College behoeve ‘ten Uitkomstenonderzoek voor doelmatigheid de beoordeling van College 2008. geneesmiddelen’. intramurale 22-10-2015. Zorgverzekeringen. voor Nederland, Zorginstituut Voorwaardelijke mw.dr.G.Ligtenberg. 6-4-2012. zorg. van toelating/financiering Garrison LP, Towse A, Briggs A, et Briggs A, A, et Towse Garrison LP, Performance-based al. risk-sharing design, - good practices for arrangements report evaluation: and of implementation, the 2013;16:703-19. Health Klappe-Sabadi FGA, Jansman G, verschillen Regionale HA, et al. Honkoop trastuzumab in voorschrijfgedrag Wetenschappelijk PW getoetst. opnieuw 2008. 2008. 2[7]. Platform Klemp M, Frønsdal KB, Facey K. What What K. Facey KB, Klemp M, Frønsdal principlesgovern should Carlson JJ, Gries KS, Yeung K, et al. Current K, Current al. et Yeung Gries KS, JJ, Carlson in performance-based trends and status between arrangements risk-sharing medical product and payers healthcare health economics Applied manufacturers. policyand health 2014;12:231-8. Carlson JJ, Sullivan SD, Garrison LP, et al. et al. Garrison LP, SD, Sullivan JJ, Carlson health outcomes: Linking to payment a between schemes based reimbursement Health manufacturers. and payers healthcare 2010;96:179-90. Policy managed entry agreements? Int J Technol Technol Int entrymanaged J agreements? 2011;27:77-83. Care Health Assess

REFERENCE LIST REFERENCE 10. 11. 9. 8. 7. 6. 5. 4. 3. 2. 1.

Use of RWE in HTA practice 5 110 Use of RWE in HTA practice 5 111 United States:United the European Union and Union and European Lessons learned to approach adaptive payer adaptive approach learned to Lessons 2016;100:730-42. Ther Pharmacol Clin pathways. Faulkner SD, Lee M, Qin D, et al. Pricing andPricing al. et D, M, Qin Lee SD, Faulkner in insights and experiences reimbursement the

20.

CHAPTER

Conditional Financing of Drugs in the Netherlands: Past, Present and Future. 6 Results from Stakeholder Interviews

A. Makady1,2, S. van Acker3, H. Nijmeijer4, A. de Boer2, H. Hillege5, O. Klungel2, W. Goettsch1,2

1 The National Healthcare Institute (ZIN), Diemen, the Netherlands 2 Department of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht, the Netherlands 3 Vrije Universiteit, Amsterdam, the Netherlands 4 Radboud University Medical Centre, Nijmegen, the Netherlands 5 Department of Epidemiology, University Medical Centre Groningen, Groningen, the Netherlands

Article prepared for submission. form a 4-year process process 4-year stakeholders on on stakeholders a Dutch healthcare healthcare Dutch drug (HTA; T=0), T=0), drug (HTA; the the the Netherlands as the drug (T=4). This article (T=4).drug This aims the need for new policy to address new policy address need for to the positive impactpositive of CF on the following topics: perceived aim of CF, functioning of of CF, aim perceived topics: following the future. However, stakeholders diverged on whether CF should whether CF should on diverged stakeholders However, future. functioning of CF, functioning CF, of the the same aims of CF in same aims of CF perceived aim of CF. Conversely, there was some agreement amongst stakeholders on on stakeholders amongst agreement some was there Conversely, aim of CF. perceived

shortcomings in

ABSTRACT Background in implemented was drugs hospital (CF) of financing Conditional the the Results & Conclusion amongst emerged Differences Thirty interviewed. were stakeholders CF, impact of CF and conclusions on CF and future perspectives. Extensive summaries were summaries were Extensive perspectives. impact on CF and future and conclusions of CF CF, analysis. content directed each interviewfor used subsequently and for generated Public and private stakeholders were approached for participation for stakeholder in approached were stakeholders private and Public to interview An e-mail developed was guide interviews standardized using invitations. covered which discussions guide Methods of Managed Entry Agreements (MEAs) between 2006 and 2012. CF was was CF 2012. and 2006 between (MEAs) of Managed Entry Agreements of assessment technology health stages: initial three comprising conduct of outcomes research studies and re-assessment and studies of research conductoutcomes of the setting and improvement points for CF. Despite stakeholders’ belief that CF only partially CF that belief stakeholders’ Despite CF. for points and improvement setting on agreement was there all, at if not met its aims, to analyze stakeholder experiences in implementing CF in practice. CF in in implementing experiences stakeholder analyze to be improved based on learnings identified and re-introduced into practice or replaced with with replaced practice or re-introduced into and on learnings based identified be improved new policy schemes.

Use of RWE in HTA practice 6 114 Use of RWE in HTA practice 6 115

then then wide wide a the collection collection generation generation assessment assessment the the the national health health national the Netherlands Healthcare Netherlands Healthcare future, partly an future, due to proven added therapeutic added therapeutic proven a the the appropriate use and/or cost- appropriate healthcare system. According According system. healthcare CED scheme was subsequently subsequently CED scheme was evidence base. However, despite despite However, base. evidence a the the the form of form marketing authorization holder (MAH; i.e. i.e. marketing(MAH; holder authorization the drug or medical technology under specified specified under technology medical or drug a the Netherlands ensued due to unequal access to to access unequal Netherlands due to ensued notable advantage of CED schemes seemed to be their be their to seemed schemes CED of advantage notable A process. For example, ZIN was responsible for for responsible was ZIN example, For process. the the allocation of resources within within of resources allocation dilemma between quick patient access to drugs and drugs to access quick patient dilemma between emergence of innovative, yet expensive drugs (2). Consequently, Consequently, (2). drugs expensive yet innovative, of emergence the the drugs in Dutch clinical practice clinical (9). in Dutch drugs the National Healthcare Institute (ZIN; formerly CVZ), Institute formerly (ZIN; Healthcare National the the CF process comprised three main stages: initial HTA (T=0), conduct of outcomes HTA initial main stages: three comprised CF process budget impact higher than €2.5 million per year, have have impactbudget than €2.5 million per year, higher Organization for Economic Co-operation and Development (OECD), pharmaceutical pharmaceutical (OECD), Co-operation and Development Economic for Organization a national healthcare insurance packagepackage) insurance (henceforth reimbursement (8). healthcare national implementation of these frameworks in these frameworks of implementation The The One policy instrument comprises managed entry agreements (MEAs). Briefly defined, BrieflyOne policydefined, (MEAs). entry managed comprises instrument agreements In public outcry 2005, in the INTRODUCTION value and there needed to be uncertainties regarding be uncertainties to needed regarding and there value effectiveness of effectiveness of evidence submitted for HTA at T=0 and T=4 and for providing feedback on outcomes on outcomes feedback providing for T=4 and T=0 and at HTA for submitted of evidence Meanwhile, T=0. at proposals research research and re-assessment (T=4)(see Figure 1 on page 62). Various stakeholders were were stakeholders Various and re-assessment (T=4)(seeresearch 1 on page 62). Figure of each phase at involved technology assessment (HTA) agency. Drugs qualifying for CF had to meet three criteria: criteria: three meet to CF had for Drugs qualifying agency. (HTA) assessment technology have delegated to to delegated of (real-world) evidence (4). (4). evidence of (real-world) capability resolve to conditions” (4). Several forms of MEAs exist, each addressing different policy questions. policyquestions. different addressing each exist, of MEAs forms (4). Several conditions” mechanisms to includes schemes, (CED) development with evidence coverage One form, uncertainties of drugs through address in clinical and/or cost-effectiveness MEAs are “arrangements between drug manufacturers and payers or providers that ensure ensure that or providers manufacturers drug and payers between “arrangements are MEAs of reimbursement or coverage to access increased trend in in trend increased generic or co-payment drugs for systems preference of policyarray (e.g. instruments (2;3). mechanisms) policy makers have been attempting to control drug expenditure through through expenditure drug control to been attempting policy have makers drug expenditure in accounts for an average of 16.9% of total healthcare expenditures expenditures healthcare total of of 16.9% average an for in accounts drug expenditure of expenditures 50% exceeding in some countries countries; 31 OECD member across in expenditure drug increased also alludes to Literature (1). to Authority (Nederlandse Zorgautoriteit; NZa) devised two policy frameworks to facilitate facilitate policy two NZa) devised to frameworks (NederlandseAuthority Zorgautoriteit; from respectively, and orphan drugs in hospitals, expensive of (CF) financing conditional the The Provided that healthcare budgets are finite, decision makers consistently face difficult difficult face consistently makers decision finite, budgets are healthcare that Provided regarding questions of additional data to resolve potential uncertainties potential in resolve to data of additional these perceived advantages, it remains questionable whether they can deliver on their their on they can deliver whether questionable it remains advantages, perceived these in practice (5;6). promises “ZIP so-called Inequality (7). led to trastuzumab in access drug, expensive yet innovative, some in 25% from varied trastuzumab to access patient whereby healthcare”, code Between in others (7). and 2012, 75% 2006 to provinces

interviews. interviews. the guide covered covered guide first phase, data data first phase, final number of final number of roles of different different of roles the The The the standardized e-mail standardized the a remaining stakeholders stakeholders remaining Ministry(VWS), Health of the history of direct involvement historydirectof involvement the a current study aims to evaluate evaluate aims to study current NZa, The The the invitation e-mail). Data saturation was e-mail). was invitation saturation Data the last drugs in 2012. HTA dossiers produced at at produced dossiers HTA last drugs in 2012. Insurance Package Advisory Committee of ZIN of ZIN Advisory Package Committee Insurance the authors selected stakeholders that were directly directly selectedauthors were that stakeholders Appendix for full details of full details for Appendix the the scheme (10). scheme (10). the the Appendix for for Appendix inclusion of of inclusion first MEAs implemented in Europe, no policy evaluation of CF no policy of CF evaluation Europe, in implemented first MEAs CF process. the first study on HTA dossiers (10)) study on HTA first the the the second phase, data was collected from from collected was data phase, second the opportunity during stakeholders other recommend to the preference for senior representatives with representatives senior for preference the a authors used purposeful- and snowballing sampling to select stakeholders to select to sampling to used purposeful- stakeholders authors and snowballing Perceived aims of CF (i.e. which purpose which it served) (i.e. aims of CF Perceived and methodological procedural, to in relation functioning CF (i.e. of Perceived decision-makingto those for these aspects aspects;correspond definitions in implemented scientific assessment committee of ZIN (“Wetenschappelijk Adviesraad”; henceforth henceforth Adviesraad”; (“Wetenschappelijk of ZIN committee assessment scientific specific stakeholder representatives approached were sampled based on seniority based sampled and were approached representatives stakeholder specific Netherlands Organization for Health Research and Development (ZonMW), Development and Health Research for members Netherlands Organization secretariat of drug assessors), pharmacotherapeutic assessors and pharmacoeconomic pharmacoeconomic and assessors pharmacotherapeutic of drug assessors), secretariat following topics: following

• An interview guide was developed for stakeholder interviews. interviews. stakeholder for interview An developed was guide • The The Despite being one of being Despite the

METHODS stakeholders’ experiences in implementing CF in practice. CF in implementing in experiences stakeholders’ invitation (see Figure 1 in (see Figure invitation discussed amongst authors and provided grounds for determining determining for grounds authors and provided amongst discussed interviews conducted. the in CF. All stakeholder representatives were approached using using approached were representatives stakeholder All in CF. involved in implementing CF in practice. Secondly, those who agreed to participate to agreed those who Secondly, CF in practice. implementing in involved provided were The function, with Data Collection Data further collection (see phases in 2 conducted Data was In details below). pharmaceutical industry), was responsible for preparing submissions for both T=0 and T=4 T=4 and T=0 both for submissions preparing industry),pharmaceutical for responsible was uncertaintiesat identified address to proposal study research outcomes an and submitting healthcare NZa), policy public included: CF in (e.g. bodies OtherT=0. involved stakeholders patient and hospitals private and/or academic societies, medical specialists insurers, in 1 Table see Please organizations. stakeholders throughout throughout stakeholders methodological procedural, assess to analyzed recently were CF drugs all T=4 for T=0 and and decision-making aspects of has been conducted since since has been conducted assessors at ZIN. In at assessors was collected from public organizations involved in designing and/or implementing policy implementing and/or in designing involved public organizations from collected was were: stakeholders These conductingand research. the of members of Committee), Assessment (e.g. ZIN at advisors senior Pakket”; Committee), henceforth Appraisal (“Adviescommissie the specialists medical insurers, namely: healthcare industry, pharmaceutical in CF, involved organizations. and patient hospitals private and/or academic societies, participation for (11). Firstly, approach

Use of RWE in HTA practice 6 116 Use of RWE in HTA practice 6 117

topics topics Dutch Dutch authors authors interview quarter of the analysis of of analysis the a the the the final interview Appendix) (13). Appendix) summaries were summaries were interviewees for interviewees for other author for for other author the the The The the the qualitative comparison comparison qualitative different topics. different a consolidated criteria for criteria for consolidated the the Appendix for for Appendix the member check. a extensive summaries generated using using summaries generated extensive codes mentioned by at least at by mentioned codes the the coding performedcoding by analyses. separate coding trees generated by by generated trees coding separate the the feedback received and sent to to sent and received feedback the empty coding tree was structured to reflect structured to was empty tree coding interviews. AM and HN conducted interviews in phase 1 interviews conducted HN and 1 phase in AM interviews. the guides. See Figure 2 in See Figure guides. The The the authors selectedauthors first 3 interviews. Based on feedback from interviewees, minor minor interviewees, from feedback Based on 3 interviews. first the the the CF scheme (i.e. its positive and negative effects on on effects negative and positive its (i.e. CF scheme themes included in included in themes two stakeholder groups deemed relevant for for relevant deemed groups stakeholder two the comparative sub-analysis was conducted for answers to both open- and both open- and to answers sub-analysis for conducted was comparative a interview guide (perceived functioning of CF, impact of CF and conclusions impact conclusions of CF and functioning of CF, interview guide (perceived the large number of codes generated for open-ended questions for three of of open-ended three for for questions generated of codes number large the the the summaries were sent to interviewees for interviewees for to sent summaries were methods used for this study were compared to to compared this study were used for methods interview guide mentioned above. In November 2016, AM and HN conducted HN conducted and 2016, AM In November interview above. guide mentioned healthcare setting) healthcare points improvement aims, its achieved if CF perspectives (i.e. and future Conclusions or replaced) re-introduced stopped, should be schemes and if CF-like CF for Impact of The The preference was made for face-to-face interviews. If these were infeasible, telephone telephone face-to-face infeasible, for made If was interviews. these were preference interview guide included both open- and closed questions. An initial version of of version initial interview An questions. open- and closed both included guide remaining interviews. Any discrepancies in codes generated between between generated in codes discrepancies Any interviews. remaining content analysis for interviews from phase 1. In May 2017, AM and SA conducted phase 1. In and SA conducted interviews AM 2017, May from for analysis content content analysis for interviews from phase 2. Each author coded half of half coded author 2. Each phase interviews from for analysis content guide was piloted in in piloted was guide meaning of meaning topics in topics • • Additionally, Additionally, Due to Due to Based on audio recordings and/or field notes, extensive summaries (3-4 pages) were were summaries pages) (3-4 extensive field notes, and/or Based on audio recordings A The The the stakeholders (≥25%) for further descriptive analyses. Illustrative quotes were cited to clarify clarify to cited were quotes Illustrative further for (≥25%) stakeholders analyses. descriptive the MaxQDA software version 11.0 (VERBI 11.0 software version MaxQDA 10-12 Bismarckstraße Software Location: GmbH, Berlin10625 (12). Germany) of the the Data Analysis Data on conducted was analysis content Directed The The the were resolved by consensus. Finally, Finally, consensus. by resolved were summaries themselves and reviewed and reviewed summaries themselves the adjustments were made to made to were adjustments and future perspectives), and future the interviews from phases 1 and 2 were combined in August 2017 by AM and SA. AM by 2017 August in combined 2 were 1 and phases interviews from functioning of perceived topics: four for and phase 2 in phase 1 provided closed questions enabled This perspectives. future and conclusions impact CF, of CF, of themes that that of themes made. made. between 04.07.2016 and 06.11.2016. Meanwhile, AM and SA conducted interviews in phase interviews and SA conducted AM in phase Meanwhile, 06.11.2016. and 04.07.2016 between 10.05.2017. and 24.03.2017 2 between interviews were held. Stakeholders were asked if interviews could be audio-recorded. Field Field if interviews asked be audio-recorded. could were Stakeholders held. interviews were during taken also were notes guide used. guide reporting qualitative research (COREQ) 32-item checklist (see Table 4 in 4 in Table checklist 32-item (see (COREQ) research reporting qualitative final approval. subsequently edited based on on based edited subsequently year for for year a CF process. CF process. extensive time time extensive the the mechanism to control control to mechanism a aim of CF was to strike strike to aim of CF was stakeholders pertainedstakeholders to Appendix. the stakeholders indicated to have have to indicated stakeholders the promise for additional evidence evidence additional for promise process lasting at least lasting at process the . a the the HO1 controlled environment whereby new new whereby environment controlled a Appraisal Committee (n=3), senior advisers at at advisers senior (n=3), Committee Appraisal the perceived aims. perceived telephone. Each interview lasted between 60 and 90 interview Each 90 and 60 between lasted telephone. the the . Moreover, stakeholders emphasized emphasized stakeholders . Moreover, roles of different stakeholders throughout throughout stakeholders of different roles MS3,PI3 interviews (response rate 100%). Eventually, 35 representatives 35 representatives Eventually, 100%). rate interviews (response the study sample, see Table 1. Table see study sample, the the envisioned 4-year timeframe. For example, stakeholders indicated that that indicated stakeholders example, For timeframe. 4-year envisioned the thus leaving less time for actual collection. data time for less leaving thus PI3 full coding tree developed, see Figures 3a-3d in in 3a-3d Figures see developed, tree coding full Assessment Committee (n=2), (n=2), Committee Assessment summary of a the the majority of stakeholders (17/30; 55%) indicated that that majority 55%) indicated (17/30; of stakeholders ambiguity regarding ambiguity regarding interview recorded.

See Figure 2 for an overview an of 2 for SeeFigure Thirty interviews were conducted between 04.07.2016 and 10.05.2017; 14 for phase 1 phase 1 Thirty 14 for and 10.05.2017; 04.07.2016 between conducted interviews were For For For Another main theme mentioned by 15/30 (50%) of (50%) 15/30 by main theme mentioned Another

balance between quick patient access to drugs and and drugs to access quick patient between balance RESULTS needed to practically set up registries for data collection; data for practically to needed set up registries drugs specific for some indications (e.g. acute diseases), 4 years may be sufficient to collect to meaningful sufficient be may years 4 diseases), acute (e.g. some indications for diseases or orphan longer much diseases), chronic (e.g. other indications for whereas data needed be would follow-up doubts towards towards doubts Perceived functioning of CF Perceived aspects Procedural (90%) of 27/30 aspects of CF, procedural to regards With Perceived aims of CF Perceived The a ZIN (n=4), pharmacotherapeutic assessors (n=4) and pharmacoeconomic assessors (n=2). (n=2). assessors pharmacoeconomic and (n=4) assessors pharmacotherapeutic ZIN (n=4), pharmaceutical from representatives in phase 2 comprised approached Stakeholders academic/ (n=3), societies specialists medical (n=3), insurers (n=5), healthcare companies approached representatives (n=3). All organizations patient and (n=3) hospitals private participate to in agreed Study sample Study bodies public non-ZIN from representatives 1 comprised in phase approached Stakeholders (n=3), interviews Twenty-five interviewees. interviews 2 or more included Three phase 2. and 16 for face-to-face held were over and 5 spanning 30 stakeholders were included. were 30 stakeholders spanning minutes. Audio recordings were made for 29 interviews; one stakeholder refused to have have to interviews; 29 refused one stakeholder for made were recordings Audio minutes. the the healthcare expenditure. Finally, 6/30 (19%) believed that CF had other aims. For example, example, For aims. CF had other that believed 6/30 (19%) Finally, expenditure. healthcare provide to CF aimed that indicated one stakeholder their additional determine to in order practice, with in clinical be experimented drugs could criteria on treatment clear agreements based on value, generation. Meanwhile, 4/30 (13%) stakeholders indicated that CF only aimed to promote promote only aimed to CF that indicated 4/30 (13%) stakeholders Meanwhile, generation. merely it was that argued (13%) 4/30 and drugs to earlyaccess

Use of RWE in HTA practice 6 118 CF

policy policy

new new introduce ‐ access)

with with

pathways)

MEAs and (early (evidence (other)

CF CF

CF/

Stop No Partially Partially generation) Partially Replace (adaptive Replace (Other) Improve (n=30)

(n=30) aims? its

50% 37% perspectives achieve

30% 6% CF 14% Future

28% 8% Did

Use of RWE in HTA practice 27% 6 119 . ZA1 loss loss a . Other. data) mid-term mid-term a the drugs are not not drugs are vs. . Additionally, . Additionally,

CF procedure, CF procedure, PE3,ZA3,FG1 HI2 the . Stakeholders also also . Stakeholders the . access access

ZS4 MS2 healthcare

generation early early

incentives structure to generate generate structure to incentives division of roles and conflicting and conflicting of roles division Balance: Promote Control Other evidence expenditure (lack of) embedded mechanisms design flaw in flaw design the procedure (8/30 stakeholders; 27%). 27%). stakeholders; (8/30 procedure a the the . (n=30) the

PE2 CF mechanisms for financing outcomes research research outcomes financing for mechanisms

pharmaceutical industry. Stakeholders argued argued Stakeholders industry. pharmaceutical of the

the independence of research conducted research of independence 55% aim

the opinion that while procedures at T=0 were clear, much was was much clear, were T=0 at procedures while that opinion relationship between between relationship outcomes research study (e.g. at T=1 year or T=3 years); ZIN ZIN T=3 years); or year T=1 at study (e.g. research outcomes guidelines was an interim time point scheduled for for scheduled point time an interim was guidelines .Therefore, stakeholders mentioned that once reimbursement reimbursement once that mentioned stakeholders .Therefore, the reimbursement package. For industry, this would result in result this would industry, package. For reimbursement authority to enforce such mid-term reviews such mid-term authority enforce to the the medical specialist, it would mean that patients would likely stop stop likely would patients that mean would it specialist, medical ZW2 the the data would lead to that conclusion, there would have been reason to to been reason have would there conclusion, that lead to would data the reimbursement package at T=0, T=0, package reimbursement at the process and roles of different stakeholders at T=4 at stakeholders different of and roles process the Perceived 19% lack of monitoring meant that errors encountered at T=4 (e.g. regarding regarding (e.g. T=4 at encountered errors that meant monitoring lack of 13% the . If the progress in progress monitoring of progress throughout throughout of progress monitoring ZW2 the disregard of disregard 13% drug from drug from the the the the – Stakeholder views on perceived aim of conditional financing (CF). financing aim of conditional on perceived views – Stakeholder results of evidence generated through outcomes research to ZIN at T=4 (i.e. T=4 (i.e. ZIN at to research outcomes through generated of evidence results Another less frequently mentioned theme was was theme mentioned less frequently Another Thirteen stakeholders (13/30; 43%) referred to to Thirteen 43%) referred (13/30; stakeholders data collection or analysis) could no longer be retrospectively corrected no longer be retrospectively could collectiondata or analysis) of revenue, whilst for whilst for of revenue, treatment their receiving evidence drastically shiftedamongst stakeholders drastically evidence review of review For example, in none of example, For interests of stakeholders. Pharmaceutical industry and medical specialists, tasked with industry with Pharmaceutical tasked stakeholders. specialists, medical of and interests been have not may respectively, T=0, after research outcomes and implementing financing that indicate could that evidence collect robust to inclined intrinsically cost-effective studies, which was mainly paid for by by mainly paid for was which studies, impacted this negatively that some stakeholders (8/30 27%) referred to to referred (8/30 27%) some stakeholders pharmaceutical industrypharmaceutical who collected specialists holders or medical as dossier For example, it was not obvious which stakeholder was responsible for communicating communicating for responsible was stakeholder which obvious not it was example, For the Figure 2 Figure iterated that that iterated remove remove was granted from from granted was Another stakeholder was of was stakeholder Another unknownabout in CF for in CF for was also not provided provided also not was namely

Geneesmiddelen (VIG) Geneesmiddelen

Yes Face-to-face 20.04.2017 PI4 1 Vereniging voor Innovatieve Innovatieve voor Vereniging

Pharmaceuticals B.V. Pharmaceuticals

19.04.2017 PI3 1 Bristol-Myers Squibb Squibb Bristol-Myers Yes Face-to-face

No Telephone 19.04.2017 PI2 2 B.V. Pharma Novartis

Yes Face-to-face 13.04.2017 PI1 1 B.V. Pharmaceuticals Janssen Industry Pharmaceutical

Yes Face-to-face 16.11.2016 FG2 2 assessors Pharmacoeconomic

Yes Face-to-face 27.10.2016 FG1 4 assessors Pharmacotherapeutic Assessors Drug – ZIN

Yes Face-to-face 30.08.2016 ZS4 1 adviser Senior

Yes Face-to-face 31.08.2016 ZS3 1 adviser Senior

Yes Face-to-face 01.09.2016 ZS2 1 adviser Senior

Yes Face-to-face 06.09.2016 ZS1 1 adviser Senior Advisers Senior – ZIN

Yes Face-to-face 14.07.2016 ZA3 1 member Committee

Yes Face-to-face 12.07.2016 ZA2 1 member Committee

Committee

Committee member Committee Face-to-face 07.09.2016 ZA1 1 Yes ZIN – Appraisal Appraisal – ZIN

Committee member Committee Face-to-face 11.07.2016 ZW2 1 Yes

Committee

Committee member Committee Telephone 20.07.2016 ZW1 1 Yes ZIN - Assessment Assessment - ZIN

zorginnovatie (ZonMW) zorginnovatie

voor gezondheidsonderzoek en en gezondheidsonderzoek voor

Face-to-face 20.07.0216 PE3 1 Yes De Nederlandse organisatie organisatie Nederlandse De

Welzijn en Sport (VWS) Sport en Welzijn

Face-to-face 04.07.2016 PE2 1 Yes Ministerie voor Volksgezondheid, Volksgezondheid, voor Ministerie

Research Bodies Research

Nederlandze Zorgautoriteit (NZa) Zorgautoriteit Nederlandze PE1 1 18.07.2016 Yes Face-to-face External Public Policy/ Policy/ Public External

(yes/no) Stakeholder Group Stakeholder Stakeholder of Interviewees of Interview Code Interview of Interview of of Interview of

Number Date Manner Interview recorded recorded Interview

vs. telephone) and whether whether and telephone) vs. interview was recorded. was interview the

– Summary of of Summary – 1 Table stakeholders interviewed, number of interviewees per interview, interview code, date of interview, manner of interview (face-to-face (face-to-face interview of manner interview, of date code, interview interview, per interviewees of number interviewed, stakeholders the

Use of RWE in HTA practice 6 120

Use of RWE in HTA practice 6 121

Abbreviations: ZIN: Zorginstituut Nederland Zorginstituut ZIN: Abbreviations:

Yes Face-to-face 03.04.2017 PO3 1 Longfonds

Kankerpatientorganisaties (NFK) Kankerpatientorganisaties

Yes Face-to-face 08.05.2017 PO2 1 Nederlandse Federatie voor voor Federatie Nederlandse

en Stofwisselingsziekten (VKS) Stofwisselingsziekten en

Yes Face-to-face 04.04.2017 PO1 1 Vereniging Volwassen, Kinderen Kinderen Volwassen, Vereniging Organizations Patient

Ziekenhuizen (NVZ) Ziekenhuizen

Yes Telephone 25.04.2017 HO3 1 Nederlandse Vereniging van van Vereniging Nederlandse

Centrum (AMC) Centrum

Yes Face-to-face 03.05.2017 HO2 1 Medisch Academisch Het

Centra (NFU) Centra

Universitair Medische Universitair

Hospitals

Face-to-face 21.04.2017 HO1 1 Yes Nederlandse Federatie van van Federatie Nederlandse Private/ Academic Academic Private/

Nederland (iKNL) Nederland

Face-to-face 10.05.2017 MS3 1 Yes Integraal Kankercentrum Kankercentrum Integraal

Samenwerking (SONCOS) Samenwerking

Face-to-face 09.05.2017 MS2 1 Yes Stichting Oncologische Oncologische Stichting

Antibioticabeleid (SWAB) Antibioticabeleid Societies

Face-to-face 24.03.2017 MS1 1 Yes Stichting Werkgroep Werkgroep Stichting Medical Specialists Specialists Medical

Zorgverzekeraars Nederland (ZN) Nederland Zorgverzekeraars Face-to-face 24.04.2017 HI3 1 Yes

1 Menzis HI2 Yes Telephone 04.04.2017

Healthcare Insurers Healthcare Zorgverzekeringen VGZ Zorgverzekeringen 1 HI1 01.05.2017 Telephone Yes

(yes/no) Stakeholder Group Stakeholder Stakeholder of Interviewees of Interview Code Interview of Interview of of Interview of

Number Date Manner Interview recorded recorded Interview

- continued - 1 Table new new main main . Other. the duration duration FG1 The The case with with case the the time of development time of development quality of outcomes quality of outcomes disease areas for which which for areas disease the the the disease area where new drugs are drugs are new where area disease role were alglucosidase alpha and and alpha alglucosidase were role perceived focus on excessive data data excessive on focus perceived a . This often led to an inflated list of list of an inflated often to led This . a . Moreover, at at . Moreover, intervention and control groups were were groups and control intervention ZS4 the PI1,ZS3 the drugs that may have been due for investigation investigation due for been have may drugs that . Moreover, different combinations of oncology oncology of combinations different Moreover, . design of outcomes research studies conducted conducted studies research outcomes of design impact of rapid changes in clinical practice on impact in clinical practice on changes of rapid start of CF summer of 2012, ZIN was mentioned in national in national mentioned ZIN was summer of 2012, the . . the the the PI3 the treatment of Pompe’s and Fabry’s diseases, respectively. respectively. diseases, Fabry’s and of Pompe’s treatment PE3,ZS2,FG1,HI3 stakeholders, there was no clear methodological guidance guidance methodological no clear was there stakeholders, result, result, primary of example control group, leading to potential selection bias potential to leading group, control the a a the the advice issued by ZIN at T=4 on drug reimbursement. reimbursement. T=4 on drug ZIN at by issued advice . control group or that or that group control latter case, patients who did not wish to be treated with treated be to who did not wish patients case, latter ZS2 the a ZW2,ZS3,ZS4,PI1,PI3,MS3 the rapid pace. As As pace. rapid a daily basis due to its preliminary advice to remove these orphan drugs from these orphan from drugs its preliminary remove daily basis due to to advice absence of absence a . drugs in question was difficult to reach to difficult was drugs in question outcomes research study research outcomes lack of consensus on methodological issues, particularly with regards to core health health core particularly to issues, on methodological regards lack of consensus with policy question PO2 the relevance of evidence generated through outcomes research studies (10/30; 33%). 33%). (10/30; studies research outcomes through generated evidence of relevance

Furthermore, 10/30 (33%) of stakeholders indicated that that indicated of stakeholders (33%) 10/30 Furthermore, Some stakeholders (8/30; (27%) mentioned Some mentioned (27%) (8/30; stakeholders Stakeholders also emphasized also emphasized Stakeholders the the the the

significant effect on effect significant and/or consensus with regards to to regards with and/or consensus only was design study on guidance methodological example, For T=4. and T=0 between years after by ZIN in 2008, 2 finalized research studies conducted was generally poor. Recurring problems in these studies studies these in Recurring problems poor. generally was studies conducted research were In not comparable. Methodological aspects Methodological (40%) of 12/30 to According aspects such as low patient recruitment and fragmented data collection in practice also collection also practicedata in fragmented and recruitment aspectspatient low as such impacted study quality drug automatically became became automatically drug treatments also became introduced after study designs for monotherapies were finalized finalized were monotherapies for after designs study introduced also became treatments T=0 at examples whereby political pressure played such played pressure political whereby examples in that recalled One stakeholder on news agalsidase alpha- and beta for and beta for alpha- agalsidase Decision-making aspects external that factors had interviewed stated stakeholders 50%) all of Half (15/30; a outcomes sets and clinical parameters sets. parameters sets and clinical outcomes collection on non-critical parameters. This supplements points previously mentioned above above mentioned previously points supplements This collection non-criticalon parameters. on stakeholders indicated that financing structures varied per registry, as was was as registry, varied structures financing per that indicated stakeholders governance structures for registries structures for governance of CF drugs, there was often limited medical knowledge often limited was on there of CF drugs, relevant are that sets outcome core on consensus Therefore, developed. were CF drugs to parameters for which data needed to be collected that were, in hindsight, of little relevance of little relevance hindsight, in were, that be collected to needed data which for parameters to of in second-line treatment at T=0 became standard first-line treatments within within treatments first-line standard became T=0 at treatment in second-line the as mentioned was Oncology introduced at at introduced

Use of RWE in HTA practice 6 122 Use of RWE in HTA practice 6 123

. . and/or and/or FG2,HI3,HO2 design of of design ZS3,ZS4,HI1,HI2 . the ZA3 FG2,PI2,MS3 impossibility of impossibilityof Dutch healthcare healthcare Dutch . treatments outcomes research research outcomes ZS3 healthcare system in in system healthcare the the drugs included in CF, in CF, drugs included . In general, this was was this . In general, the the the the PE2,FG2 . value-based of assessment result of CF, cost-effectiveness cost-effectiveness CF, of result fact that their negotiation power fact power negotiation their that a PE3,FG1,PI1 studies cited above cited studies the opinion that CF delivered valuable valuable delivered CF that opinion opinion that that opinion and for the for and the sustainability of design of better future schemes future of better design the the positive effects of CF in CF of effects positive FG2,HI1 the the the shortcomings encountered in CF implementation in CF implementation shortcomings encountered The The . . Pressure from patient organizations and medical societies societies medical and organizations patient from . Pressure ZS1 general public) on public) on general CF scheme (e.g. eculizumab and pertuzumab)eculizumab (e.g. CF scheme use of real-world evidence (RWE) evidence making decision in real-world use of reimbursement package at T=4, even if ZIN’s advice recommended recommended advice if ZIN’s even T=4, package reimbursement at PE2,ZA2,HI1,MS2 the reference value of €80.000 per quality-adjusted life-year (QALY) (14) (14) (QALY) life-year per quality-adjusted of €80.000 value reference the need for multi-stakeholder collaboration to make MEAs successful successful MEAs make to collaboration multi-stakeholder need for . Other stakeholders stated that while outcomes research studies studies research . Other while outcomes that stakeholders stated the the . Another stakeholder referred to to referred stakeholder Another . displacement of healthcare due to exorbitant drug expenditures drug expenditures exorbitant to due of healthcare displacement policy perspective. PE1 methodological limitations of limitations methodological the . a the PI2 the feat the ZW2,FG2,HO2 legal implications associated with drug removal were often deemed too large large often too deemed were removal with drug associated implications legal the The The . appropriate use of drugs in clinical practice. In some instances, this allowed medical medical this allowed In instances, some practice. use of drugs in clinical appropriate HI3 third of stakeholders (10/30; 33%) stated that for many of many for that 33%) stated (10/30; stakeholders of third result of result reimbursement packagereimbursement Thirdly, 10/30 stakeholders (33%) were of of were (33%) stakeholders 10/30 Thirdly, Many stakeholders (14/30 (43%) expressed expressed (43%) (14/30 stakeholders Many Another theme referred to by stakeholders (12/30; 40%) was was 40%) (12/30; stakeholders by to referred theme Another Finally, some stakeholders identified that CF increased awareness amongst all amongst all awareness CF increased that identified stakeholders some Finally, A Secondly, 12/30 (40%) stakeholders stated that valuable RWE was generated, particularly RWE valuable that generated, was stated (40%) stakeholders 12/30 Secondly, the light of high drug prices of high drug light would provide concrete recommendations for for recommendations concrete provide would experiences from from experiences studies conducted as part of CF contributed little to decision making at T=4. In fact, several InT=4. fact, several part as conducted studies making decision to little at of CF contributed T=4 in comparison at diminished rarely uncertainties that were indicated stakeholders analyses cost-effectiveness particularlyT=0; to regards with to to do so. so. do to attempt to drugs that came after that drugs provided useful, product-specific insights, they did not deliver any new conclusions for for conclusions new any did not deliver they insights, product-specific useful, provided makingdecision the Several themes were identified in relation to relation in identified themes were Several as that stated 16/30 (53%) stakeholders Firstly, setting. and of drugs amongst all created was awareness In discussion. of societal topics became other words, (including stakeholders Impact of CF scheme Two stakeholders asserted that such learnings have already been applied for for applied been already asserted stakeholders learningssuch have that Two (9)) Toelating” (“Voorwaardelijke MEAs ongoing skepticism regarding regarding skepticism removing drugs from drugs from removing and their argumentation to discontinue drug reimbursement was highly compromised compromised highly was drug reimbursement discontinue to argumentation and their at T=4 at the stakeholders for for stakeholders conclusions at T=4 on appropriate use and cost-effectiveness were predictable based on predictable on based were use and cost-effectiveness on appropriate T=4 at conclusions drugs for some (ICERs) ratios cost-effectiveness incremental example, T=0. For at insights so far above were practice was their use in that proven have could evidence of additional no amount that cost-effective on treatments start- develop criteria for to and stop societies on decision makers was also high so as to not deny patients access to to access patients not deny so as to also high makers was on decision

. ZA1,ZA2 CF

inclusion inclusion need for need for start policy policy

the the the scheme, as well as well scheme, . new new fact that greatly fact greatly that introduce ‐ access) aim and relevance relevance aim and

goal of (real-world) goal of (real-world) a re framework whereby whereby framework the

a with with

the pathways) the stakeholders answered answered stakeholders

MEAs and (evidence (other) (early

CF CF

ZS1,FG2,PI1,MS3 the CF/

need for need for Partially generation) Partially Replace Replace Improve Stop No Partially (adaptive (Other) the (n=30)

(n=30)

negative effects of CF met of effects negative stakeholders emphasized emphasized stakeholders the aims and importance of the aims?

the . Moreover, they thought that CF drugs should not not CF drugs should that thought they . Moreover, its ZS3

responses were divided as follows: 8/15 indicated that that indicated 8/15 as follows: divided were responses 50% 37% the stakeholders emphasized emphasized stakeholders perspectives scheme

the the importance of inter-stakeholder collaboration to achieve these these achieve to importance collaboration of inter-stakeholder achieve

). 30% 6% . PO3 the CF 14% Future

themes identified regarding regarding themes identified 28% 8% outcomes of outcomes Did PI1,PI2,HI2,PO3 the 27% the - Stakeholder views on achievement of aims of conditional financing (CF). financing aims of conditional of on achievement views - Stakeholder recognition of recognition CF scheme. Firstly, 11/30 (37%) of 11/30 (37%) Firstly, CF scheme. underlying incentives structure can ensure that different stakeholders take up their take up their stakeholders different that structure can ensure underlying incentives

goal of early patient access to drugs was met, 3/15 indicated that that 3/15 indicated met, drugs was to access goal of early patient None of Secondly, 10/30 (33%) of (33%) of 10/30 Secondly, Two main themes were identified with regards to improvement points for for points improvement to regards with identified were themes main Two a

consensus amongst all stakeholders on all stakeholders amongst consensus as criterion (i.e. were mentioned by <25% of stakeholders). <25% by mentioned were criterion (i.e. responsibilities and be held accountable if such responsibilities are not met. For example, example, For met. not are responsibilities such if accountable held be and responsibilities sanctions no CF included that mechanisms, indicated stakeholders the impacted aims. These diverging interests impact stakeholders’ perception of perception impact stakeholders’ interests diverging These aims. of CF-like schemes, implying that consensus on scheme aims is vital from aims is vital from on scheme consensus that implying of CF-like schemes, Figure 3 Figure (8/30; 27%). This collaboration spans all stakeholder groups mentioned above and extends and extends above mentioned groups all stakeholder spans collaboration This 27%). (8/30; design, study research) (outcomes design, scheme including: aspects of MEAs, several into making and decision collection,data value on added discussions “No”, 15/30 (50%) answered “Partially” and 0/30 (0%) answered “Yes” (see Figure 3). For those those 3). For (see Figure “Yes” answered (0%) and 0/30 “Partially” answered 15/30 (50%) “No”, (n=15), “Partially” who answered the its aims CF fulfilled that other aspects 4/15 indicated and met (e.g. was generation evidence drugs specific for only When asked if CF had achieved its perceived aims, 15/30 (50%) of (50%) 15/30 aims, its perceived achieved had asked if CF When Conclusions and Future Perspectives and Future Conclusions In light of this, collaborative efforts on designing and implementing CF-like schemes are are CF-like schemes efforts and implementing on designing collaborative In of this, light also essential the

Use of RWE in HTA practice 6 124 vs.

access

healthcare

generation early early

Control Other Promote Balance: expenditure evidence (n=30)

55% aim

Perceived 19% 13% 13% Use of RWE in HTA practice 6 125 CF

. policy policy

FG1 improvement of of improvement scientific concept concept scientific new new introduce temporary funding ‐ access)

a a re

the with with opinion that definitive definitive that opinion

few stakeholders (2/30, stakeholders (2/30, few pathways) budget impactbudget of these

points mentioned above above mentioned points MEAs and (early (evidence (other)

CF CF

the the CF/ the . Furthermore, they indicated indicated they . Furthermore,

Improve Stop Replace Replace (adaptive (Other) No Partially Partially generation) Partially FG1,PI4 . . In CF, patient inclusion was done on done on was inclusion patient . In CF, (n=30)

(n=30)

MS1,MS2 FG1,ZA1 future, 11/30 (37%) stakeholders suggested suggested stakeholders 11/30 (37%) future, the aims?

reference value of €80.000/QALY). Subsequently, Subsequently, €80.000/QALY). of value reference its

50% 37% the perspectives reimbursement package, but rather from from package,rather but reimbursement

need for better governance structures and distribution of roles, structures and distribution of roles, governance better need for the achieve scheme that resembles adaptive pathways; pathways; adaptive resembles scheme that

the a 30% 6% CF 14% Future

28% need for monitoring procedures and pre-defined time points for progress and pre-definedfor progress procedures monitoring time points need for drug should have been considered have drug should 8% . See Figure 4 for views on future perspectives in relation to CF. to perspectives in relation future on views 4 for . See Figure Did 27% the the

. Knowing that drug availability is thus temporary, stakeholders responsible for for responsible stakeholders temporary, thus is availability drug Knowing. that ZA3,HI3 – Stakeholder views on future perspectives. on future views – Stakeholder or enforce strict reference values for ICERs for values strictreference or enforce FG1 PE2

use of electronic health records (EHRs) for evidence generation. Additionally, some some Additionally, generation. evidence for (EHRs) use of electronic records health CF scheme were were CF scheme When asked how to proceed with CF in proceed to how asked When Other topics mentioned by stakeholders (9/30; 30%) on 30%) (9/30; stakeholders Other by mentioned topics voluntary basis, leading to many under-powered studies and selection studies bias under-powered many leading to voluntary basis, Figure 4 Figure MEA: Managed Entry Agreement. Financing; Conditional CF: Abbreviations: have been financed from from financed been have source that conditions of obligatory inclusion of patients in outcomes research in return for for in return research obligatory of outcomes in conditions that of patients inclusion to access data collection would thus be better incentivized to do so do to incentivized better thus be collectiondata would 7%) suggested to stop all forms of CEDs; in their opinion, such schemes do not work in in work schemes do not such in their opinion, CEDs; of forms all stop to 7%) suggested practice to replace CF with CF with replace to but also of were 30%) (9/30; some stakeholders Additionally, reviews. the conclusions should be formulated regarding highly unfavorable ICERs at T=0 (i.e. those those T=0 (i.e. at ICERs highly unfavorable regarding formulated should be conclusions disproportionately than higher reduce to measures reach to choose could makers decision drugs a for medicine development and data generation whereby an iterative approach to to approach iterative an whereby generation data and development medicine for Meanwhile, (15;16). lifetime their drugs throughout for is adopted generation evidence pricing or adaptive as new policies such other CF with replace to suggested (30%) 9/30 the on CF based improve to 27%) suggested (8/30, stakeholders during Finally, interviews re-introducing it. subsequently then

vs.

access

healthcare

generation early early

Other Balance: Promote Control evidence expenditure (n=30)

55% aim

Perceived 19% 13% 13% CF result, result, 4-year 4-year the a perceived perceived the CF scheme. CF scheme. CF scheme CF scheme other hand, other hand, first study on on first study the appendix. the the the studies provided provided studies the the other half believed believed other half the aim of the the critical need for clear frameworks frameworks clear criticalfor need a Dutch healthcare setting (e.g. open open (e.g. setting healthcare Dutch the research questions on cost-effectiveness. on cost-effectiveness. questions research wide array of different stakeholders, each with each with stakeholders, of different wide array the a impact of external factors making on decision at dossiers analysis indicating that only couple of all CF CF of all only couple that indicating analysis dossiers dossiers analysis indicated that that indicated analysis dossiers the Appendix, respectively). Similarly, themes identified for for identified themes Similarly, respectively). Appendix, envisioned period. Moreover, stakeholders indicated that that indicated stakeholders Moreover, period. envisioned the the majority of stakeholders indicated that CF should either be either be should CF majority that indicated stakeholders of complex network of interactions amongst them. As them. As amongst network interactions of complex the the negative reimbursement advice at T=4. On T=4. at advice reimbursement negative a The The a Appendix for an overview of all themes identified for an overview identified of all themes for Appendix governance perspective, there is there perspective, governance a the findings summarized above correspond to those from to those from correspond above summarized findings findings above. These themes can be found in Table 5 of Table 5 in found be themes can These above. findings the the 4-year timeframe, poor methodological quality of outcomes research studies and and studies research quality poor methodological of outcomes timeframe, 4-year questions on whether CF had achieved its aims and how to proceed with CF in with CF in proceed to how its aims and CF had achieved on whether questions future (Figures 4 and 5 in 5 and 4 (Figures future Appendix for illustrative quotes per theme. quotes illustrative for Appendix

stakeholders thought that CF had partially CF that thought stakeholders it aims while achieved See Table 3 in Table See Healthcare systems worldwide include include worldwide systems Healthcare Some of

DISCUSSION This study examined experiences of stakeholders in implementing CF in Dutch practice. practice. CF in Dutch in implementing of stakeholders experiences examined study This of perceptions different had stakeholders that Results indicated Comparative sub-analysis 2 stakeholders) 1 and phase (phase Comparative namely did not varygroups; closed questions Responses both to between significantly the the outcomes research studies were often of low methodological quality thus of little relevance quality methodological relevance little thus of often of low were studies research outcomes making. decision Meanwhile, to interests. specific to their trade-offs relation in for each stakeholder different present MEAs from Consequently, functioning of CF, impact of CF, and improvement points for CF. Additionally, see Table 4 in 4 in Table see Additionally, CF. for points and improvement impact of CF, functioning CF, of the open questions did not vary significantly. Several unique themes were identified which which identified were themes unique Several vary did not questions open significantly. supplement Moreover, stakeholders highlighted numerous shortcomings how in numerous highlighted stakeholders Moreover, decision-makingand methodological aspects (e.g. procedural, to functionedregards with the stakeholders to this, respectively). contrast T=4, In at on advice external influence political on CF of effects positive several mentioned of Half of healthcare). and displacement of drugs cost-effectiveness on discourse public the HTA dossiers analysis (10). One example relates to stakeholders’ critique on stakeholders’ to relates One example (10). analysis dossiers HTA short often only one CF too being that indicating dossiers for timeframe findings from and within completed CF drug was of half almost for evidence inadequate from findings to T=4 correspond received drugs eventually not be could complementary that topics on interviewsstakeholder insights provided also impact of perceived stakeholders’ such as analysis, dossiers HTA through addressed perspectives. future and on CF conclusions scheme, and mandates their differing Finally, stakeholders’ emphasis of stakeholders’ Finally, it had not achieved its aims. its aims. it had not achieved pathways). adaptive new policy with (e.g. or replaced and re-introduced improved

Use of RWE in HTA practice 6 126 Use of RWE in HTA practice 6 127 consequence consequence analyses (e.g. (e.g. analyses availability of of availability establishment establishment a methodological methodological analysis of RWD; analysis the the the the the need for considerable considerable for need the conduct reporting and of RWD validity of RWE (32;33). However, validity of RWE However, (32;33). the the results (e.g. HTA agencies). As As agencies). HTA (e.g. results the product- or disease registries for separate separate for registries or disease ad hoc product- training of personnel conducting of personnel training analysis of real-world data (RWD) and interpretation (RWD) data and interpretation real-world of analysis scheme aims, thus their own gains, greatly matter; matter; greatly gains, thus their own scheme aims, financing of outcomes research studies, it would be would be it studies, research of outcomes financing the the the absence of “exit strategies” (21). Although such concepts concepts such Although (21). strategies” “exit of absence the the high workload experienced by healthcare professionals in in professionals healthcare high workload by experienced International Society for Pharmacoeconomics and Outcomes Outcomes and Pharmacoeconomics SocietyInternational for the the implementation of CEDs poses additional challenges relating to to relating poses additional challenges of CEDs implementation the current model for creating creating for model current absence of sanctions mechanisms. Previous experiences from England also also England from experiences Previous of sanctionsabsence mechanisms. The The the necessary infrastructure in place, healthcare professionals in clinical practice would in clinical practice would professionals healthcare necessary in place, infrastructure In particular, In particular, Another importantAnother is challenge methodological perspective, many advances have been made in been made in have advances many perspective, methodological point to problems arising from arising from problems to point costs, including: reasons, various due to unsustainable be may questions research (5;22;23). research for accessibility and data registration extra of data burden administrative collection data for systems (digital) establish needed to are investments major Meanwhile, In (6;21). or EHRs of light questionnaires paper-based through whether and analysis, on above comments stakeholders’ for financing responsible should be specify which stakeholders to difficult on governance may seem quite elementary, their importance cannot be underestimated importance their be underestimated cannot elementary, seem quite may on governance literature. and in in this study both their recurrence provided decision for analysis collection data data subsequent and (real-world) for infrastructure making. Finally, investments. such to commit to difficult be thus may it general, need to be trained to use such IT systems, requiring both financial and time investments and time investments financial both requiring use such IT systems, to trained be need to Provided on their behalf. of information technology (IT) infrastructures for implementing EHRs. Even with (IT) technology with of information Even EHRs. implementing for infrastructures the difficulties associated with analyzing RWD and using RWE in decision making (26-28). From with analyzingFrom associated difficulties RWD and using RWE in decision making(26-28). a of real-world evidence (RWE). Findings above allude to skepticism amongst decision makers decision makers amongst skepticism to allude above (RWE). evidence Findings of real-world to articles numerous refer RWE. on decisions in basing Furthermore, that entail stakeholders’ roles, responsibilities, incentives and sanctions (18;19). To begin begin To sanctions and (18;19). incentives responsibilities, roles, stakeholders’ entail that of perceptions stakeholders’ with, in Germany, similar schemes failed due to clinicians perceiving them as posing limitations limitations them as posing perceiving clinicians to due failed schemes similar in Germany, nearly to impossible remains still it in Italy, Meanwhile (20). choices on their prescribing possibly (6), schemes pay-for-performance from patients non-responder for costs reclaim due to as purposes, analysis for data to access guarantee does not automatically EHRs within data (23-25). in literature examples numerous by illustrated implementing state-of-the-artimplementing training extensive RWD requires for methodology analysis again yet implying biostatistics; and pharmacoepidemiology in for and time investments financial Research (ISPOR) and International Society for PharmacoEpidemiology (ISPE) provide an an (ISPE) provide Society PharmacoEpidemiology and International (ISPOR) Research for for on good procedures of clear guidance example both alone or in combination with RCT data (29-31). Moreover, recent guidelines issued issued guidelines recent RCT with both alone or in combination Moreover, (29-31). data effortscombined of by industry)pharmaceutical or interpreting still stakeholders and private public in both makers decision of factors above, discussed (23). RWE processes incorporating in in current experience little have studies to increase decision makers’ confidence in confidence makers’ decision increase to studies

future, future, authors authors research research thorough thorough the a the the implementation implementation development of of development reimbursement of of reimbursement routine analysis of of analysis routine context of adaptive of adaptive context the the challenges associated associated challenges the the qualityof the authors are not aware of of aware not are authors the implementation of CF in of CF in implementation the studies provide provide studies the the the national CED in an attempt to to CED in an attempt national scope of this study would thus thus would study this of scope a the authors are aware of of aware are authors Appendix cover additional themes that may not not may that themes additional cover Appendix interviews for this study. However, However, this study. interviews for the the the policy of evaluation a means for routine collection of patient-level data in practice practice in data collectionpatient-level of routine for means challenges, most stakeholders encouraged encouraged stakeholders most challenges, future. Ideally, evidence generated through through generated evidence Ideally, future. the design of better CEDs. of better design the stakeholders, is not standard. However, However, is not standard. stakeholders, the the the number of interviews. number interview analyses conducted provide complementary findings to those from complementaryto those from interview findings provide conducted analyses 25% threshold implemented. threshold 25% development of future schemes. Ideally, Ideally, schemes. future of development the the the threshold implemented to select and include themes from content analysis, using using analysis, select content to and include themes from implemented threshold the existence of standard thresholds for such criteria in literature on qualitative methods. methods. qualitative on such criteria in literature for thresholds of standard existence previous study on HTA dossiers analysis (10). Together, Together, (10). analysis dossiers HTA study on previous COREQ checklist for validation of both aspects. of both aspects. checklist validation for COREQ

The The Finally, this study represents represents this study Finally, However, despite despite However, Finally, Finally, comprehensive range of stakeholders were included, spanning different stakeholder stakeholder different spanning included, were stakeholders of range comprehensive

EHRs would subsequently facilitate iterative HTA of drugs within of drugs within HTA iterative facilitate subsequently EHRs would principles of whereby exist currently concepts on similar literature in Examples pathways. clinical patient personalized generate to EHRs existing to applied artificialare intelligence Though (35;36). practice in pathways have met met have at least 25% of 25% least at used several sampling methods (purposive sampling and snowballing) to ensure that that ensure to and snowballing) sampling methods (purposive sampling used several a grounds and provided authors amongst discussed was saturation data Moreover, groups. limiting for the in cited quotes illustrative Moreover, new CEDs to address dilemmas encountered in decision making in decision on encountered dilemmas address to new CEDs inform to in order drugs. Literature also alludes to increasing trends in conditional marketing authorizations marketing authorizations in conditional trends increasing to alludes also Literature drugs. (particularly and oncology for HTA uncertainties for larger in evidence with relatively Arguably, use (5;22). MEAs in trends orphan (34) and increasing drugs) provide EHRs may of for information robust more delivering analyses, for advanced detail is of sufficient that decision-making in in HTA into incorporated be could they that it is our hope approaches, with such the In this study, standardized methodology was implemented for identifying stakeholders, stakeholders, identifying for implemented was methodology standardized In study, this content conducting and Furthermore, interviews. representatives stakeholder approaching discrepancies all with authors two by conducted was group each stakeholder for analysis Moreover, consensus-seeking by authors. amongst addressed Strengths inform inform Limitations individual all not include could we involved, were groups stakeholder all relevant Although CF in with involved stakeholders Dutch practice. Dutch and systematic evaluation of experiences gained with with gained of experiences evaluation and systematic conducted and subsequent reporting thereof was compared to recommendations of of recommendations to compared reporting was and subsequent conducted thereof the

Use of RWE in HTA practice 6 128 Use of RWE in HTA practice 6 129 authors authors need for need for design of of design the the onslaught of of onslaught the the placement of placement positive impact positive CF on of implementation of MEAs of MEAs implementation the the the functioning of CF (i.e. relating to to functioning relating of CF (i.e. future. However, stakeholders diverge diverge stakeholders However, future. the the CED scheme). Results demonstrate differences differences ResultsCED scheme). demonstrate a aim of informing ongoing international discussions discussions international ongoing of informing aim future. the perceived aim of CF. Conversely, there is some agreement is some agreement there Conversely, CF. of aim perceived shortcomings in the the United Kingdom (21)). However, KingdomUnited However, (21)). complexity of designing and implementing CEDs, we therefore therefore we CEDs, implementing and of designing complexity the same aims of CF in the the the stakeholders on stakeholders the design and implementation of future MEA schemes. Provided Provided MEA schemes. of future and implementation design thorough, systematic analysis of CF. Such access may not be equally facilitated in in facilitated not be equally may access Such CF. of analysis systematic thorough, Dutch healthcare setting and improvement points for CF. Despite stakeholders’ belief belief stakeholders’ Despite CF. for points setting and improvement healthcare Dutch United States (5) and and (5) States United design of better schemes in schemes in of better design a the This study was conducted with conducted was study This CONCLUSION encourage further research on experiences gained in gained further experiences on encourage research other settings. Provided Provided other settings. within Dutch institutions provided extensive access to national stakeholders thus allowing allowing thus stakeholders national to access extensive provided institutions Dutch within for complementary learnings for provide to countries CEDs) in other (including schemes. future on whether CF should be improved based on learnings identified and re-introduced into re-introduced into and learnings on based identified be improved CF should on whether policy new with practice schemes. or replaced on innovative, yet expensive drugs and increasing trends of MEAs use by HTA agencies and and agencies HTA use by of MEAs trends drugs and increasing expensive yet innovative, inform critical is thus to gained with other MEAs further experiences on research payers, the procedural, methodological and decision-making methodological aspects), procedural, that CF only partially met its aims, if not at all, there is still agreement on agreement is still there CF only partiallythat all, if not at aims, met its the new policy address to This study provides insights on stakeholders’ experiences in implementing CF in Dutch Dutch CF in in implementing experiences on stakeholders’ insights provides study This (namely of MEAs example an practice, amongst amongst on stakeholders amongst include MEAs implemented in other countries (e.g. Italy (6), France (37), Sweden (37), (37), Sweden (37), Italy France (6), (e.g. countries in other implemented MEAs include the context very short the “cost-sharing “cost-sharing A German Medical the the 32-item checklist interviews32-item and focus for Association. Opinion on on Opinion Association. initiatives” and “risk-sharing agreements” agreements” “risk-sharing and initiatives” manufacturers pharmaceutical between 2017 2008.[cited and Hospital. and Health http://www.akdae. from: Available April 14]; de/Stellungnahmen/Weitere/20080508.pdf of adaptive pathways for medicines in in medicines for pathways of adaptive and opportunities. challenges Clin Europe: Ther Pharmacol 2016;100:594-7. and Pricing al. et M, Qin D, Lee SD, Faulkner in and insights experiences reimbursement Zorginstituut Nederland. Beoordeling Beoordeling Nederland. Zorginstituut https:// procedure, geneesmiddelen /pakket/ www.zorginstituutnederland.nl pakketbeheer/werkwijze+ beoordeling+ 2016. procedure. geneesmiddelen/ Adaptive Medicines Agency. European October 2017 2017.[cited 30]; Pathways. http://www.ema.europa. from: Available regulation/ eu/ ema /index.jsp?curl=pages/ general_content_000601.jsp general/ Health al. C, et Longson Bouvy P, JC, Jonsson in assessment technology Beoordeling Nederland. Zorginstituut praktijk. en 2015. de wetenschap stand van Nederland. Zorginstituut Bevir Governance: M. 2012. Oxford, OUP introduction and Complexity: Governance V. Duit A, Galaz Theory. Governance for Issues Emerging 2008;21:311-35. Governance of Drug Commission Palinkas LA, Horwitz SM, Green CA, et al. Horwitz LA, Palinkas CA, al. et Green SM, data qualitative for sampling Purposeful method in mixed collection and analysis Administration research. implementation Mental Mental and in Health Policy and ServicesHealth 2015;42:533-44. Research approaches Three SE. Shannon Hsieh HF, Qual Health analysis. content qualitative to Res 2005;15:1277-88. criteria Consolidated J. Craig A, Sainsbury P, Tong (COREQ): research reporting qualitative for a 2007;19:349-57. Int J Qual Health Care groups.

21. 18. 19. 20. 15. 16. 17. 12. 13. 14. 11.

novel novel A American journal of American journal of critical cost containment containment critical cost The The a Dutch reality check. reality Dutch (UNDER the United States: trends, barriers, barriers, trends, States: United current performance-basedcurrent schemes the systematic review. Journal services health of review. systematic taxonomy and examination of performance- of and examination taxonomy a measure for cost-containment of drug drug of cost-containment for measure 2015;18:131-6. Health Value expenditure. Makady A, van Veelen A, De Boer A, et al. DeA, Boer al. et A, Veelen Makady van A, Implementing Managed Entry Agreement Practice: in Nederlandse Zorgautoriteit (NZa). (NZa). Nederlandse Zorgautoriteit Dure BR/CU-2017 BELEIDSREGEL Geneesmiddelen. 2011. mw.dr.G.Ligtenberg. Nederland, Zorginstituut van toelating/financiering Voorwaardelijke 6-4-2012. zorg. in Italy really work? “Success Fee”: Fee”: “Success in Italy work? really Klappe-Sabadi FGA, Jansman G, verschillen Regionale HA, et al. Honkoop trastuzumab in voorschrijfgedrag Wetenschappelijk PW getoetst. opnieuw 2008. 2008. 2[7]. Platform Navarria A, Drago V, Gozzo L, et al. Do Do L, et al. Gozzo V, A, Drago Navarria the Carlson JJ, Sullivan SD, Garrison LP, et al. al. et Garrison LP, SD, Sullivan JJ, Carlson outcomes: health Linking to payment a between schemes based reimbursement Health manufacturers. and payers healthcare 2010;96:179-90. Policy Bajaj PS, et al. JJ, Carlson Garrison LP, Jr sector agreements risk-sharing Private in and prospects. 2015;21:632-40. care managed Godman B, Shrank W, Wettermark B, et al. et al. B, Wettermark W, Shrank Godman B, of generics: Use Iyn-Hyang L, Karen B, Catherine H, et al. H, Iyn-Hyang Catherine Karen L, B, in controlling experience International influencing expenditure: pharmaceutical industry regulating and and providers patients - & policy 2014;20:52-9. research in professionals all healthcare for measure 2010;3:2470-94. Pharmaceuticals Europe? OECD. OECD Data: Pharmaceutical Pharmaceutical Data: OECD OECD. December 2017 2016.[cited Spending. https://data.oecd.org/ from: Available 28]; healthres/pharmaceutical-spending.htm REVIEW).2018. Policy Health

REFERENCE LIST REFERENCE 9. 10. 8. 7. 6. 5. 4. 3. 2. 1.

Use of RWE in HTA practice 6 130 Use of RWE in HTA practice 6 131 joint ISPOR- joint the COWpath: Learning and visualizing and visualizing Learning COWpath: Comparative Study of Six HTA Agencies. Agencies. of Six HTA Study Comparative Wang SV, Schneeweiss S, Berger ML, et al. al. et ML, Berger S, Schneeweiss SV, Wang and reproducibility Reporting improve to healthcare for validity assessment facilitate 0. Pharmacoepidemiol V1. studies database SafDrug 2017;26:1018-32. Relative et al. TL, Kleijnen S, Lipska I, Alves medicines of oncology assessments effectiveness decisions in pricing and reimbursement for 2016;mdw233. Oncol Ann countries. European N. Paving R, Patel Padman Y, Zhang the electronic health from pathways clinical Journal of biomedical data. record 2015;58:186-97. informatics Creating et al. YF, Li JS, Zhang HQ, Wang semantic by clinical pathways personalised with electronic health interoperability Artif Med Intell 2013;58:81-9. records. MakadyBoer Ham R, de al. A, et A, ten in Data Real-World of Use for Policies (HTA): Assessment Technology Health A 2017. Health Value Garbe E, Suissa S. Pharmacoepidemiology. Garbe Suissa S. Pharmacoepidemiology. E, of Handbook I, eds., Pigeot W, In: Ahrens Springer New NY: York, New Epidemiology. 1875-925. p. 2014. York, et T, Debray Mavridis D, Efthimiou O, non- and randomized Combining al. network in meta- evidence randomized Med2017;36:1210-26. Stat analysis. Goodal. et RJ, Willke H, Sox ML, Berger ofstudies data real-world practicesfor effectiveness: and/or comparative treatment from Recommendations on real-world Force Task ISPE Special making. decision in health care evidence Drug Saf 2017;26:1033-9. Pharmacoepidemiol treatment in observational studies are in observational are studies treatment 2004;57:1223-31. J Clin Epidemiol reviewed.

37. 34. 35. 36. 33. 32. 30. 31. United States: United PHAROS registry. PHAROSregistry. the the practical guide for using using practical guide for A European Union and Union European cost effectiveness of new cancer drugs: drugs: cancer new of effectiveness cost meeting held March 30th, 2016. 2016. 2016. 2016. 30th, held March meeting test. BMJ 2012;344:e55. BMJ test. case of oxaliplatin for treatment of stage stage of treatment for case of oxaliplatin Lessons learned to approach adaptive payer adaptive approach learned to Lessons 2016;100:730-42. Ther Pharmacol Clin pathways. Klungel OH, Martens EP, Psaty BM, et al. BM, et al. Psaty Klungel OH, Martens EP, of drug effects intended assess Methods to Alemayehu D, Riaz Ali MPP, Alvir JM, et al.et JM, Alvir Riaz MPP, Ali D, Alemayehu Issues and Analytical of Data, Examination Conducting Comparative Methods for Proposed Data”. “Real-World Using Research Effectiveness 2011;17:S3-S37. Pharm J Manag Care Mohseninejad L, van Gils C, Uyl-de Groot Uyl-deC, Gils Mohseninejadvan L, Groot registries of patient Evaluation CA, et al. supporting decisions: reimbursement the 2015;18:84-90. Health Value cancer. III colon CTTI. Registry Trials Project Multi- CTTI. Project Trials Registry ExpertStakeholder Meeting: Summary of the MG, Uyl-de HM, Franken Blommestein CA. Groot about decisions inform registry to data the learnedlessons from van Staa TP, Goldacre B, Gulliford M, et al. M, et al. Gulliford B, Goldacre TP, Staa van trials using routine randomised Pragmatic putting them to electronic records: health the Carlson JJ, Gries KS, Yeung K, et al. Current K, Current al. et Yeung Gries KS, JJ, Carlson in performance-based and trends status between arrangements risk-sharing medical product and payers healthcare health economics Applied manufacturers. policyand health 2014;12:231-8. Makady A, et Ciaglia H, Stegenga A, of using real- implications Practical al. (RWE) evidence world in comparative IMI- learnings from research: effectiveness Medicine. Future 2017. GetReal. the Pharmacoecon 2015;33:551-60. Pharmacoecon

26. 25. 24. 23. 22. 29. 28. 27.

SECTION Access to Real-World Evidence IV

CHAPTER

Practical Implications of Using Real-World Evidence in Comparative Effectiveness 7 Research: Learnings from IMI-GetReal

A. Makady1,2, H. Stegenga3, A. Ciaglia4, T.P.A. Debray5,6, M. Lees7, M. Happich8, B. Ryll9,10, K. Abrams11, R. Thwaites12, S. Garner3, P. Jonsson3, W. Goettsch1,2 (on behalf of GetReal Work Packages 1 & 4)

1 The National Healthcare Institute (ZIN), Diemen, the Netherlands; 2 Department of Pharmacoepidemiology & Clinical Pharmacotherapy, Utrecht University, Utrecht, the Netherlands; 3 The National Institute for Health and Care Excellence (NICE), London, the United Kingdom; 4 International Alliance of Patients’ Organizations, London, the United Kingdom 5 Julius Center for Health Sciences and Primary Care. University Medical Centre Utrecht, Utrecht, the Netherlands; 6 Cochrane Netherlands, University Medical Center Utrecht, Utrecht, Netherlands 7 Bristol-Myers Squibb, Paris, France; 8 Eli Lilly, Hamburg, Germany; 9 Melanoma Patient Network Europe, Uppsala, Sweden; 10 Uppsala University, Uppsala, Sweden; 11 Department of Health Science, University of Leicester, Leicester, the United Kingdom 12 Takeda, London, the United Kingdom

Published in the Journal of Comparative Effectiveness Research. Citation is as follows: Makady A, Stegenga H, Ciaglia A, Debray TP, Lees M, Happich M, Ryll B, Abrams K, Thwaites R, Garner S, Jonsson P. Practical implications of using real-world evidence (RWE) in comparative effectiveness research: learnings from IMI-GetReal. Journal of Comparative Effectiveness Research. 2017 Aug 6(6). conduct conduct context of of context the the experiences gained gained experiences the equivalent term for CER as used CER as used for term equivalent purposes of this commentary, we we purposes of this commentary, The The the use of Real-World Evidence (RWE) Evidence in decision use of Real-World the real-world setting) (4). setting) real-world the European context of Health Technology Assessment (HTA) and decision making decision and is (HTA) Assessment Technology of Health context European Innovative Medicines Initiative GetReal Medicines Initiative ConsortiumInnovative (IMI-GetReal) and discuss their (2)

the

ABSTRACT Relative Effectiveness Assessment (REA). Assessment Effectiveness Relative in and/or synthesis of research comparing different benefits and harms of alternative harms and benefits alternative of different comparing research of synthesis and/or in conditions health and monitor treat, diagnose, prevent, to and strategies interventions clinical practice (i.e. routine RCTs (3). Meanwhile, we define Comparative Effectiveness Research (CER) as (CER) Research Effectiveness Comparative define we (3). Meanwhile, RCTs define RWE as evidence generated based on health data collected outside data based on health define generated RWE as evidence implications for RWE use in decision-making. For RWE for For use in decision-making. implications in accessing and using RWE for Comparative Effectiveness Research (CER) as partas (CER) of Research Effectiveness RWE Comparative using and for accessing in the making in recent years (1), this commentary reflect on years to aims making recent in In light of increasing attention towards towards attention In of increasing light

Access to RWE 7 136 Access to RWE 7 137 presence presence presence presence established established the the the insights provided provided insights real-world setting setting real-world detection rare- of or The The the lack of randomisation of of lack of randomisation the regional or national level. level. or national regional number of CER scenarios. of CER scenarios. number a a the efficacy of health and safety observed efficacy interventions of phenomenon often referred to as as to oftenphenomenon referred the a the protocol-driven patient-follow up, patient populations populations patient up, patient-follow protocol-driven use of predictive modelling techniques. RWE may also RWE also may techniques. modelling predictive use of a past 20 years, including formal checklists for assessing checklists assessing including formal for past 20 years, implementation of such methods can be used to increase increase can be used to of such methods implementation the discrepancy between a highly-controlled conditions characteristic of RCTs may not not may characteristic of RCTs conditions highly-controlled lack of comparability between treatment groups, groups, treatment between lack of comparability the context of RCTs (3), may help identify, quantify and address and address quantify identify, help may (3), of RCTs context a the the statistical analysis, synthesis and critical appraisal of RWE have of RWEcritical and appraisal synthesis have analysis, statistical the the number of importantnumber shortcomings RWE with particularly are that problematic a robustness of estimates derived from analyses using RWE analyses in from derived of estimates robustness efficacy-effectiveness (6;7). gap Summary data (also known as aggregate data), whether from RCTs or RWE, such as RWE,or as such RCTs from whether data), Summaryknown (also data aggregate as Real-World Evidence (RWE), broadly defined as evidence generated based on health health based on generated (RWE), as evidence defined Evidence broadly Real-World WHY IS INDIVIDUAL PATIENT-LEVEL DATA (IPD) IMPORTANT IMPORTANT (IPD) DATA PATIENT-LEVEL INDIVIDUAL IS WHY CER? IN RWE USING WHEN WHY IS REAL-WORLD EVIDENCE (RWE) RELEVANT FOR FOR RELEVANT (RWE) EVIDENCE REAL-WORLD IS WHY (CER)? EFFECTIVENESS RESEARCH COMPARATIVE the patients which can result in in which can result patients of Therefore, confounders. developed considerably in considerably developed multivariable analyses, variable instrumental techniques, scoring propensity risk of bias, methods can begin These (8-11). methods meta-analysis and advanced analyses, regression address to (CER), such as research effectiveness comparative in its use for of missing observations on relevant patient outcomes or covariates, and or covariates, outcomes patient observationsof missing on relevant estimates of comparative treatment effect are often of limited value for CER. For instance, instance, For CER. for value often of limited are effect treatment of comparative estimates Methodologies for for Methodologies method for providing information pertaining information to providing for method always accurately represent clinical practice (5). Whereas RCT patient populations are highly RCT highly are Whereas populations patient clinical practice (5). represent accurately always with selected homogenous, and interventions. However, However, interventions. co-morbidities. with and oftenpresent typically heterogeneous practice are seen in clinical preventing durations, short often have follow-up RCTs Moreover, Traditionally, Randomised Controlled Clinical Trials (RCTs) are considered considered are (RCTs) Trials Clinical Randomised Controlled Traditionally, through evidence synthesis or synthesis evidence through provide information on parameters not examined in clinical trials, such as adherence to to as adherence such in clinical trials, not examined on parameters information provide settings. clinical use in resource and events, adverse rare treatment, this efficacy-effectiveness gap in treatment effects where needed. For example, RWE may example,RWE For effects needed. where efficacy-effectivenessthis treatment in gap in effects of treatment estimates RCTsupplement improve to data data collected outside collected data in RCTs and their effectiveness in clinical practice, practice, in clinical and their effectiveness in RCTs the long-term adverse events of interventions. Surrogate endpoints measured in RCTs, such as such as in RCTs, measured endpoints Surrogate of interventions. events adverse long-term decision-making to relevant less be also may survival patients, oncology in progression-free vary practice clinical may on survival. Furthermore, than overall and particularly, regulators for drug developers, implications significant RWE have from may on evidence rely whose decisions and payers agencies (HTA) Assessment Technology Health (1). effectiveness of comparative These differences can lead to can lead differences These other other series of seriesof registries registries the a researchers researchers the the lifecycle of drug of drug lifecycle multi-stakeholder multi-stakeholder public and industry and public a the a 3-year project exploring project exploring 3-year a limited ability to explore individual patient patient individual ability explore limited to the acceptability of these methods amongst relevant relevant amongst these methods of acceptability combined WP1/4 case study whereby co-leads co-leads whereby case study WP1/4 combined a the public-private partnershippublic-private with methods explored included extrapolation of long-term long-term of extrapolation included explored methods a the project comprised 5 Work Packages (WP1-5), Packages Work each addressing 5 project comprised case study teams, each jointly co-lead jointly each by case study teams, the the case study team of upcoming data updates (9;13). In 4 instances across In (9;13). across 4 instances updates data of upcoming team case study project was project was the The The positive example relates to to relates example positive A Innovative Medicines Initiative (IMI)-GetReal Medicines Initiative Innovative project was use of RWE to improve drug effectiveness research throughout throughout research drug effectiveness RWEuse of improve to

In total, 7 case studies were conducted as part of WP1 and WP4 work, spanning multiple WP4 work, spanning multiple WP1 and as part conducted 7 case studies were of In total, RWE case per varied IPD from repositories with accessing encountered Experiences major drawback of aggregate data (AD) is is (AD) data of aggregate drawback major series of case studies. Some of series of case studies. WHAT WERE IMI-GETREAL’S EXPERIENCES IN ACCESSING IPD IPD ACCESSING IN EXPERIENCES IMI-GETREAL’S WERE WHAT STUDIES? CASE THROUGHOUT secured access to IPD from registries in 2 different countries. Moreover, Moreover, countries. in 2 different registries IPD from to access secured disease indications and each lasting approximately 1.5 years. RWE repositories approached RWE approached repositories 1.5 years. approximately and each lasting indications disease co-leads case study and 8 observational Eventually, registries studies. 12 indication included observational and 3/8 registries 4/12 studies, IPD from to access secure to managed On of cases. in only 35% succeeded RWE from sources access IPD that indicating study. partner, sought to access individual patient-level data (IPD) from both RWE repositories and RWE both and from repositories (IPD) data patient-level individual access to sought partner, conductthese analyses. to RCTs IPD that indicating in 41/43 studies, granted and 43 RCTs from requested IPD was hand, 95%. exceeded RCTs from retrieval actively informed actively informed different case studies, registry holders and observational case studies, indicated study authors initially different different objectives (2). Work conducted for 2 work packagesfor 2 conducted Work attempts involved (WP1 & 4) objectives (2). different (WP1)1 Package Work conducted RWE from IPD use and RCTs. and/or access to case studies in multiple disease areas, aiming to explore methods for using RWE to improve RWE using improve methods for to explore to aiming areas, in multiple disease case studies examine to and estimates effectiveness 4 (WP4) Package Work explored Meanwhile, workshops. stakeholder through stakeholders and reviews literature through RWE from RCTs and/or synthesis evidence practicesfor best a RWE with (NMAs) of Network enrichment Meta-Analyses trial durations, beyond outcomes scoring propensity through populations RCT of generalization and real-world to results Together, techniques. constituency including industry, regulators, HTA agencies, academia and patient patient and academia agencies, HTA regulators, constituencyindustry, including In total, organisations. a characteristics which may influence or confound treatment outcomes. Therefore, in order to to order in Therefore, outcomes. treatment confound or influence characteristics may which often require agencies and payers decision-making, HTA can inform CER that conduct robust individual adjust for can researchers whereby be conducted to analyses sophisticated more (1;12). Several of effectiveness estimates accurate more generate characteristics to patient further be discussed do so and will to teams below. research by be employed may strategies by whether IPD, based on analyses require such strategies Importantly, party. alternative or an themselves, The The the development.

Access to RWE 7 138 Access to RWE 7 139 same same The The aggregate aggregate robustness robustness absence of of absence considerable considerable the the the the data (14). data case studies, IPD was was IPD case studies, the the other hand, other hand, WP1 case study, whereby one one whereby study, WP1 case fee. However, However, fee. a the a tailored portiontailored on of IPD based a direct barrier IPD retrieval. to registry completed their dissertations, in dissertations, their in registrycompleted a original patient population). However, this this However, population). patient original the the requested analyses. On analyses. requested negative example relates to to relates example negative third of all requests for IPD access from RWE repositories RWE from repositories access IPD for requests all of third A distribution of covariates within patient populations thus thus populations patient within distribution of covariates the only one (9;13). For example, in order to make use of IPD of IPD make use to in order example, only one (9;13). For a remaining case studies, access to IPD was denied. Reasons for Reasons for denied. IPD was to access case studies, remaining the the the prominent issue encountered in using RWE in using CER in IMI-GetReal for issue encountered prominent team (14;15), or attempted to use AD as reported in literature (9;14;15). (9;14;15). reported as use AD literature in to attempted or (14;15), team a the AD retrieved from both approaches was subsequently used in several ways to perform to ways used in several subsequently was approaches both from AD retrieved research proposal submitted could also be bought for for be bought also could submitted proposal research In summary, IMI-GetReal’s experiences in accessing IPD from RWE repositories were RWE were from IPD repositories accessing in experiences IMI-GetReal’s Insummary, As an alternative to accessing IPD, case study teams explored options for using using for options explored case study teams IPD, accessing to an alternative As Another point worth noting is that although accessibility of IPD from RWE worth IPD from of point Another accessibility although noting is that disparate. In general, only In general, disparate. amount of this fee (surpassing acted €100,000) as fee of this amount results back to to back results limited literature from AD retrieved within covariates on patient information aggregate data (AD) from registries and observational studies. To do so, case study teams teams study case so, do To observationaland registries studies. from (AD) data aggregate pre-scripted run report and on IPD registries analyses that requested either the project within timelines research-ready not being datasets to mostly related inaccessibility general important raise reasons regarding questions These data. share to or unwillingness research, for available readily qualitybe to sufficient of sets data in generating competence respectively. ownership, as data as well submitted across all case studies were successful. For half of half For successful. all case studies were across submitted accessed from registries and observational studies. Furthermore, co-leads iterated that that co-leads iterated and observational Furthermore, registries studies. from accessed case those for problems structures considerable and pose not did sharing agreements data for However, studies. CER, for example by simulating patient-level data or as direct input for effect estimates in in estimates effect direct or as for input data patient-level simulating by example for CER, Importantly, easier. relatively was approaches both AD through to Access models. NMA conducting for insights relevant pre-scripted on IPD provided by analyses AD generated illustrating by CER (e.g. cooperation on heavily expertise relies and considerable implement to requires approach registry run to holders from AD can be although Therefore, such data. from generated estimates of health outcomes to itself lending mostly usefulness, it is often of limited literature, from easily obtained different across effects treatment of analysis to than rather analyses statistical descriptive (9;14). and populations settings The The allowing for more accurate simulations of simulations accurate more for allowing registry indicated earlier in negotiations that access to to access that earlier in negotiations registry indicated repositories was was repositories their willingness to provide access to IPD. However, they eventually communicated that that communicated eventually they However, IPD. to access provide to willingness their for need an extensive within project timelines due to research-ready not were their datasets & trimmingcleaning (9;13;14). registry refused to discuss possibilities for collaboration upfront, due to being approached approached being due to upfront, collaboration for possibilities discuss to registry refused with 16 months lasting an industry negotiations by prolonged co-lead Meanwhile, (14). be would IPD to access that iterated representatives when abandoned registryanother were with associated students PhD all until refused otherwise would they that on based of findings lose ownership fear case studies, it was not it was case studies, lack lack a potential potential reporting of consortium’s consortium’s the the the latter aspectlatter implies the dataset or imputing missing missing imputing or dataset necessary registry disclaimers to the the other hand, other hand, the outcome measures across different studies studies different across measures outcome case studies. More importantly, importantly, More case studies. the wide lack of trust among decision-makers regarding regarding decision-makers wide lack of trust among the IMI-GetReal consortium. Arguably, this inaccessibility inaccessibility this IMI-GetReal consortium. Arguably, robustness of findings based on RWE compared with with on based findings of compared robustness RWE lack of concrete examples demonstrating demonstrating examples lack of concrete the the the the former aspect through guideline development on topics on topics aspect development guideline former through issue of accessibility to IPD from RWE repositories and its its RWEand from IPD repositories to accessibility of issue the acceptability of RWE use in CER and subsequent decision-making. of RWEacceptability decision-making. and subsequent use in CER synthesis of IPD from RWE and RCT sources. These issues raise raise issues These RWE from of IPD RCT and synthesis sources. the lack of experience with using RWE in currently available methods methods RWE using with available experience of lack currently in the a the lack of trust in in trust of lack a multi-stakeholder nature of IMI-GetReal case study teams, adherence to to of IMI-GetReal adherence nature multi-stakeholder case study teams, case study teams often had to invest considerable time and effort in effort and time in considerable invest often had to teams case study the the persistent issue beyond beyond issue persistent a reasons behind such controversy are multi-factorial, yet generally hinged on two inter- on two hinged generally yet multi-factorial, are controversy such behind reasons robustness of findings based on based of findings robustness RWE.

Despite Despite

WHAT ARE THE CONSEQUENCES OF INACCESSIBILITY TO TO INACCESSIBILITY OF CONSEQUENCES THE ARE WHAT FOR USE POTENTIAL ITS ON REPOSITORIES RWE FROM IPD HEALTHCARE? IN DECISION-MAKING added value of RWE of added value in CER and use the to IPD RWE both contributes to IPD RWE to to contributes both of published examples exploring advanced methods for RWE use in CER and subsequent RWE methods for subsequent and CER in use advanced exploring examples of published decision-makers. relevant methods from on these feedback as as well access, data for procedures application experience with inaccessibility of IPD RWE for research purposes was echoed by many many echoed by purposes was IPD RWE of research with inaccessibility experience for RWE IPD to access that implying workshops, in stakeholder external present stakeholders remains data values). Occasionally, observational studies only investigated treatment patterns, patterns, treatment observational only investigated studies Occasionally, values). data making involving analyses use to of little them outcomes, treatment than rather outcomes treatment where Moreover, head-to-head of effectiveness. comparisons varying of definitions recorded, were often complicated often complicated accessed, accessed, RWE using CER and decision-making. for for implications making datasets research-ready (e.g. by trimming by (e.g. makingresearch-ready datasets additional methodological and practical concerns in applying RWE to CER, some of RWE in applying practical and some of CER, concerns methodological additional to by all considered should be and literature scientific in addressed been have which in subsequent undertake to attempting stakeholders efforts similar However, (10;16). on sectionsfocus we IMI-GetReal case study workshops demonstrated considerable variability in external considerable IMI-GetReal demonstrated workshops case study on views stakeholders’ including: good practices to ensure data quality and standardised core outcomes datasets datasets outcomes core quality data and standardised ensure good practices to including: of RWE analysis statistical (17;18), CER and (19-21) inform to registries within The The aspects: related RCTas well as data, ongoing Numerous effectiveness. drug (comparative) to relating questions address to address aim to initiatives observational from On (22;23). studies results owners and study authors approached, accessibility to IPD from RWE repositories proved to to RWE proved from IPD repositories to accessibility and study authors approached, owners novel explore to thoroughly available were data insufficient Consequently, be challenging. RWE for methods of almost half use in

Access to RWE 7 140 Access to RWE 7 141 sentinel sentinel

review of of review the spirit of joint spirit of joint the a dynamic, multi- dynamic, use of this data; use of this data; the the role RWE can play in RWE in role play can the de-centralised network of a general understanding amongst amongst understanding general a collaborative approach to solving solving to approach collaborative a ideal implementation of healthcare healthcare of ideal implementation wider community about governance wider community about governance data is guaranteed through transparent transparent through is guaranteed data the the model is its ability to circumvent sensitivities sensitivities model is its ability circumvent to the a whole. Moreover, as RWE is generated by patients patients by RWEas generated is Moreover, whole. a future. However, bearing in mind that patient-level patient-level bearing that in mind However, future. FDA-Sentinel initiative, whereby external researchers external researchers whereby initiative, FDA-Sentinel the the healthcare sector warrants sector warrants healthcare patient-centred goals behind healthcare in general, as well as RWE as well general, in healthcare behind goals patient-centred the main advantage of such main advantage the analysis protocols. protocols. analysis The The the EU Clinical Trial Directive was recently established, whereby sponsors of RCTs of RCTs sponsors whereby established, recently was Directive Trial EU Clinical the scientific rigor of analyses exploiting exploiting of analyses rigor scientific worthwhile endeavour for for endeavour worthwhile the publication of similar IPD for RWE generated through observational studies and may observational through RWE may studies and for of similar IPD generated publication a An important aspect to enable collaborative efforts importantAn is aspect enable collaborative to to leading contracts, set up based on undisclosed currently are registries Furthermore, Developments on other fronts may provide additional potential solutions. For For solutions. potential additional provide may fronts on other Developments Another example relates to to relates example Another strong commitment to involve all key actors in setting-up and developing procedures procedures all key actors developing and in setting-up involve to commitment strong THE FUTURE? THE WHAT ARE POTENTIAL SOLUTIONS TO ADDRESSING ISSUES ISSUES ADDRESSING TO SOLUTIONS POTENTIAL ARE WHAT IN REPOSITORIES RWE FROM IPD TO ACCESS WITH FACED all stakeholders of all stakeholders In regard, this healthcare. improve to collection analysis and situations where it is difficult to deduce why accessibility is difficult and which stakeholders stakeholders accessibility which and is difficult why deduce to difficult it is where situations making transparent contracts such Therefore, requests. on data in deciding involved are clarity importantis another increase in to step issues pertaining to governance of RWE repositories, including accessibility to IPD. IPD. to accessibility including of RWEpertainingissues repositories, governance to make stakeholders if healthcare understood best be will goals patient-centred pursuing a relevant all practice, contrary current to that, requires This registries. to enable access to whereby of these registries, participatestakeholders committees steering in success. action for is crucial and is not misused data such that guarantee to committees relevant by protocols research that be example, example, patient- all to access provide to agreed marketing applications authorisation for conducted for exists initiative no equivalent reports clinical level of trial subjects Presently, online (24). the issues such as data ownership, gate keepers for data access, funding sources and conflicts and conflicts sources funding access, data for keepers gate ownership, issues such as data of interests. is subjectdata strict not preclude should to privacy endeavours such rules, stakeholder nature of of nature stakeholder participating or out of can opt-in databases (25). In databases model, this run external still yet their IPD, to access complete relinquish to having without initiative queries. research publication of publication can send standardised data queries to multiple nodes of of nodes multiple queries to data standardised can send interventions to achieve maximum societal benefits, inaccessibility to IPD from to IPD from RWE inaccessibility benefits, maximum societal achieve to interventions societyaffects as adversely repositories Bearing in mind that CER aims to shed light on light shed Bearing to CER aims that in mind within routine healthcare, it is essential that patients benefit from benefit from patients that essential it is healthcare, routine within decision-making and CER all should benefit IPD RWEto improve accessibility to increased Consequently, data. such to access control who those just not patients, potential potential the considerable barrier to barrier to considerable a choice regarding which path path which regarding choice the effectiveness of treatments for all for all treatments of effectiveness required information for furthering for information required choice, as well as its implementation, as its implementation, as well choice, the the a IMI-Big Data for Better Outcomes (IMI- Outcomes Better for Data IMI-Big the consortium poses the Environmental Core Project (BioSHaRE-EU) (26). Other (26). (BioSHaRE-EU) Project Core Environmental the current state of accessibility to RWE to during of accessibility IMI-GetReal experienced state current the collaborative effort spanning all relevant stakeholders; from decision-makers, to to decision-makers, from stakeholders; relevant effort all spanning collaborative a long way. long a Healthy Obese Project and and Obese Project Healthy

In conclusion,

industry representatives. and patient scientific pursuits. This approach towards remote data querying has demonstrated potential querying data potential remote demonstrated has towards approach This pursuits. scientific implemented been have frameworks similar in IMI-GetReal Moreover, (14;15). studies case part as research conduct of to international as DataSHIELD such research, of in other fields the include frameworks such exploring initiatives relating to full-fledged access to IPD while delivering while delivering IPD to access full-fledged to relating to take, addressing trade-offs addressing with such take, to associated patients in real practice; insights that RCTs are often not designed to provide. An array of of array An provide. to often designed not are RCTs that insights practice; real in patients RWE to inaccessibility persistent this overcoming to solutions lend themselves potential However, CER. in use its from gain societal maximising and requires BD4BO) (27). Arguably, equivalent systems for existing registries would bring RWE would use in registries existing for systems equivalent Arguably, (27). BD4BO) CER furthering RWEsuch Bearing decision-making. that mind in healthcare and CER in use barrier this diminishes practice, in clinical patients by is generated data on critical insights of using benefit to provide RWE case studies and stakeholders beyond beyond and stakeholders case studies

Access to RWE 7 142 Access to RWE 7 143 tool for assessing risk of bias in risk assessing bias in of for tool a EUnetHTA. Work Package 5 - Life cycle cycle 5 - Life Package Work EUnetHTA. Generation. Evidence improve to approach http://eunethta.eu/ from: 2017.Available activities/eunethta-joint-action-3-2016-20/ work-package-5-life-cycle-approach- improve-evidence-gener Ericson A, Gsteiger S. Deliverable D1.6 D1.6 S. Deliverable Ericson Gsteiger A, Review: Study Case WP1 Rheumatoid 24]; Arthritis. 2017 March 2016.[cited https://www.imi-getreal.eu/ from: Available Portals/1/Documents/01%20deliverables/ Deliverable%201.6%20Report%20-%20 Rheumatoid%20Arthritis_webversion.pdf M. Lees MakadyW., A, Kalf R, Goettsch Review: Study Case WP1 D1.6 Deliverable 2017 Metastatic Melanoma. 2016.[cited https://www.imi- from: Available 24]; March getreal.eu/Portals/1/Documents/01%20 WP1%20deliverables/Deliverable%20 1.6%20Report%20-%20Rheumatoid%20 Arthritis_webversion.pdf of Leicester. Novartis, University NICE, (Multiple 1.6 Sclerosis Deliverable from: 2016.Available Study). Case https://www.imi-getreal.eu/Portals/1/ Documents/01%20deliverables/ Deliverable%201.6%20Report%20-%20 multiple%20sclerosis%20case%20study_ webversion.pdf al.et JM, Alvir Riaz MPP, Ali D, Alemayehu Issues and Analytical of Data, Examination Conducting Comparative Methods for Proposed Data”. “Real-World Using Research Effectiveness 2011;17:S3-S37. Pharm J Manag Care Sterne JA, Hernán MA, Reeves BC, et al. et al. BC, Reeves JA, Hernán MA, Sterne ROBINS-I: of interventions. studies non-randomised 2016;355:i4919. BMJ An et al. Schuit Efthimiou E, O, TP, Debray network meta- overview for methods of participantindividual using analysis data: Stat arise? do benefits Methods when Med Res 2016;0962280216660741. randomized evidence in network in meta- evidence randomized Med2017;36:1210-26. Stat analysis.

16. 17. 14. 15. 12. 13. 11. Instituteof the efficacy-effectiveness gap: ISPOR Real World Data Task Task Data World ISPOR Real report from the The The a Efficacy-Effectiveness Gap: Historical regulator’s perspective on addressing perspective on addressing regulator’s Efthimiou O, Mavridis D, Debray T, et et T, Debray Mavridis D, Efthimiou O, non- and randomized Combining al. Garbe E, Suissa S. Pharmacoepidemiology. Garbe Suissa S. Pharmacoepidemiology. E, Handbook of I, eds., Pigeot W, In: Ahrens Springer New NY: York, New Epidemiology. 1875-925. p. 2014. York, et al. Didden EM, TPA, Debray Hummel N, recommendations guidance, Methodological (network) for case studies and illustrative predict to modelling and meta-analysis using individual effectiveness real-world 2017.[cited data. participant and/or aggregate https://www. from: Available 20]; May 2017 imi-getreal.eu/Portals/1/Documents/01%20 deliverables/2017-03-30%20-%20WP4%20 -%20Methodological%20guidance%2C%20 recommendations%20and%20 illustrative%20case%20studies.pdf Nordon C, Karcher H, Groenwold RH, et al. al. RH, et Karcher C, Groenwold H, Nordon The Conceptualization. and Current Background 2016;19:75-81. Health Value Freemantle N, Strack T. Real-world Real-world T. N, Strack Freemantle be should medicines of new effectiveness clinical designed appropriately by evaluated 2010;63:1053-8. Epidemiol J Clin trials. et A, Breckenridge E, HG, Abadie Eichler Bridging al. Sox HC, Greenfield S. Comparative effectiveness Comparative S. Greenfield HC, Sox research: Intern Med Ann 2009;151:203-5. Medicine. a Rev Drug Nat variabilityresponse. drug of 2011;10:495-506. Discov IMI-GetReal. IMI-GetReal: objectives. Overall Available 13]; December 2016 2017.[cited https://www.imi-getreal.eu/About- from: GetReal/Overall-objectives Using et al. Erickson P, PJ, Neumann Garrison LP, and Payment Coverage for Data World Real Decisions: 2007;10:326-35. Health Value Report. Force Makady A, Goettsch W. Review of Policies Review Policies of W. MakadyGoettschA, 2015. Data. on Real-World and Perspectives April 6]; 2017 [cited

REFERENCE LIST REFERENCE 10. 9. 8. 7. 6. 5. 3. 2. 1. 4. data to data the Reporting of data, not data, the the analysis to to analysis the Strengthening Strengthening analysis. Int J Epidemiol 2014;43:1929-44. analysis. policy framework for public health uses of of uses health public policy for framework McGraw D, Rosati K, Evans B. K, Rosati B. Evans D, McGraw A Pharmacoepidemiol electronic data. health SafDrug 2012;21:18-22. DataSHIELD: al. et J, Isaeva Y, A, Marcon Gaye taking the Big data, Medicines Initiative. Innovative disease respiratory and safety medicines proposals. for €93 million IMI Call of focus from: 28]; Available May 2017 2015.[cited http://www.imi.europa.eu/content/imi-2- call-6-launch Von Elm E, Altman DG, Egger M, et al. al. M, et Egger DG, Altman Elm E, Von The Observational Epidemiology in Studies reporting for guidelines (STROBE) Statement: observational Journal International studies. of Surgery 2014;12:1495-9. trials Clinical - Commission. European 2017.[cited EU No 536/2014. Regulation https:// from: Available 24]; March 2017 ec.europa.eu/health/human-use/clinicaltrials/regulation_en#ct4 studies/ENCePP%20Code%20of%20 Conduct_20100507.pdf

27. 24. 25. 26. 23. International SocietyInternational for Design of Nonrandomized of Nonrandomized Design the Cross-Border Patient Registries Patient Cross-Border The The ENCePP Code of Conduct for of Conduct for Code ENCePP The The The The Conduct of Pharmacoepidemiological Conduct of Pharmacoepidemiological the Initiative. 2016.[cited 2017 March 24]; Available 24]; Available March 2017 2016.[cited Initiative. http://patientregistries.eu/ from: Pharmacoeconomics and Outcomes Research and Outcomes Pharmacoeconomics Retrospective for Practices Good Research ReportGÇöPart Force Task Analysis Database 2009;12:1053-61. Health Value II. OHDSI. Observation Health Data Sciences ObservationOHDSI. Sciences Health Data from: 2017.Available and Informatics. https://www.ohdsi.org/ ENCePP. Transparency Scientific and Independence in 2010. Studies. Pharmacovigilance and http:// from: July 3]; Available 2015 [cited www.encepp.eu/documents/encepp_ Hall GC, Sauer B, Bourke A, et al. Guidelines Guidelines Bourke al. et A, Hall GC, Sauer B, selection database good and use for research. in pharmacoepidemiology Saf Drug 2012;21:1-10. Pharmacoepidemiol Cox E, Martin BC, Van Staa T, et al. Good Research al. et T, Staa Van MartinE, BC, Cox Effectiveness Comparative for Practices Mitigate and Bias to Approaches Research: in Confounding Using Secondary Effects Treatment of Studies Sources: Data PARENT JA. JA. PARENT

22. 21. 20. 19. 18.

Access to RWE 7 144

SECTION

Novel Sources for Generating Real-World Evidence V

CHAPTER

Use of Social Media in the Assessment of Relative Effectiveness: An explorative 8 review with examples from oncology. Hopeful or hopeless?

R. Kalf1, A. Makady1,2, R. ten Ham2, K. Meijboom3, W. Goettsch1,2 (on behalf of GetReal Work Package 1)

1 The National Health Care Institute, Eekholt 4, 1112 XH Diemen, the Netherlands. 2 Department of Pharmacoepidemiology and Clinical Pharmacology, Utrecht University, Utrecht, the Netherlands. 3 VU University Amsterdam, Department of Health Sciences, Amsterdam, the Netherlands

Accepted for publication in the Journal of Medical Internet Research clinical effectiveness effectiveness clinical the source of RWD. RWD. of source a articles, such as efficient collection of patient collectionarticles,patient efficient of as such the degree of usefulness for REA should be furthershould be REA for of usefulness explored. degree the potential source of RWD REA, particularly for source potential on aspects such a extent to which social media-generated health data has provided insights insights has provided health data social media-generated which to extent occurrence of disease-specific symptoms. Several strengths of social media- strengths of disease-specific Several occurrence symptoms. the the real world is available directly after market access, therefore randomized controlled controlled randomized therefore directly after market access, is available world real

descriptive qualitative analysis was performed. performed. was analysis qualitative descriptive

ABSTRACT experiences and recruiting patients with rare diseases. Conversely, limitations included included limitations Conversely, diseases. with rare patients and recruiting experiences and selection bias. of information of authenticity presence and validation generated health data were highlighted in in highlighted were data health generated Social media may provide provide Social may media Conclusions Of 1032 articles identified, 8 were included: 4 articles identified adverse events in response response in events articles 4 Of included: were articles 1032 adverse identified 8 identified, and 1 study surveys, 3 articles (QoL) quality life of treatment, disseminated cancer to assessed has potential This behaviour. QoL, and adherence occurrence, symptom events, as adverse and realised been fully not yet Results Methods identify to guidelines PRISMA following conducted was review literature explorative An social media. Scientific using collected were data health where oncology in examples January between published by identified was 2016 June and 2010 literature and grey eligibility extracted and data. for studies screened who independently reviewers, four A We assessed assessed We Objective for REA. REA. for Background assessing constitutes assessment health technology of element An of drugs, generally called relative effectiveness assessment (REA). Little evidence from from evidence Little (REA). assessment effectiveness relative called generally of drugs, the in using interest is growing there However, REA. for information obtain used to trials are provide Social REA. may media (RWD) data for real-world

Novel sources for generating RWE 8 150 Novel sources for generating RWE 8 151 properties considerable considerable onslaught of of onslaught a the effects of health of health effects the the British Pharmaceutical IndustryBritish Pharmaceutical chronic condition are more likely likely more are condition chronic monitoring and management of of management and monitoring tightly-controlled conditions and and conditions tightly-controlled a the the the systematic evaluation of evaluation systematic source to search for information on their on their information for search to source a latest health care developments and increase and increase developments care health latest the value of health technology assessment (HTA) (HTA) assessment technology health of value context of conventional randomized controlled controlled randomized of conventional context the study showed that blogs could be used to collect collect used to be blogs could that study showed Association of Association the the a the evidence available in published literature (12). More recently, recently, More (12). literature in published available evidence the extent to which an intervention – provided under routine clinical clinical routine under – provided an intervention which to extent the supplementary source of evidence to assess relative effectiveness. Real- effectiveness. assess relative to supplementary of evidence source health technology’ (1). Health technologies are defined as ‘interventions ‘interventions as defined are technologies Health (1). technology’ health new drug is extrapolated using health outcomes (e.g. mortality) (e.g. obtained health outcomes using is extrapolated a new drug relative effectiveness assessment (REA) conducted directly aftermarket (REA) assessment effectiveness relative a a a new drugs examined in RCTs, while these patient populations are generally generally are populations these patient while in RCTs, examined new drugs context of rising health care costs, limited budgets, and budgets, limited costs, of rising care health context internet for such purposes than are healthy social media users (5). By assessing By (5). users social media assessing healthy purposes such are than for internet the the the content viewed, generated and exchanged by patients via social media, patients by and exchanged generated viewed, content Social media are often used by patients as as patients often used by Social media are Although social media have been used for different aspects of research, such as patient patient as such aspectsresearch, of different for used been have media social Although number of pharmaceutical companies have begun to make use of social media to media to social make use of begun to have companies of pharmaceutical number INTRODUCTION trials’ (1). Patient registries and electronic health records are established examples of RWD examples established are electronic and health records registries Patient (1). trials’ RWD of media. social be may source potential another but sources, health conditions, share their experiences and find social support (3,4). For example, many many example, For supportsocial find and experiences their share (3,4). conditions, health with date up to stay to Twitter use patients conditions – does more good than harm in comparison to one or more alternatives (1). (1). alternatives one or more harm to than good in comparison – does more conditions Traditionally, of authorization However, trials (RCTs). controlled randomized from innovative yet expensive medications, medications, expensive yet innovative becoming is and patients companies pharmaceutical regulators, decision-makers, for ‘ as is defined important.increasingly HTA of effects and provide health, promote medical conditions, or treat diagnose prevent, to developed is relative delivery important care health An (2). of HTA organize or element rehabilitation, i.e. effectiveness, regarding for data term ‘an umbrella as can be defined data world in not collected are that interventions their knowledge on their disease, while Facebook is more often used for social support social often for used is more and while Facebook their knowledge their disease, on (3). Social users who have media experiences exchanging a their assess to and (13,14) (AEs) events perspectivesadverse on patient into insight gain Similarly, (15). behaviours switching the little is known (11), and education (9,10) interventions of dissemination (6-8), recruitment REA. In to 2008 its contribution about to use use to can be gathered. perspectivesexperiences and on patient of information amount information provide to management and diabetes diabetes regarding experiences patient enhancing by HTA for (ABPI) has published guidelines on best practices for for practices best on guidelines published has (ABPI) excluded from such RCTs. Therefore, researchers may additionally resort to real-world resort additionally real-world may to researchers Therefore, such RCTs. from excluded (RWD)data as highly selective patient groups within RCTs may result in findings that are not generalizable not generalizable are in findings that result may RCTs within groups selectivehighly patient practice, In routine heterogeneous. more are patients where settings clinical routine to eventually may comorbidities with elderlyand patients people children, women, pregnant receive Within Within

REA of REA of Google Google rapid pace pace rapid the the a creation and exchange of of and exchange creation the (Facebook(tiab) OR Twitter(tiab) OR OR Twitter(tiab) OR (Facebook(tiab) PRISMA guidelines (22). To identify identify To (22). guidelines PRISMA the period 1 January between 2016 2010 and 28 June literature, which were included based on title and and title on based included were which literature, extent to which health data generated from social social from generated health data which to extent following keywords: ‘social media, online patient, patient, online media, ‘social keywords: following Food and Drug Administration (FDA) is increasingly increasingly is (FDA) Administration and Drug Food the the purposes of this explorative review, social media were media were social review, purposes of this explorative the the vast amount of websites retrieved through through retrieved websites of amount vast the the the European Medicines Agency reported one-third that 2015 in European the new active substance recommended for market access were for for were market access for recommended new active substance search for peer-reviewed published articles published carried was out in peer-reviewed for search a following search query search used: was following a browser was cleared to ensure findings would not be influenced by would not be influenced findings ensure to cleared was browser The The reference lists from from lists reference considerable number of innovative drugs being developed at at being developed drugs number of innovative considerable PubMed interface for interfacePubMed for use of health data from social media by collaborating with PatientsLikeMe; PatientsLikeMe; with collaborating by social media from data use of health the the The The

group of Internet-based applications that allow allow that applications of Internet-based group the the 2016. 2016. a th aim of this articleaim of assess is to medicines with medicines Google search was conducted in July and August 2016 to identify grey literature, such such literature, grey identify to 2016 in July and August conducted was Google search history of

The The A the platform where patients can share their health data online to gain insight into patient patient into gain insight online to health data their can share patients where platform

METHODS defined as ‘ of health based on traditionally drugs are of REAs earlier, mentioned As (19). treatment cancer of often in light survival. However, survival progression-free and such as overall outcomes drugs, survivalfor oncological survival overall progression-free and in differences marginal important more in REA even becoming are and quality of life adherence on AEs, information such sources data other therefore difficult, be can aspects Collectingthese RCTs (20). from For be useful. may as social media in this area. For example, example, For in this area. user-generated content’ (21). (21). content’ user-generated media have provided insights for REA. We conducted an explorative review to identify identify to review an explorative conducted We REA. for insights provided media have was Oncology media. using social collected were data health where oncology in examples due to chosen new therapies. perspectives on adverse events (17,18). In light of these initiatives, it may become become may it In (17,18). these initiatives, of light events perspectives adverse on to contribute to on social media patients reported data health by for possible AEs via such sources (16). Moreover, Moreover, (16). such sources AEs via a focusing on on focusing An explorative review was performed was review on based explorative An scientific literature, literature, scientific blog(tiab) OR blogging(mesh) OR “social media”(tiab) OR ehealth(tiab) OR e-health(tiab) OR ehealth(tiab) OR OR media”(tiab) blog(tiab)“social OR blogging(mesh) OR patient”(tiab) OR “health OR “online communities”(tiab) community”(tiab) OR “online “online OR (personal*(tiab) AND platform*(tiab)) AND (tiab) AND research(tiab) OR (online data”(tiab) AND communit*(tiab)) AND patient(tiab) OR (online(tiab) health(tiab) AND record*(tiab)) OR OR cancer(tiab) AND data(tiab) AND shar*(tiab))) AND (oncolog*(tiab) OR (online(tiab) OR tumor(tiab) OR OR metast*(tiab) OR neoplasms(mesh) OR melanoma(tiab) carcinoma(tiab) tumour(tiab)). MEDLINE via MEDLINE 28 on June as relevant websites, by combining combining by websites, as relevant abstract, were hand-searched to identify additional literature. literature. additional identify to hand-searched abstract, were research, effectiveness research, health effectiveness, relative platform, research online search, each Before quality of life’. measure to and/or adherence, pharmacovigilance, the Due to queries. search previous

Novel sources for generating RWE 8 152 Novel sources for generating RWE 8 153 total a total topics, topics, resulting resulting titles and and titles the the hand search of of hand search The The a Google search due due Google search the strengths, limitations and and limitations strengths, the peer-reviewed original research research original peer-reviewed a collection of health data. Data were were Data collection data. health of the reviewers for eligibility. eligibility. for reviewers the public domain. We excluded literature that did did that literature excluded We domain. public extracted data was carried out, since extracted carried since was out, data the the Google search is inherently different from searching searching from different is inherently Google search a website in search of relevant reports or documents and by using reports using by and or documents of relevant in search website platform (i.e. (i.e. platform the the total of 8 scientific publications (Figure 1). 1). (Figure publications 8 scientific of total 35 full scientific publications and documents assessed, 27 were excluded: excluded: were 27 assessed, and documents publications 35 full scientific a following websites were included: PatientsLikeMe, Microsoft HealthVault, Microsoft HealthVault, PatientsLikeMe, included: were websites following predefined data abstraction form. Information on study characteristics (e.g. characteristicson study form. (e.g. abstraction data Information predefined selection of relevant websites was also based on consensus between between consensus on based also was websites selectionrelevant of a the The The report that was available in in report available was that a The The review was conducted by four reviewers (RK, RtH AM, reviewers KM) and and four by conducted was review character of descriptive qualitative analysis of analysis qualitative descriptive authors RK and RtH. These websites were hand-searched to identify grey literature by by literature grey identify to hand-searched were websites These and RtH. RK authors following keywords: ‘social media, internet, Facebook, Twitter, pharmacovigilance and/ pharmacovigilance Twitter, Facebook, internet, media, ‘social keywords: following The The A Two reviewers (RK and AM) independently extracted data from all included articles all included extracted from independently (RK and AM) data and reviewers Two the total of 1032 citations were identified from scientific literature (n=879), (n=879), literature scientific from identified were of 1032 citations total website). website). RESULTS methods and outcomes of included literature were notably diverse. diverse. notably were literature of included and outcomes methods a WEB-RADR, National Association, Drug Information MyGly, WhatNext, CureTogether, Dossio, Geneesmiddelen, van Beoordeling Network, ter Clinical Research College Patient-Centered Monitoring WHO Lareb, Medicine, Personalised for Alliance MyHandle European Health, Social Media Research PEW Uppsala, Center, Research Pharmacovigilance for Centre iVitality. and Healthy.me, MediGuard, Treato, Foundation, to by screened independently was literature Disagreements extracted. were health data generate media to social of using acceptability RK and AM. amongst consensus by resolved extractedin data were or health research’. These keywords were different from those used for those used from different were keywords These research’. or health the abstracts from scientific literature were assessed by RK, AM and KM, while grey literature literature by RK,grey assessed while AM and KM, were literature scientific abstracts from when it was: inclusion for eligible considered was RK and RtH by assessed was Literature . 1 January between 1) published 3) examples in English, 2) available 2016, June 2010 and 28 on focused collect 4) literature health data, used to social media were where provided were either was 5) literature and treatment, or cancer cancer reports using article or reviewed and full articles criteria. Relevant all inclusion not meet retrieved and reports were inclusion. for study design, study period, type study period, and used), social media of study design, search, only websites that collect health data online, focus on patient-reported outcomes, outcomes, on patient-reported focus collect online, that health data websites only search, further for relevant deemed were conditions and drugs on information online provide or analysis. the through browsing A reference lists from scientific literature (n=56), and grey literature (n=97). From these, these, From (n=97). literature grey and (n=56), literature scientific from lists reference were or abstract, based on title 9 duplicates additionally excluded were citations of 988 Of excluded. 15 citations did not provide an example of health data collection, 9 were not oncology- collection, of health data 9 were an example did not provide 15 citations on information insufficient provided 3 and specific, abstracted from abstracted from

analysis. analysis.

the main focus of of main focus The The survey was posted survey posted was

a 97

Grey Literature Search Literature Grey publications. publications.

the Grey literature 79 literature Grey Reference lists 46 lists Reference and/or abstract 988 abstract and/or

Scientific literature 863 literature Scientific

Excluded based on title on based Excluded

total of 4 studies collected health data on AEs on AEs data health collected of 4 studies total 8 scientific publications included. Different types types Different included. publications 8 scientific number of respondents included in included respondents of number a the the 879 patient community or support group, in 1 study Twitter Twitter in 1 study community or supportpatient group, a Included in in Included 8 studies, 8 studies, feasibility and added value of generating health data from from data health of generating added value and feasibility for eligibilityfor 35 Reference lists 6 lists Reference 0 literature Grey PubMed Search PubMed Grey literature 13 literature Grey 2 lists Reference Scientificliterature 6 Scientific literature 16 literature Scientific explorative review8 the the literature review process review literature Full paper or document assessed assessed document or paper Full the survey. Of survey. a Hand search of of search Hand reference lists 56 lists reference - Flowchart of - Flowchart Facebook page of either page Facebook

the process review literature of 1. Flowchart Figure

number of posts analysed and analysed posts of number

Table 1 provides an overview of 1 provides Table period, study design, in study substantially differed publications that shows 2 Table the

Figure 1 Figure Forum topics and discussions were assessed in 4 papers, in 2 studies in 4 papers, assessed discussions were and topics Forum on to used was platform patient an online assessed and in 1 study were conversations disseminate of cancer and medications were assessed in each of were and medications of cancer the all 8 articles testing was from and experience occurrence symptom QoL, adherence, AEs, social media, such as media. social (23,24,27,29). Another 3 studies collected health data on quality of life (QoL) (25,26,30). (25,26,30). (QoL) on quality data health collected 3 studies of life Another (23,24,27,29).

Novel sources for generating RWE 8 154 Novel sources for generating RWE 8 155 a N/A N/A Aromatase inhibitors N/A Drug N/A OAN agents agents OAN Chemo- therapeutic agents Methotrexate cervix the Pigmented Pigmented villonodular synovitis Chronic lymphocytic leukaemia Cancer Breast Cancer Breast Cancer Type Cancer Cancer Neuroendocrine carcinoma of Colorectal Colorectal cancer N/A Concerns About Recurrence Scale Scale Recurrence About Concerns the total of 23 drugs and 4 vaccines, including 1 drug including 1 drug 4 vaccines, drugs and of 23 total a French French the aforementioned main outcome measures, measures, main outcome the aforementioned breast cancer cancer breast a regulatory agency ; AE’s, adverse events; N/A, not applicable; PVNS, pigmented villonodular synovitis; synovitis; villonodular pigmented PVNS, applicable; not N/A, events; adverse AE’s, ; a

feasibility of using social media media social of using feasibility level of concordance between Twitter Twitter between of concordance level characteristics of AE’s reported reported characteristicsAE’s of feasibility of using social media-based media-based social of using feasibility the research study and who responded to to responded study and who research the the a the real world setting world real symptom checklist symptom To investigate whether we could use crowdsourcing use crowdsourcing could we whether investigate To research medical online surveys and for Facebook via purposeson PVNS To explore explore To on data qualitative gather networks to patient CLL to relevant concepts outcome patient-reported and AE’s of frequency understand and content To breast by discussed behaviours adherence associated AI using to related patients cancer patterns symptom and examine identify To extracted data from by generated describe To online in an agents OAN to exposed patients by reported with those these and compare discussion, in as recorded professionals health by forum, and compare these findings to an analysis to an analysis these findings compare and forum, survivors cancer breast reported by of symptoms in enrolled a Aim To determine determine To performto and QoL cross-sectional epidemiologic tumours gynaecologic with rare on patients research pharmacovigilance database pharmacovigilance To better understand patient experience with CCC in in with CCC experience patient understand better To the evaluate To AE-like reactions spontaneous and mentioning posts by reports received - Overview of included scientific publications. This study assessed adverse events reported in social media for reported media for social in events adverse study assessed This

CLL, chronic lymphocytic leukaemia; AI, aromatase inhibitors; OAN, oral antineoplastic; QoL, quality of life. QoL,quality antineoplastic; oral life. of lymphocyticchronic CLL, OAN, leukaemia;inhibitors; AI, aromatase a CCC, colorectal cancer chemotherapies cancer colorectal CCC, van der der van Heijden (25) 2016 McCarrier McCarrier (26) 2016 Mao (27) 2013 Marshall (28) 2015 Pages (29) 2014 (methotrexate) specific for oncology. specific (methotrexate) Study Zaid (30) 2014 Beusterien (23) 2013 Freifeld (24) 2014 Table 1 Each study used different QoL instruments, such as such QoL instruments, different Each study used scores (30), and short form-36 health survey (25). Finally, 1 study focused on identifying identifying on 1 study focused and short (30), survey health scores form-36 (25). Finally, In (28). addition to (co-)symptom occurrence on socio- data collected (25,26,30) al. Zaid et and McCarrier et al., et al., der Heijden van Furthermore, et al. Beusterien factors specific characteristics. and disease demographic functioning emotional impacts and on physical (23), and Mao health data collected et al. or stopping continuing about decisions mapping by adherence on information collected (27). treatment

Disease-specific characteristics include clinical presentation, initial work-up, treatments, past and current disease status, follow-up, and recurrence pattern. recurrence and follow-up, status, disease current and past treatments, work-up, initial presentation, clinical include characteristics Disease-specific

Disease-specific characteristics include self-reported current CLL stage, performance status, and past and current treatment. current and past and status, performance stage, CLL current self-reported include characteristics Disease-specific

and post-operative complications. post-operative and

Disease-specific characteristics include clinical presentation, findings on imaging and biopsy material, type and localization of disease, surgical and adjuvant treatment, local recurrences, recurrences, local treatment, adjuvant and surgical disease, of localization and type material, biopsy and imaging on findings presentation, clinical include characteristics Disease-specific

AE’s, adverse events; N/A, not applicable; QoL, quality of life. life. of quality QoL, applicable; not N/A, events; adverse AE’s,

specific characteristics specific 2014 (30) 2014 , and QoL and ,

Socio-demographic factors, disease- disease- factors, Socio-demographic group) (support Facebook 57 A N/ days 30 Cross-sectional Zaid

AE’s forums health 5 66 posts 111 year 1 Retrospective (29) 2014 Pages

25 pre-selected symptoms. symptoms. pre-selected 25

co-occurrence, and similarity index of of index similarity and co-occurrence,

1 disease- specific forum specific disease- 1 12,991 posts 50,426 years 8 Retrospective (28) 2015 Marshall Symptom occurrence, occurrence, Symptom

12 disease- specific forums specific disease- 12 N/A posts 1,235,400 years 8 Retrospective (27) 2013 Mao AE’s and adherence and AE’s

treatment, and QoL QoL and treatment,

of symptoms, perceptions about about perceptions symptoms, of

specific characteristics specific , experience experience ,

Online patient platform patient Online 50 A N/ months 4 Cross-sectional (26) 2016 McCarrier Socio-demographic factors, disease- disease- factors, Socio-demographic

outcome, and QoL and outcome,

, functional functional , specific characteristics specific community) 2016 (25) 2016

Socio-demographic factors, disease- disease- factors, Socio-demographic Facebook (patient (patient Facebook 272 A N/ months 70 Prospective van der Heijden Heijden der van

Twitter N/A posts 6,900,000 months 7 Retrospective (24) 2014 Freifeld AE’s AE’s

2013 (23) 2013 emotional impacts emotional

Beusterien Cross-sectional 52 days 52 1522 posts posts 1522 264 2 disease specific forums specific disease 2 AE’s, physical functioning & & functioning physical AE’s,

collect health data health collect Type of health data collected data health of Type respondents Study Study design Study Study period Study Posts analysed analysed Posts

Type of social media used to to used media social of Type Number of of Number

- Study characteristics of included scientific publications that use social media to collect health data. data. health collect to media social use that publications scientific included of characteristics Study - 2 Table

Novel sources for generating RWE 8 156 Novel sources for generating RWE 8 157 2 the ability ability patient patient different different a The The a accuracy of accuracy of forum posts posts forum web crawler crawler web a the the survey (25,26,30). a other four publications, publications, other four efficient collectionpatient- efficient of The The collection method used (23,29) collection(23,29) used method the inability to actively probe patients patients inabilityactively probe to the Natural Language Processing (NLP) (NLP) Processing Language Natural difficulty of verifyingdifficulty of a difference between results from social social from results between difference the a the patient population that use social media media social use that population patient Twitter application programming interface programming application Twitter drugs (23,26) they discuss. Regarding selection Regarding they discuss. drugs (23,26) the the the remaining five publications made use of content content of made use publications five remaining use of automated processes to collect health data collect data health to processes use of automated selected forum (28). (28). selected forum the The The social media platform to distribute distribute to social media platform large volume of health data (24) or access to to (24) or access data of health volume large the a short time-period survey and (28,30), to needed patients the the 8 included publications are presented. Five publications identified identified publications Five presented. are publications included 8 major strength by 5 publications (25-28,30). Furthermore, Freifeld Freifeld Furthermore, (25-28,30). 5 publications by major strength a the access of access included publications indicated to have collected all all collected have to indicated publications included strengths and limitations of health data generated through social media social media through generated of health data and limitations strengths the the the disease (26,30) or are indeed on or are (26,30) disease four publications that used forums to collect health data varied substantially in in collect substantially varied health data to forums used that publications four the Limitations of social media-generated health data mainly focused on validating on validating focused mainly health data of social media-generated Limitations health data collected (24,27,28). For example, example, For (24,27,28). collected data health explanation for their forum selection (Table 3). For example, Beusterien et al. used two two used Beusterien et al. example, For 3). selection (Table forum their for explanation ability to assess patient perspectives important as an ability patient (23,24,27-29). assess to strength identification of new or unlabelled AEs (29). AEs new or unlabelled of identification In Table 4 In Table The The

community (25,26,30). Regarding community Regarding (25,26,30). making use of Twitter, Facebook or an online patient platform, selected this social media selected social media this platform, online patient or an Facebook Twitter, making of use due to platform from social media, two publications specifically indicated to have used used to have indicated specifically publications two social media, from search engines and two different computers for their forum search which they repeated repeated which they search forum their for computers different and two engines search selection used they include criteria to Additionally, weeks. two every for day other browsing, currently >20 individuals forum, posts on >12,000 active >5 years, site (i.e. forums without selected Marshall forum one et al. Meanwhile, (23). day) per and >10 new posts clarifying selection criteria for the and three publications used publications and three were patients example, For (24,28,29). be present may methods media and conventional provide who traditionally professionals to health compared AEs report to different shown Other (29). included identified this information strengths et al., Marshall et al. and Pages et al. emphasized that social media should complement complement should media social that emphasized et al. and Pages Marshall et al. et al., since methods, (pharmacovigilance) conventional (27,28) and one publication made use of made use one publication and (27,28) of Two (24). indicating specifically without terms search to related analyse to and Marshall al. et Mao al. et et al., Freifeld by used were techniques Automated the were algorithms mining AEs (24), or data to identify used was classifier semi-automated (28). symptoms identify used to or (25,30), test) chi-squared (e.g. analysis and/or quantitative descriptive (23,26), analysis (29). manually posts forum labelled in identified were that geographic wide over distributed are and/or diseases rare who have patients access to considered was areas the authenticity, selection bias, information bias, and bias, information selection bias, authenticity, conventionally media are social using patients example, those who do not; for to compared have may (25,26), be female likely to more 28), are (23, educated highly more reported outcomes (23), reported outcomes bias, publications reported differences in reported publications differences bias, for responses. Validating authenticity focuses on on authenticity focuses Validating responses. for actually posters whether verifying such as (25,28), via social media provided information have the inability of using inability of using the No No Yes Automated technique used used technique Automated of health data analysis for Yes No Yes No No patient perspective provided by by perspective provided patient the b a b b No No No No Yes Web crawler used for used for crawler Web collectingsocial media health data No No Yes

importance into of insights social media platform. platform. social media acceptability of using social media to generate health data, Pages Pages data, health generate to media of using social acceptability the same patients (27), and Freifeld et al. indicated that patients may not not may patients that indicated al. et Freifeld and (27), patients same the the the use of social media to generate health data (24). Reasons for their caution their caution (24). Reasons for data health generate use of social media to Yes Yes Yes Yes No Clear explanation for for explanation Clear selection of social media platform Yes Yes Yes

the

need to still establish its role in pharmacovigilance as social media are not yet used used not yet are media as social pharmacovigilance in its role establish still need to

- Selection of social media platform and use of automated techniques included by literature that

the Twitter application programming interface (API) was used to identify relevant tweets. relevant identify used to interfacewas (API) programming application Twitter

With regards to to regards With survey via distributed was

The The Zaid 2014 (30) 2014 2014 (24) 2014 Freifeld Freifeld Pages (29) 2014 Beusterien Beusterien (23) 2013 2015 (28) 2015 Marshall Marshall Study a b 2016 (26) 2016 van der Heijden Heijden der van (25) 2016 McCarrier Mao (27) 2013 use social media collectto health data. Table 3 in routine surveillance. In addition, they indicated that data acquisition from social media media social from acquisition data that In surveillance. indicated they addition, in routine be improved. need to thereof and automation et al. indicated that pharmaceutical companies are already using this type of data to gather gather type this using to of data already are companies pharmaceutical that indicated et al. indicated Beusterien et al. perspectives Furthermore, (29). patient on AEs from information accepted commonly perspectives are patient research, outcomes in patient-reported that der van and al. et impact Freifeld (23), and both disease and treatment to with regards noted al. et Heijden was was symptom experience (27), and are generally younger (26,28,30). With regards to information information to regards With (26,28,30). younger generally (27), and are experience symptom reported reported Mao (24, 29), posts of et al. duplication et al. and Pages al. et Freifeld bias, by posts multiple social media to actively probe patients for responses (23, 26, 28). For example patients patients example For 28). 26, (23, responses for patients activelyprobe to media social lead to which could anticipate, which researchers than that wording alternative use may (28). experiences symptom misclassifying also was et al. Freifeld regulatory authorities However, (24,25). for social media research on cautious identify AEs correctly (24). Finally, several publications mentioned mentioned publications several (24). Finally, AEs correctly identify

Novel sources for generating RWE 8 158 Novel sources for generating RWE 8 159

Validating authenticity; Validating Selection bias; No active probing of patient responses; of patient No active probing saturation; concept Not achieving needed; sample sizes Larger Selection bias; bias; Information of an indication is not data Frequency AEs; prevalence authenticity; Validating Selection bias; data; Noisy responses; of patient No active probing of sample; information Incomplete inadequate; qualityData format or considerations; Ethical of posts; Misinterpretation bias; Information Validating authenticity; Validating Selection bias; Information bias; Information posts; of Volume data; Noisy authenticity; Validating Selection bias; participation rate; Low Limitations authenticity: Validating Selection bias; responses; of patient No active probing of sample; information Incomplete

the use of social media generate to health data

Access to patients with rare diseases and and diseases with rare patients to Access wide over distributed are that areas; geographic Short time-period; patients; Motivated Complementarypharmacovigilance; to Vast quantities of data; data; of quantities Vast information; accessible Easily perspective; Patient Short time-period; perspective; Patient studies Complementarytraditional to AEs new/unlabelled Identification Patient perspective; Patient diseases; with rare patients to Access costs; Low Access to patients distributed over wide wide over distributed patients to Access areas; geographic more generalizabilityIncreased due to population; patient diverse Observed frequency reflected AEs key reported studies those traditional in Access to patients with rare diseases; with rare patients to Access patients; Motivated patient per enrolled costs Lower Alternative approaches to qualitative qualitative to approaches Alternative collection; data Support development of PRO instruments; Support of PRO instruments; development Convenient to fill in; in; fill to Convenient Rapid information on AEs; Rapid information diseases; rare with patients to Access Collection of PROMS; follow-up Long-term Patient perspective; Patient Complementarypharmacovigilance; to Strengths perspective; Patient collection comprehensive and Efficient of PROMS;

- Strengths and limitations specificto

PROMS, patient-reported outcome measures; AEs, adverse events; PRO, patient-reported outcomes patient-reported PRO, events; adverse AEs, measures; outcome PROMS, patient-reported 2014 (30) (30) 2014 2016 (28) 2016 (29) 2014 Marshall Marshall Pages 2013 (27) 2013 Mao Zaid 2016 (26) 2016 McCarrier McCarrier van der Heijden Heijden der van (25) 2016 2014 (24) 2014 Study Beusterien (23) 2013 Freifeld Table 4 few issues issues A the extraction, risk of bias risk of bias the the authenticity of health data authenticitydata health of use of correct search terms, terms, search use of correct the the great potential for social media- for potential great a field of oncology, social media could could media social field of oncology, drug or its effects) (42, 44). To manage To manage 44). (42, effects) drug or its vast amounts of data available on social on social available of data amounts vast the the the health data collected via social media. Extracting collected data health the possibility of social media to generate data on AEs, QoL, and QoL, and on AEs, data possibility generate to media of social the quality of the results found in this review on social media-generated data in oncology in oncology data on social media-generated in this review found results specific drug but fail to reportto drug but fail specific a the lack of clear methodological guidance. Standardized approaches to collecting health collecting health to approaches Standardized guidance. methodological lack of clear interpretation of these posts could also be done manually or by automated processes. processes. automated or by also be done manually of these posts could interpretation

One caveat of using social media to collect health data that requires special attention attention special requires collect that data health to media social of using One caveat Arguably, Arguably, the

DISCUSSION is interpret successfully to unable be may processes automated that argue some However, natural automated that while others argue on social media (24), in text posted sarcasm assist in analysing could processing language generated health data to enrich REA by incorporating information on these aspects. on information incorporating enrich REA by to health data generated between and reproducibility necessary comparability social media are ensure to from data processes. be extracted either automated may posts manually or by example, For studies. The may not be generalizable to other fields of medicine. However, many studies conducted conducted studies many However, other fields of medicine. to not be generalizable may AEs (31-37), on identifying focused similarly oncology other than of medicine in fields partially least little is known at Although about are generalizable. our results suggesting mode this for is potential there QoLassessing medicine, of other fields in media via social observational and in to measure QoL collection often is of health data since difficult RCTs may that effectiveness aspect another of relative show, results as our Finally, (20). studies behaviour. and adherence is treatment-switching be assessed via social media as posts may include misspellings, non-medical terms, and slang (24,34,41). Additionally, Additionally, (24,34,41). and slang terms, non-medical include misspellings, may as posts reported when assessing studies important consider to several limitations methodological be not may posts authenticity (e.g. validating include which media, social from data ethnicity gender, in age, differ media users may social selection (42-44), genuine) bias (e.g. patients bias (e.g. information and (41,43,44) non-users) to compared location and physical be takingmay media (34,39,40). Another methodological issue involves issue involves methodological Another media (34,39,40). assess is important it limitations methodological systematically these to from social media is difficult, and is prone to selection and information bias. Furthermore, Furthermore, bias. to selection prone information and and is social media is difficult, from of research. this typebe used complementary forms should data of traditional to its thus demonstrating this aspect been assessing already, have companies pharmaceutical Given (14,15,38). potential determine to in order treatment-switching and adherence behaviour, there is there behaviour, and adherence treatment-switching be used for assessing AEs by collecting health data from forums and to evaluate QoL via QoL via evaluate and to forums collectingfrom data health by AEs assessing used for be opportunity an Social provides media efficiently to platforms. or online patient Facebook are that diseases rare with patients perspectives collect and from data health assess patient validating However, areas. geographic wide over distributed to due and challenging be difficult media may social from health data relevant improve may guidance methodological and uniform Clear described above. collect health data. to social media use of subsequent and interpretation This explorative review demonstrates that, within within that, demonstrates review explorative This

Novel sources for generating RWE 8 160 Novel sources for generating RWE 8 161 ability of ability of European European difference difference forum) and and forum) a a the the assessment of of assessment 2016 elections2016 the the comprehensive overview overview comprehensive a recent example of manipulation in in of manipulation example recent vast amount of health data available available of health data amount vast A collection of use of social media-generated health health media-generated use of social the the the possibility of missing important insights. possibility important of missing insights. use of social media for collecting health data collectinghealth data for media social use of impact severity on disease of menopause in European MedicinesAgency and European review through data abstraction conducted by abstraction data by conducted through review focus on oncology, which may have resulted in in resulted have which may on oncology, focus survey on Facebook). There may be may There survey on Facebook). the the the field of oncology, studies that assess these new new these assess that studies oncology, of field a the the the search strategy employed in this explorative review. review. explorative this in employed strategy search the the better substantiation of deductions made. of deductions substantiation made. better qualityof a focus on oncology in this review was deemed appropriate since since appropriate deemed was review in this on oncology focus the circulation of fake news on social media duringsocial media on news fake of circulation comprehensiveness of this review is that we assessed both academic academic both assessed we is that of this review comprehensiveness The The effectiveness of off-label drug use(42,49). Additionally, PatientsLikeMe PatientsLikeMe Additionally, drug use(42,49). of off-label effectiveness the the the perceived risk of easy manipulation. risk manipulation. of easy perceived study focused on assessing on assessing study focused broad definition of social media that was used in this review may not allow for for not allow review may this in used was of social media that definition broad a the added value of collecting information on patients’ perspectives and experiences perspectives and experiences on patients’ of collectingadded value information United States of America (45-47). These kindThese affects of examples of America (45-47). States United the information available from passively collecting information that patients discuss and and discuss patients that collecting information passively from available information second limitation relates to to relates limitation second the better source to identify AEs that are mild or symptom-related compared to more more to compared or symptom-related mild are AEs that identify to source better An additional caveat that may hamper hamper may that caveat additional An Although caveats can be recognized in can be recognized caveats Although Important for A One limitation of this review was was review this of One limitation a the the few studies on few social media was was social media for REA is REA for in although However, information. correct and is true what discern to users social media exchange to and information find use social media to still many occur, may manipulation harnessing and analysing Therefore, experiences. drugs can be small and incomplete, and incomplete, and small be can drugs on focus putting also are Assessment Technology Health network for patients with multiple sclerosis.(50) These types of data may contribute to providing providing to contribute types may These of data multiple sclerosis.(50) with patients REA. for information in developed are drugs new many on social media remains important. media remains on social REA. for information of aspectscan provide than oncological drugs. drugs. oncological in data, data, using collected be potentially could that REA to on other aspects related missing literature to patients allows that platform patient an online PatientsLikeMe, example, social media. For published and medications, on conditions experiences and/or exchange health data share a published traditional methods (43). However, health data collected via social media should be used in be used in via social media should collected data health However, (43). methods traditional ensure conjunction methods to with traditional towards relative effectiveness (e.g. AEs, quality of life, switching-behaviour) should be should be switching-behaviour) quality of life, AEs, (e.g. effectiveness relative towards that AEs uncover may media via social collected health data example, For highlighted. to detect AEs earlier they may compared or after of new drugs, use long-term occur literature in published available not are that insights or provide (43,48), methods traditional social media may Additionally, (12). therapy) with laser experiences patient diabetes (e.g. be Additionally, we ensured ensured we Additionally, and grey literature, which minimizes minimizes which literature, and grey actively collecting data (e.g. by posting posting by actively collecting (e.g. data differentiating between passively collecting data (e.g. by collecting by posts from collecting (e.g. data passively between differentiating Firstly, Firstly, two authors, which allowed allowed which authors, two survey. survey. limited limited a the PubMed and and PubMed the degree of usefulness of such data for relative relative for of usefulness of such data degree reference lists of included studies, based on title title based on studies, lists of included reference Google search engine. To overcome this limitation limitation this overcome To engine. Google search the the the limitations mentioned above. mentioned limitations the few articles in few not been captured had that a potential source of RWD REA, particularly for source potential on aspects AEs, such as a

CONCLUSION Google search. Google search. and abstract, and identified and abstract, and identified Secondly, by employing one database for our scientific and grey literature search we may we may search literature grey and our scientific for database one employing by Secondly, or grey PubMed by indexed not are journals that in relevant published missed studies have by not identified was that literature to some extent, we hand-searched hand-searched we extent, some to occurrence of disease-specific symptoms, adherence behaviour, and QoL. This potential has has This potential and QoL. behaviour, disease-specific of adherence occurrence symptoms, media- social accompany that limitations methodological due to realised fully been not yet as bias and selection well as bias, information such as data, health generated acceptability of such data. However, However, of such data. acceptability post on social media, compared to actively posing questions to these patients in in these patients actively to questions posing to media, compared on social post Social media may be be Social media may effectiveness should be further explored. Moreover, methodological guidelines and tools tools and guidelines methodological should be furtherMoreover, effectiveness explored. address to be developed should

Novel sources for generating RWE 8 162 Novel sources for generating RWE 8 163

systematic systematic A management of of management literature and weblogs literature the the Pharmaceutical Journal. http:// Journal. Pharmaceutical use of social media in oncology practice. J practice. use of social media in oncology FDA Sign Research Collaboration Agreement. Agreement. Collaboration Research Sign FDA PatientsLikeMe. 2015. PatientsLikeMe and PatientsLikeMe 2015. PatientsLikeMe. the http://blog.patientslikeme.com/2015/06/15/ patientslikeme-and-the-fda-sign-research- http:// at: Archived collaboration-agreement/. www.webcitation.org/6q0x3uFXv Powell G, Seifert H, Reblin T, Burstein PJ, PJ, Burstein T, Seifert G, Reblin H, Powell Thomas JL, Menius Painter JA, J, Blowers Brownstein Rodriguez CE, HW, M, Pierce Bell N. HG, Dasgupta CC, JS, Freifeld post- routine for Social media listening Drug surveillance. marketing safety SafPMID:26798054 2016;39(5):443-54. Sukkar social Searching 2015. E. reactions. detectnetworks adverse to The www.pharmaceutical-journal.com/ news-and-analysis/features/searching- social-networks-to-detect-adverse- at: reactions/20067624.article. Archived http://www.webcitation.org/6q0wwYsjE Bonzani I, Huisman A, Saini D, V, Risson in switching treatment of Olson M. Patterns in US patients therapies sclerosis multiple of active media: Application social on health to analysis content social media Med J Internet research. outcomes Res PMID:26987964 2016;18(3):e62. Expert Network. ABPI Pharmacovigilance on notes Guidance Dizon D, Graham D, Thompson M, Johnson M, Thompson D, Graham D, Dizon guidance: Practical MJ. Fisch C, Johnston LJ, The PMID:23277774 2012;8(5):e114-24. Pract Oncol K, AJ, Facey Braunack-Mayer JM, Street Virtualcommunity Hiller RE, JE. Ashcroft Using consultation? and product complaints events adverse 2013. media. digital from on health behavior change: change: behavior on health MedJ Am Inform and meta-analysis. review PMID:25005606 2015;22(1):243-56. Assoc to link community link perspectives and healthto Expectations Health assessment. technology 18430153 PMID: 2008;11(2):189-200.

16. 17. 14. 15. 12. 13. 11.

literature. J Med literature. the new model for promoting promoting for model new STARSurg Committee. Social Committee. STARSurg a review of review the A diagnosis difference. http://www. difference. diagnosis influence of social networking influence sites Twitter use of cancer patients in Japan. in Japan. patients of cancer use Twitter Laranjo L, Arguel A, Neves A, Gallagher AM, A, Gallagher AM, Neves A, Arguel L, Laranjo Kaplan R, Mortimer N, Mendes AYS. GA, Lau The Cavallo D, Chou W, McQueen RileyRamirezA, A, W, Chou D, Cavallo interventions and control prevention Cancer W. User-generated media: using social Biomarkers Epidemiol Prev Cancer approaches. PMID:25103820 2014;23(9):1953-6. Educ 2013;28(1):134-7. PMID:23292877 2013;28(1):134-7. Educ recruitment Online J. Gordon J, Armin T, Lane web-based health and mobile for methods studies: Kapp J, Peters C, Oliver D. Research Research D. C, Oliver Kapp Peters J, advertising: facebook using recruitment J Cancer challenges. big Big potential, M, Kelly KhatriJ, Glasbey S, Chapman C, on JE, Fitzgerald Bhangu A, D, Nepogodiev of behalf Pew Research Center. 2013. Center. Research Pew The pewinternet.org/2013/11/26/the-diagnosis- http://www. at: Archived difference/. webcitation.org/6q0wYe4XI recruitment: study driven and internet media Evaluating Tsuya A, Sugawara Y, Tanaka A, Narimatsu A, Narimatsu Tanaka Y, Sugawara A, Tsuya Examining Tweet? patients H. Do cancer the J Med Internet Res PMID: 2014;16(5):e137. 24867458 HTAi. HTA Glossary: Health Technology. http:// Glossary:Technology. Health HTA HTAi. Archived htaglossary.net/health+technology. http://www.webcitation.org/6uV8kwKnY at: T. K, Nieboer Tates M, Antheunis use and health professionals’ Patients’ Motives, care: health in media of social Educ Patient barriers and expectations. PMID:23899831 2013;92(3):426-31. Couns Makady A, Goettsch W, on behalf of GetReal behalf of GetReal on W, Makady A, Goettsch Workpackage 1. IMI-GetReal: Glossary of 2015. terms. common of definitions Internet Res PMID:26202991 2015;17(7):e183. collaborator engagement and participation. and engagement collaborator PMID:25775005 ONE 2015;10(3). PLoS

REFERENCE LIST REFERENCE 7. 6. 5. 4. 3. 2. 1. 10. 9. 8.

national national a cross-sectional a review. J Biomed J Biomed review. A feasibility survey. Gynecol survey. feasibility A research study. Qual Life Res Res Qual Life study. research a pilot study of adverse drug drug study of adverse pilot Scoping Review. J Med Internet Review. Scoping A A cervix: the Case Study for Diabetes. Intelligent Systems Systems Intelligent Diabetes. for Study Case epidemiologic and qualtity of life survey and qualtityepidemiologic of life of carcinoma neuroendocrine with patients of PMID:24145111 2014;132(1):149-53. Oncol Topaz M, Lai K, M, Dhopeshwarkar Seger N, Topaz Zhou R, Rozenblum GossR, F, Sa’adon DL, reports in electronic health L. Clinicians’ in social concerns patients’ versus records media: 2016;25(3):547-57. PMID:26476836 2016;25(3):547-57. L, C, Cantaloube Rossard M, Taam Abou Montastruc L, N, Roche Pochard G, Bouscaren Herxheimert A, Montastruc Bagheri JL, F, posted narratives of patients’ H. Analysis on benfluorex’s websites media on social J Clin in France. withdrawal (Mediator(®) PMID:24304185 2014;39(1):53-5. Ther Pharm H, Asfari Bellet R, F, Abdellaoui J, Lardon M, Beyens M, N, Jaulent Texier J, Souvignet Reaction Drug A, Bousquet Adverse C. Burgun Extraction and Identification in Social Media: Adverse Identifying Chen H. Liu X, Social Media: Patient from Drug Events A Sarker A, Ginn R, Nikfarjam A, O’Connor T, K, S, Upadhaya K, Smith Jayaraman Gonzalez for G. Utilizing social media data pharmacovigilance: PMID:25720841 2015;54:202-12. Inform Marshall S, Yang C, Ping Q, Zhao M, Avis N, Ip Ip N, Avis M, Zhao Q, Ping C, Yang MarshallS, breast with in women clusters Symptom E. social from of data analysis cancer: An media and Bondon-Guitton A, Montastruc J, E, Pages to oral related effects Bagheri Undesirable H. between drugs: Comparison antineoplastic and narratives internet patients’ Drug database. pharmacovigilance Saf PMID:25027671 2014;37(8):629-637. Basen-Engquist K, J, Bodurka Burzawa T, Zaid M. Use Frumovitz J, DC, Ramondetta Brown LM, conduct of social media to reactions of aspirin and atorvastatin. Drug reactions aspirin of and atorvastatin. Drug SafPMID:26715498 2016;39(3):241-50. Res PMID:26163365 2015;17(7):e171. DOI:10.1109/MIS.2015.7 2015;30(3):44-51. IEEE

34. 35. 32. 33. 30. 31. 28. 29.

world, unite! unite! world, the feasibility study study feasibility A PRISMA Statement. J Clin Statement. PRISMA The The challenges and opportunitieschallenges of social PRISMA Statement. Preferred Reporting Preferred Statement. PRISMA crowdsourcing study of two hundred hundred two of study crowdsourcing McCarrier K, Bull S, Fleming S, Simacek McCarrier Simacek S, K,Fleming S, Bull R. Concept Pierson D, Cella P, Wicks K, patient-powered within elicitation networks: research Value lymphocytic chronic in leukemia. Hill A, Chung A, Benton Ungar S, Mao J, JH. Holmes S, Hennessy CE, Leonard L, and effects side of drug discussion Online cancer among breast discontinuation and Drug survivors. Pharmacoepidemiol van der Heijden L, Piner S, van de Sande Sande de S, van Piner L, Heijden der van synovitis: villonodular M. Pigmented A Int Orthop patients. two and seventy PMID:27169531 2016;40(20):2459-68. PMID:26797235 2016;19(1):42-52. Health SafPMID:23322591 2013;22(3):256-62. Freifeld C, Brownstein J, Menone C, Bao W, W, Menone C, Bao J, C, Brownstein Freifeld Digital N. Dasgupta T, Kass-HoutR, Filice surveillance:drug safety Monitoring Drug Twitter. products in pharmaceutical SafPMID:24777653 2014;37(5):343-50. Items for Systematic Reviews and Meta- Reviews Systematic Items for Analyses: Y. S, Su S, Gholizadeh Tsay Beusterien K, with colorectal experience Real-world forum web patient chemotherapies: cancer 2013;7:361. Ecancermedicalscience analysis. PMID:24143155 Kaplan of Users M. Haenlein A, 2010;53(1):59-68. Horizons Business media. DOI:10.1016/j.bushor.2009.09.003 D, Altman J, Tetzlaff A, Liberati Moher D, the Report 2016. 2015. Meijboom K, T, Kleijnen S, Lipska I, Alves A, Timoney A, d’Andon V, Vervölgyi Elsada A, Relative WG. Boerde HG, GoettschA, Leufkens oncology of assessments effectiveness pricing and reimbursement medicines for Oncol Ann countries. decisions in European PMID:27329251 2016;27(9):1768-75. The Heywood B. Advancing knowledge from knowledge from Advancing Heywood B. 2016. experience. patient Annual Medicines Agency. European Epidemiol 2009;62(10):1006-12. PMID:20171303 Epidemiol 2009;62(10):1006-12.

27. 26. 25. 24. 23. 22. 21. 19. 20. 18.

Novel sources for generating RWE 8 164 Novel sources for generating RWE 8 165

2016 Election. Stanford Election. 2016 Stanford the patient-matching algorithm. Nat algorithm. Nat patient-matching a Perrott K. 2016. ‘Fake new’ on social media media on social new’ ‘Fake K. 2016. Perrott experts US electionvoters, say. influences http://www.abc.net.au/news/2016-11-14/ fake-news-would-have-influenced-us- election-experts-say/8024660 at: . Archived http://www.webcitation.org/6q8F0ScwQ S, C, Proestel Bouri CE, K, Pamer Pierce Brownstein CC, Freifeld H, Le van Rodriguez HW, N. IR, Dasgupta M, Edwards Walderhaug JS, monitoring Twitter and Facebook of Evaluation products: medical for detect signals safety to alerts. safety Drug Saf FDA of recent analysis An 28044249 PMID: 40(4):317-31. 2017; MassagliM, Heywood T, Vaughan P, Wicks self- discovery using clinical Accelerated J. online collected data reported patient and PMID:21516084 2011;29(5):411-4. Biotechnol Katic B, T, Vaughan E, Secor B, Healy R, Bove report Patients PL. Jager de P, Wicks T, Chitnis after menopause: MS symptoms worse cohort. online an from Mult Scler Findings PMID:25787049 2015;4(1):18-24. Disord Relat Leng H. Methodological issues in in issues H. Methodological Leng networking social from using data Soc Behav Cyberpsychol sites. NetwPMID:23679568 2013;16(9):686-9. Social M. Media H, Gentzkow and Allcott in News Fake Shows Analysis This 2016. C. Silverman Election Stories News Face Viral How OutperformedFacebook. On Real News https://www.buzzfeed.com/craigsilverman/ viral-fake-election-news-outperformed- real-news-on-facebook?utm_term=. at: . Archived ymmRRxdMPk#.mgpLLo5NM9 http://www.webcitation.org/6q8Eg1VLZ data in social media. Br J Clin Pharmacol J Clin Pharmacol media. Br in social data PMID:26271492 2015;80(4):878-88. University. 2017. 2017. University.

50. 48. 49. 46. 47. 44. 45.

mixed- source of of source figurative figurative A a review of review The The Internet. J MedInternet. A the prevalence, frequency and frequency and prevalence, the 6th International Joint Conference on on Conference Joint 6th International opportunities Br J Clin and challenges. the Golder S, Norman G, Loke Y. Systematic Systematic Y. Golder Norman S, Loke G, on review Frost J, Okun S, Vaughan T, Heywood J, Wicks Wicks Heywood J, T, Vaughan Okun S, J, Frost as outcomes Patient-reported P. events of adverse value comparative language of social media. Data & Knowledge Data of social media. language 10.1016/j. doi: 74:1-12. 2012; Engineering datak.2012.02.005 Bollegala D, D, Lewis O, Sloane R, Osanlou M. Social mediaMaskell S, Pirmohamed and pharmacovigilance: Analysis prescribing: in off-label evidence J Med Internet PatientsLikeMe. from of data Res PMID:21252034 2011;13(1):e6. Nautral Language Processing (IJCNLP), 2013, 2013, (IJCNLP), Processing Language Nautral Japan. Nagoya, humor From D. Buscaldi Rosso A, P, Reyes detection: irony to recognition Petersen C. 2015. Concrete Examples of Social Examples Concrete C. 2015. Petersen Impact https://www. Pharma. on Listening’s pm360online.com/concrete-examples-of- social-listenings-impact-on-pharma/. Archived http://www.webcitation.org/6q8ERpEcP at: MacKinlay M, A, Lui P, Cook T, Baldwin how text, media social noisy L. How Wang Proceedings sources? media social diffrnt of Med PMID:23450374 2013;52(2):152-159. consumers online Can D. Cohen Hughes S, drug knowledge? to contribute of consumer-generated methods comparison psychotropic controlled and professionally on informations medication Wu H, Fang H, Stanhope S. Exploiting Online Online Exploiting S. Stanhope H, H, Fang Wu Unrecognized Discover to Discussions Methods Effects. Drug Side Inf Internet Res PMID:21807607 2011;13(3):e53. the PMID:26147850 2015;80(4):910-20. Pharmacol

43. 42. 41. 40. 39. 38. 37. 36.

CHAPTER

Social Media as a Tool for Assessing Patient Perspectives on Quality of Life in 9 Metastatic Melanoma: a feasibility study

A. Makady1,2*, R. Kalf1,2*, B. Ryll3,4, G. Spurrier3, A. de Boer2, H. Hillege5, O. Klungel2; W. Goettsch1,2 (on behalf of GetReal Work Package 1).

1 The National Healthcare Institute (ZIN), Diemen, the Netherlands. 2 Department of Pharmacoepidemiology and Clinical Pharmacology, Utrecht University, Utrecht, the Netherlands. 3 Melanoma Patient Network Europe, Uppsala, Sweden 4 Uppsala University, Uppsala, Sweden. 5 Department of Epidemiology, University Medical Centre Groningen, Groningen, the Netherlands.

*R. Kalf and A. Makady contributed equally to this work

Currently under review in Health and Quality of Life Outcomes. feasibility feasibility the survey. Patients indicated that family, family, that indicated Patients survey. three most importantthree aspects of HRQoL. more positive tone. tone. positive more the a the assessment of such drugs. This study explores explores study This such drugs. of assessment valuable tool in assessing patient perspectives regarding HRQoL. HRQoL. perspectives regarding in assessing patient tool valuable the a three most important aspects of HRQoL for patients. Respondents importantmost Respondents three aspects patients. HRQoLof for the normal life, and enjoying life were were life enjoying and normal life, a

survey was distributed on social media channels of Melanoma Patient Network Europe to to Network Europe survey of Melanoma channels media on social distributed Patient was

ABSTRACT Social media may provide provide Social media may having having Conclusions Additionally, perspectives. carer and patient between to emerge seem differences However, fully with correlate to do not seem HRQoL and melanoma-specific questionnaires cancer- perspectives. patient Results completed and 17 carers 72 patients In total, Methods A independently researchers Two HRQoL. perspectives regarding patients’ assess melanoma to compared subsequently were which key themes, identify to analysis content conducted and Research for Organisation European (e.g. HRQoL questionnaires three from questions Assessment Functional and Melanoma (EORTC-QLQ-MEL38) Module for of Cancer Treatment Melanoma - (FACT-M)). Treatment of Cancer Background gains Incremental is occurring melanoma rapidly. for drugs of innovative Development in clinical to measure difficult be products may survival innovative amongst in overall Health Therefore, profiles. toxicity with increased be associated their use may and trials, on Health-Related information require agencies increasingly (HTA) Assessment Technology (HRQoL)Quality for of Life seem to find some questions from HRQoL questionnaires relevant (e.g. ‘Have you felt able able felt you ‘Have (e.g. relevant HRQoL questionnaires from questions find some to seem had swelling you ‘Have carryto (e.g. normal?’) as with things on less relevant others and and HRQoL patients between differ may wording Additionally, site?’). melanoma near your use generally patients whereby questionnaires, Carers indicated that being capable, having manageable adverse events, and being and being events, adverse manageable having capable, being that indicated Carers were pain-free of social media use to assess patient perspectives on HRQoL in melanoma, and whether perspectives on HRQoL whether in melanoma, and patient assess to of social media use these perspectives. represent HRQoL and melanoma-specific questionnaires cancer- current

Novel sources for generating RWE 9 168 Novel sources for generating RWE 9 169 field of field of context properties the the the means to assess assess means to a implementation of health health of implementation the routine setting of health care care setting of health routine extent to which an intervention an intervention which to extent the aim of informing decision-making of informing aim collection of HRQoL data for HTA HTA collection HRQoL of for data the the the general prolongation of overall survival overall of prolongation general systematic evaluation of evaluation systematic the the past 5 years (3;4). One positive consequence of of consequence (3;4). One positive past 5 years the prolonged periodsof survival prolonged as development of innovative yet expensive therapeutic therapeutic expensive yet of innovative development the the toxicity profiles associated with their use may be considerable, considerable, be with their use may associated profiles toxicity third of HTA submissions for oncologic treatments in 6 different in 6 different treatments oncologic for submissions of HTA third rapid pace. An illustrative example can be provided in in can be provided example illustrative An pace. rapid a a the particular disease (e.g. metastatic melanoma), thus possibly identifying identifying thus possibly melanoma), metastatic particular disease (e.g. a field of oncology, field of oncology, desired results and when provided under and when provided results desired the the added value of innovative products within relative effectiveness assessments (REA) (REA) assessments effectiveness products within relative of innovative added value increased number of treatments has been has treatments of number increased Conventionally, HRQoL of patients is measured using validated questionnaires which which questionnaires validated using measured is HRQoL of patients Conventionally, encouraging guidance HTA despite However, Within Within number of reasons, including its sheer scarcity (5). Moreover, it can be argued that available available that be argued it can (5). Moreover, sheer scarcity its including of reasons, number INTRODUCTION Health Technology Assessment (HTA) entails entails (HTA) Assessment Technology Health drugs is occurring at drugs is occurring at can either be generic (e.g. EuroQol 5 Dimensions; EQ-5D) (8), disease-specific (e.g. European European EQ-5D) Dimensions; 5 (8), disease-specific EuroQol (e.g. generic be can either (e.g. (EORTC- Questionnaire Quality of Life of Cancer Treatment and Research for Organization or (9;10) Melanoma- (FACT-M)) Treatment Cancer of Assessment Functional or QLQ-30) using HRQoL acquired perspective, an HTA From (11). measures include individualised even disease different gains across healthcare of comparison enable can generic questionnaires through acquired pulmonary data chronic Meanwhile, diseases). vs. oncology (e.g. areas disease-specificat HRQoL between distinguish experienced to can help questionnaires of stages different generic whether or disease-specific, HRQoL do not accurately questionnaires, validated the (5-7). several in HTA to HRQoL can contribute data Therefore, per disease stage. need medical or data effectiveness primary as whether relative or secondaryways, for outcomes health (5-7). treatments new oncologic of analyses utilityused in cost-utility values for as sources HRQoL that shown has research Recent in submissions. it is seldom included submissions, only in features data jurisdictions its impact that and European on decision-making due to low generally is a and effects of health technologies, addressing their direct and intended effects, as well as well as effects, their direct and intended addressing technologies, of health and effects with consequences indirecttheir unintended and or cost-effectiveness effectiveness, relative their as such practice clinical in technologies is defined as effectiveness Relative (1). use appropriate for interventions alternative or more one to than harm, good compared when more does achieving practice (2). practice (2). of action new mechanisms numerous 7 drugs exhibiting melanoma whereby metastatic gained market in authorisation have the HTA agencies increasingly require information on Health-Related Quality of Life (HRQoL) (HRQoL) Quality on Health-Related Life of information require increasingly agencies HTA during patients by experienced of clinical trials, and trials, of clinical (1). Such HTA assessments can encompass several aspects of several can encompass assessments HTA Such (1). of metastatic melanoma patients (3;4). However, provided that incremental gains in overall in overall gains incremental that provided However, (3;4). patients melanoma of metastatic in to measure be difficult products may survival innovative with associated robust robust private a series of series of use social use social a the the case study on on study case the new source of RWE new source 3-year public-private public-private 3-year a science of using social social of using science a MPNE mailing list and by and by list MPNE mailing diagnosis of melanoma on on of melanoma diagnosis the a the generally low completion rates rates completion low generally website of MPNE. Respondents Respondents of MPNE. website GetReal consortium, EORTC-QLQ-C30 questionnaire to to questionnaire EORTC-QLQ-C30 survey on posted was the the the the the web-based survey. An announcement with with announcement An web-based survey. melanoma patient population accessible via via accessible population patient melanoma a melanoma patient. patient. melanoma survey on single e-mailsingle to a the a the potential use of social media as use of social media potential pilot literature review (12). This research demonstrated demonstrated research This (12). review literature pilot study if they self-reported a the the use of Real-World Evidence (RWE) in early drug development (RWE) earlyEvidence in development drug Real-World of use survey to goals and link comparability of comparability feasibility study, aiming to advance advance to aiming study, feasibility a the the relevance of questions from from of questions relevance the general melanoma population, evaluate what melanoma patients and and patients melanoma what evaluate population, melanoma general the the needs of patients, possibly contributing to to contributing possibly patients, of needs announcement and link to link to and announcement tool to gather melanoma patients´ perspectives on HRQoL. More specifically, perspectives on HRQoL. specifically, More melanoma patients´ gather to tool the a web-based survey Socio-demographic 2). Appendix (see items 25 included the Innovative Medicines Initiative (IMI)-GetReal Medicines Initiative is Innovative initiative What is HRQoL in melanoma for you? you? HRQoL is melanoma for in What potential value of using social media to inform several parameters of HTA in oncology, in oncology, of HTA parameters several inform to media of using social value potential online surveyonline of be carer reported or to

Building upon results from this pilot review, this article explore to aims pilot review, this from upon results Building The The The The • brief description of

METHODS this article assess will: media as social media with social media with carers perceive as important in relation to HRQoL and compare this to validated cancer- validated to this HRQoL as important and compare to relation in perceive carers melanoma-specific HRQoL current and assess whether HRQoLspecific questionnaires, perspectives on HRQoL. It is important to melanoma patients’ represent questionnaires this is that emphasize for HTA was investigated within within investigated was HTA for than conducting perspectives on HRQoL, rather on patients’ insights gain media to pre-definedcollected answer through to hypotheses based on data analysis quantitative media. social including: adverse events (12;16-18), treatment adherence (17) and HRQoL (17) and (12;19-21). adherence treatment (12;16-18), events adverse including: the of such questionnaires by patients (12-14). patients by questionnaires of such partnership exploring 1 of Package Work Within (15). assessment and drug is example One aim. this exploring conducted been have studies case melanoma whereby metastatic posting posting a Members of Melanoma Patient Network Europe (MPNE)(22), an established patient network network patient an established (MPNE)(22), Network Europe Members of Melanoma Patient media social via multiple approached were their carers, and patients melanoma for complete anonymously to MPNE of channels represent represent MPNE Facebook group, MPNE LinkedIn group, and MPNE twitter account. Members of Members of and MPNE twitter account. MPNE LinkedIn group, group, MPNE Facebook sending by also approached MPNE were the were eligible for inclusion in in inclusion for eligible were and clinical characteristics were collected, including gender, country of residence, age, age, country of residence, gender, including collected, and clinical characteristics were treatments and of disease, stage melanoma diagnosis, since years level, educational which 1) assessing by collected perspectives on HRQoL were and carer Patient received. HRQoL patients’ melanoma regarding aspects most important or carers are patients to and 2) assessing perspectives on HRQoL Survey included: or carer on patient questions respondents.

Novel sources for generating RWE 9 170 Novel sources for generating RWE 9 171

survey EORTC EORTC general general patients’ patients’ relevance relevance responses responses the the the relevance of of relevance the the 2 respondents 2 respondents the the The The analysis as patients only. All All only. patients as analysis melanoma patient’s HRQoL HRQoL patient’s melanoma analyses. analyses. 30 day period. By completing By completing period. 30 day member clicked on clicked member grouping of similar codes were were codes similar of grouping a the the the the the qualitative analysis was performed was analysis qualitative a melanoma patient’s HRQoL today? HRQoL today? patient’s melanoma most important aspect to survey(26). Content analysis allows for for allows analysis survey(26). Content survey was open for 30 days from January from 30 days survey for open was the content analysis to assess to analysis content total frequency with which generated codes codes frequencytotal with which generated the the The The socio-demographic variables (e.g. gender, age, age, gender, socio-demographic(e.g. variables the the the web-based survey. Of these 70 indicated to be patients, be patients, Of to 70 indicated these web-based survey. basis of the ongoing survey were posted on MPNE’s private Facebook Facebook private on MPNE’s posted surveyongoing were the the separate screen. screen. separate a EORTC-QLQ-C30. Furthermore, Furthermore, EORTC-QLQ-C30. melanoma patient, 2 indicated to be both patients and carers, and 7 7 and carers, and patients both be to indicated 2 patient, melanoma the a single thing that would improve your/ your/ improve would that thing single extent to which current cancer-specific HRQoL questionnaires represent represent HRQoL questionnaires cancer-specific current which to extent 3 things that deteriorate your/ your/ deteriorate that 3 things comparability and generalizability of our study population to to and generalizability of our study population comparability the the the the open-ended in posed questions Name Name Name right now? now? right organisation and cataloguing of respondent’s descriptions of key aspects regarding descriptions of key aspects regarding of respondent’s and cataloguing organisation survey was conducted using Survey Monkey, and once and once survey Survey using conducted was Monkey, EORTC Module for Melanoma (EORTC-MEL-38) and FACT-M. FACT-M. and Melanoma(EORTC-MEL-38) Module for EORTC survey, respondents gave their informed consent for this study. study. this for consent informed their gave respondents survey, the Subgroup analyses were performed for patients and carers separately and were were and separately and carers performed patients for were analyses Subgroup To assess To • In order to evaluate what patients and carers regard important for HRQoL, two important HRQoL, two for regard carers and patients what evaluate to Inorder • 2016. Two reminders of of reminders Two 2016. wordcloud was created on created was wordcloud total of 96 respondents filled in 96 respondents of total th RESULTS ANALYSIS stratified by stage when possible. All data were coded, stored and analyzed using R using analyzed and stored coded, were All data when possible. by stage stratified of questions from of questions from namely respondents, to HRQoL melanoma-specific two in questions questionnaires the (27). 4.00.03.05 version HRQoL (Figure 1). 1). HRQoL (Figure melanoma patients’ perspectives on HRQoL, respondents were asked about asked about were perspectivesHRQoL, on respondents patients’ melanoma were cited (either by patients or carers) as or carers) patients (either by cited were aspects on key based during identified group, LinkedIn group, and Twitter account throughout throughout account Twitter and LinkedIn group, group, the A To assess To melanoma population, we compared compared we population, melanoma reviewed by both researchers and any discrepancies in coding were resolved by consensus. consensus. by resolved were coding in discrepancies and any both researchers by reviewed A melanoma patient views on HRQoL. Assigned codes and codes on HRQoL. Assigned views patient melanoma the did not report either and were therefore excluded from from excluded therefore did not report and were either 17 were carers of of carers 17 were and educational level) with values reported by Bay et al., Eriksson et al, and Eriksson et al, et al., Bay reported values with by level) and educational researchers independently performed inductive content analysis on analysis performed independently inductivecontent researchers reference values (23-25). (23-25). values reference to link it was presented on on presented link it was 8 The The indicating to be both patients and carers were included in included were carers and patients both be to indicating type type lesser lesser MPNE MPNE a the paper by by paper HRQoL in HRQoL in the carers who who carers paper of Bay of Bay paper the the BRAF mutation BRAFmutation carers indicated indicated carers the The The survey indicated survey indicated a the high education (e.g. (e.g. education high the a age distribution found age distribution found distribution of age in our distributionour in age of the The The distribution of age was similar similar distribution age was of MPNE website (9%) or (9%) MPNE website survey represented all stages of stages of all survey represented total of 65% of total the the A the study population are shown in Table Table in shown are study population United KingdomUnited (50%). respondents indicated to be between 55 55 be between to indicated respondents treatment most often received by patients patients by most often received treatment educational level reported in level educational the the the survey via the general melanoma population. melanoma population. general carers who responded to to who responded carers the patient who had stage IV melanoma (71%), while while (71%), who had stage IV melanoma patient patient with stage II melanoma. More than 60% of than 60% of More stage II melanoma. with patient the the a the a survey via Facebook (77%), Twitter was used to to used was Twitter (77%), survey Facebook via the tumour. Surgery was tumour. the survey, most cared for for most cared survey, EORTC as reference values for melanoma patients (Table 3) (24). Both Both 3) (24). (Table patients melanoma for values as reference EORTC patient who had been diagnosed more than 2 years ago. Most patients with with Most patients ago. 2 years than more who had been diagnosed patient a the the EORTC reference values also showed that that also showed values reference EORTC Berlin Support Group, Melanoma Romania Association, and Dutch Melanoma Melanoma and Dutch Melanoma RomaniaBerlin Association, Support Group, carers indicated to care for for care to indicated carers patient had been diagnosed with cutaneous melanoma. Cutaneous melanoma had had melanoma Cutaneous melanoma. with cutaneous been diagnosed had patient socio-demographic characteristics of the The The the overall HRQoL values reported by our study population are comparable to that that to comparable are HRQoL our study population reported values by overall small number of respondents. small number the

patients with stage II, III, and IV melanoma indicated to have been diagnosed more than than more diagnosed been have to stage II, III, and IV melanoma indicated with patients The The Table 2 shows that patients who responded to to who responded patients that 2 shows Table The The Most respondents accessed accessed Most respondents the

college degree or higher), our study sample was more highly educated with 64% having having 64% with highly educated more study sample was our higher), or degree college stratified and non-stratified overall HRQoL values were similar, indicating that that indicating similar, were values HRQoL overall and non-stratified stratified our study population is similar to that in that to is similar population our study been diagnosed in 44% of patients with stage I melanoma, while 50% had been diagnosed diagnosed been had 50% while melanoma, I stage with patients of 44% in diagnosed been know didn’t patients these of 6% and melanoma, or choroidal uveal with ocular, of melanoma they had been diagnosed with. Most patients with stage I, II, or III melanoma I, II, or III melanoma with stage with. Most patients diagnosed had been they of melanoma 46% to compared tumour, their in present mutations any with unfamiliar be to indicated that indicated who carers of 53% and melanoma IV stage with patients of in found had been in this study. and carers patients 92% of to according reported by none of the I. stage with 44% of patients to compared ago, 2 years to take care of of take care to as as well IV melanoma II, III and stage that that an university degree or higher (23). or higher degree an university a melanoma, and that most patients had stage IV melanoma (39%). Of IV melanoma (39%). stage had most patients that and melanoma, to responded Eriksson et al., where only 25% of melanoma patients in Sweden had in Sweden patients melanoma only 25% of where Eriksson et al., et al. showed that approximately 50% of patients with melanoma in Denmark were female female in Denmark with melanoma were patients of 50% approximately that showed et al. female were population in our study of patients 74% that shown was it stratified when (25); female. were population study our in carers of 53% whereas and patients, carers between stages of melanoma and most for similar was study population of half III where with stage patients for except and 64. in our study population (24). Compared to to (24). Compared in our study population Association Forum. Forum. Association mailing list (1%). Finally, 11% of respondents indicated to have used other online channels, online channels, other used have to indicated respondents of 11% (1%). Finally, list mailing such as university were 64 (82%), 35 and between (70%), female mostly were 1. Respondents from originated or higher (64%), and graduates to comparable and were & IV, I & II and stage III stage between analyses were stratified by stage for patients, however this was not possible for carers due due for carers possible not was this however patients, for by stage stratified were analyses to extent (2%). Some respondents accessed accessed Some (2%). extent respondents

Novel sources for generating RWE 9 172

Novel sources for generating RWE 9 173

10% 17% Higher degree or doctorate or degree Higher 24% 21% 6%

60% 50% Graduated with university degree level degree university with Graduated 35% 50% 38%

31% 10% 22% Graduated from college from Graduated 29% 21% 25% High education High

25% 20% 11% Graduated from secondary school secondary from Graduated 25% 25% 12% 7% 39% education

- - - Finished school after primary school primary after school Finished 39% 39% Intermediate Intermediate - - 36%

- - - Did not attend school attend not Did 36% 36% Low education† Low - - Patients

Highest educational level: educational Highest Male Female All All

0% - 4% - - 6% >75 0%

1% 80+ 6% 7% - 10% 6% 65-74 6%

5% 70-79 24% 18% 50% 30% 33% 55-64 18%

22% 60-69 29% 36% 25% 30% 28% 45-54 23%

24% 50-59 24% 29% 13% 20% 22% 35-44 26%

23% 40-49 12% 4% 13% 10% 6% 25-34 27%

25% <40 6% 4% - - - 18-24 III&IV

Stage Stage Stage I&II Stage Age: Age:

Female Female 83% 70% 75% 68% 53% Female 57% 48% 44% 42%

Male 17% 30% 25% 32% 47% Male 43% 52% 56% 58%

Gender: Gender:

(n=28) Stage I Stage Stage II Stage Stage III Stage Stage IV Stage (n=16) (n=10) (n=18) (n=17)

Stage IV IV Stage Stage III III Stage Stage II II Stage Stage I I Stage Carers Carers

Reference Reference Patient Population* Patient (n=72) Patients

study population study the of characteristics Socio-demographic - 1 Table

USA and 1 respondent from Serbia. Serbia. from respondent 1 and USA the from originated

reference it is given that in Sweden low education corresponds to mandatory school, intermediate to high school, and high to college/university; ‡5 respondents respondents ‡5 college/university; to high and school, high to intermediate school, mandatory to corresponds education low Sweden in that given is it reference

reference patient population was based on (25) for gender, (24) for age, and (23) for educational level; - no respondents ticked this answer (e.g. 0%); †In this this †In 0%); (e.g. answer this ticked respondents no - level; educational for (23) and age, for (24) gender, for (25) on based was population patient reference The *

41% 50% 50% 60% 50% UK UK

18% 7% - 10% 11% Romania

6% 4% 6% - 17% Other‡

- 7% 13% 20% 6% Norway

24% 7% 6% - - Netherlands

- - 6% - - Italy

- 11% 6% - - Ireland

- - 13% 10% 17% Germany

6% 4% - - - France

6% 11% - - - Belgium

Country of Residence: Residence: of Country

(n=28) Stage I Stage Stage II Stage Stage III Stage Stage IV Stage (n=16) (n=10) (n=18) (n=17)

Stage IV IV Stage Stage III III Stage Stage II II Stage Stage I I Stage Carers Carers

Reference Reference Patient Population* Patient (n=72) Patients

- continued - 1 Table

Novel sources for generating RWE 9 174 Novel sources for generating RWE 9 175 ₸ Carers (n=17) Carers 53% 18% - 6% - 18% 6% - 100% 26% 21% 56% 50% 6% - 24% 71% - - - 6% 29% 24% 41% 62% 12% 6% - 19% total number of respondents on on of respondents number total 46% 18% 7% 4% 4% 11% 4% 7% 89% 39% 25% 81% 27% Stage IV IV Stage (n=28) NA NA NA 39% - - 4% 11% 25% 43% 18% 64% 18% - - 18% The The 27% - - - 7% 67% - - 94% 13% 6% 25% 6% Stage III III Stage (n=16)* NA NA 22% NA - - 7% 13% 7% 40% 33% 57% 7% - - 36% total number of respondents on type of melanoma is 16, on type 16, is of melanoma number of respondents total Patients (n=72) Patients The The

₸ - - - 60% 10% 10% 90% 20% 11% - - Stage II II Stage (n=10)+ NA 14% NA NA - - - 10% 20% 50% 20% 70% 10% - - 20% 20% - patient they care(d) for; ‡Other melanoma mutations mentioned by 4 4 ‡Other by for; mentioned mutations melanoma care(d) they patient the 89% 39% - - - Stage I I Stage (n=18) 25% NA NA NA - 6% 17% 28% 6% 22% 22% 44% 50% - - 6% 11% - - - - 78% 6% 6% - Clinical characteristics of the study population total number of respondents on treatments received (chemotherapy) was 9; * was (chemotherapy) received on treatments of respondents number total The The Treatments received Treatments Surgery Radiotherapy Chemotherapy Immune Therapies Therapies Targeted + melanoma diagnosis is 15, on type of melanoma is 14, melanoma mutation is 15, and on treatments received is 15; † Carers † Carers is 15; received on type is 15, on treatments and is 15, diagnosis melanoma melanoma mutation of melanoma is 14, for characteristicsspecific disease provided on treatments received is 16; - no respondents ticked this answer (e.g. 0%); NA: Not Applicable NA: Not 0%); (e.g. this answer ticked respondents - no is 16; received on treatments respondents were mutations in chromosome 3, 6 and/or 8; in chromosome mutations were respondents Stage of MelanomaStage 1 Stage 2 Stage 3 Stage 4 Stage Melanoma diagnosis: ago month < 1 ago months 1-3 ago months 3-6 ago months 6-12 ago years 1-2 ago years 2-5 ago years > 5 Type of Melanoma: Type melanoma Cutaneous Choroidal Ocular/ Uveal/ melanoma melanoma Acral melanoma Mucosal know I don’t Melanoma mutations: Melanoma mutations: BRAF mutant BRAF wild-type NRASmutant c-kit mutant GNAQ/GNA11 know I don’t None Other‡ Table 2 7 25% 20% 20% 18% 22% 16% single most most single 6 36% 31% 25% 26% 29% 27% the 5 21% 28% 27% 30% 25% 28% EORTC reference value for for value reference EORTC the 4 11% 16% 11% 18% 11% 17% normal life while being ill. being while normal life a Overall quality life of Overall 3 7% 6% 11% 6% 10% 8% patients’ HRQoL. ‘Family’ was was ‘Family’ HRQoL. patients’ 2 0% 1% 2% 2% 1% 2% the 1 0% 0% 2% 2% 1% 2% study population compared to the second most important aspect in HRQoL was ‘Normal Life’, ‘Normal Life’, important most second aspect in HRQoL was most important aspects mentioned by patients and carers in importantmost in carers and aspects patients by mentioned the the - Key aspects- Key important find patients in QoL - Overall quality of life in

Figure 1 shows shows 1 Figure

Respondents: Stage I & II (n=28) Stage Respondents: EORTC Reference Value: Stage I & II Stage Value: Reference EORTC Respondents: Stage III & IV (n=44) Stage Respondents: EORTC Reference Value: Stage III & IV Stage Value: Reference EORTC Respondents: All patients (n=72) patients All Respondents: EORTC Reference Value: All Stages All Value: Reference EORTC Table 3 the EORTC QLQ-C30 our study sample that are important are that sample study our to important aspect while important it highly find patients to lead that implying Figure 1 Figure

Novel sources for generating RWE 9 176 Novel sources for generating RWE 9 177 patients’ patients’ a relevance of of relevance EORTC-QLQ-C30 EORTC-QLQ-C30 the Uncertainty‡ Carers (n=17) Carers Capability* No AEs* free* Pain effectiveness† Drug Family† Normallife† medicines‡ to Access Cure‡ Finances‡ Good care‡ Good health‡ the same number of respondents same number of respondents ₸ most important aspects The The the relevance of several questions questions of several relevance top 10 most important 10 top aspects different perspective regarding regarding perspective different ₸ a ₸ question in question the the most important aspects to patients’ most important aspects patients’ to relevant question. Another example example Another question. relevant a the different disease stage found different different found stage disease different the Pain free Pain Stage IV (n=28) Stage Family* Good medicines* Normallife* Capability† life† Enjoy Support† Good care Good health‡ Good information‡ medicines to Access Friends a same number of respondents reported this aspect be of respondents same number to second most important most second aspect in The The the Stage III (n=16) Stage Family* Worry* Normallife burden Therapy Counselling† life† Enjoy Good care‡ Good doctors‡ worry‡Not to free‡ Pain Patients (n=72) Patients same number of respondents reported aspect this be important of respondents same number to HRQoL in their EORTC-QLQ-C30 to HRQoL (see Table 5 and Appendix Table 1). It 1). can Table 5 and Appendix Table HRQoL (see to EORTC-QLQ-C30 The The

Stage II (n=10) Stage Family* Fear* life Enjoy Capability† Good doctors† Good health† Normallife† free† Pain Relapse† Worry† ₸ majority of patients with stage I and II melanoma, while approximately majority while approximately II melanoma, I and with stage of patients survey respondents were asked to rate rate asked to were survey respondents the more detailed analysis is shown whereby whereby shown is analysis detailed more ₸ the the a ₸ ₸ ₸ ₸ ₸ - Top 10 aspects 10 patients - Top and carers indicate that are important in patients’ HRQoL ₸ same number of respondents reported aspect this be importantsame number of respondents to in their HRQoL; † As partAs of In Table 4 In Table The The important HRQoL; in their Positive mood Positive No anxiety network Patient Work Stage I (n=18) Stage Family* Good care* Finances† Normallife† Support† Life† Enjoy medicines‡ to Access Fear‡ Good doctors‡ Capability Friends Good health * reported aspect this be important to their HRQoL; in ‡ Table 4 HRQoL. According to carers ‘Family’ was was ‘Family’ carers to HRQoL. According have may carers and patients that HRQoL, which showed HRQoL. most important is what in patients’ be seen that in our study sample patients with with patients sample our study in be seen that originating from from originating in their HRQoL, together with ‘Good Care’ by patients with stage I melanoma, ‘Fear’ by by ‘Fear’ with stage I melanoma, patients by Care’ ‘Good with HRQoL,in their together ‘Good stage III, and with patients melanoma by ‘Worry’ II melanoma, stage with patients that indicate stage IV melanoma. Carers with patients by ‘Normal Life’ and medicines’ were free’ ‘Pain and Events’, ‘No Adverse ‘Capability’, on HRQoL were stratified by disease stage for patients, and for patients and carers separately. separately. and carers for patients and for patients, stage by disease stratified on HRQoL were of as one ‘Family’ rated It themselves patients seen that be can rate to seemed also HRQoL, their carers and that to relevant questions questions differently than patients. For example, example, For than patients. differently questions did all) or (at be relevant to not seem did activities’ doing strenuous ‘Trouble regarding to not apply 50% of stage III and IV melanoma patients found this found patients III and IV melanoma 50% of stage

89 the survey via melanoma melanoma survey was survey was the the short period of the a MPNE website or other online or other online MPNE website the means to collect patient and carer and carer collect patient means to a EORTC-QLQ-MEL38 and FACT-M were were and FACT-M EORTC-QLQ-MEL38 relevance of questions posed in EORTC in EORTC posed questions of relevance 30 days during which 30 days the the the HRQoL questionnaires seemed to be aspects to seemed HRQoL questionnaires majority of respondents accessed accessed majority respondents of the The The general melanoma population in some aspects (e.g. aspects some in population melanoma (e.g. general the number one most important aspect, whereas for carers one most importantnumber carers for aspect, whereas quick and time-efficientquick data valuable to assemble manner a the second most important. Some aspects among patients were most important.second Some aspects were patients among the EORTC-QLQ-MEL38 focused on patients being given enough time time enough being given on patients focused EORTC-QLQ-MEL38 treatment options available. However, patients seemed to be more more be seemed to patients However, options available. treatment feasibility of using social media as using of feasibility the smaller proportion doing so via Twitter, Twitter, proportionsmaller so via doing question ‘Have you had pain?’ was rated as not relevant or does not apply apply does not or as not relevant rated was pain?’ had you ‘Have question the a the the majority of stage II melanoma patients, while more than 50% of stage III melanoma III melanoma of stage 50% than more while majority melanoma patients, of stage II melanoma site. melanoma

Table 6 indicates that some questions in questions in some that indicates 6 Table Social media may provide provide Social media may Based on responses compiled, patients with different stages of melanoma and carers of melanoma and carers stages different with patients compiled, Based on responses the

DISCUSSION respondents discussed pain, 33 respondents focused more on being pain free as important as free pain on being more focused 33 respondents pain, discussed respondents HRQoL. their for Facebook, with Facebook, In this study, In this study, site, surgical site or headaches posed in posed or headaches site surgical site, interested in discussing access to adequate and clear information on treatment options. options. on treatment information clear and adequate to access in discussing interested how from differ may HRQoLin posed questions of questionnaires wording Additionally, at pain questions regarding example, For these questions. interpret patients relevant to our study population, while other questions seemed less relevant. For example, example, For less relevant. while other questions seemed our study population, to relevant in one question about think to pain in discussed respondents on. Instead, did not focus study population our of pain that of 14 while Additionally, pain). pains or experiencing future (e.g. terms general more showed that that showed to question as this rated 60% of carers Also as (very) this question relevant. rated patients (very) relevant. on patient perspectives on HRQoL. Responses can be gathered within within perspectives on HRQoL. Responseson patient can be gathered QLQ-C30 differently, such as questions relating to trouble doing strenuous activities strenuous doing or to trouble relating such as questions differently, QLQ-C30 in questions on received responses of analysis qualitative Moreover, pain. of occurrence some that revealed questionnaires FACT-M and QLQ-MEL38 EORTC melanoma-specific relevant of less Examples population. our study to relevant others less and relevant were as those on pain near well as options, ponder treatment include those on time to questions the clinical trials (RCTs) controlled randomised in included usually not audiences from time and stage-specific, such as “Fear” and “Good medicines” for stages II and IV, respectively. respectively. II and IV, for stages “Good medicines” and “Fear” stage-specific, such as rated also stages and carers of different Patients this was cited as one of as one cited this was identify and rank aspects important to HRQoL differently. For example, for all melanoma for all melanoma example, For aspects and rank identify important HRQoL differently. to one of was “Family” patients posted, 89 full responses were received. received. were 89 full responses posted, distribution, gender HRQoL) (e.g. but not others stage distribution,melanoma average spread). geographic level, educational channels. Respondents resembled resembled Respondents channels. perspectives on HRQoL was explored. Within Within perspectives on HRQoL explored. was

Novel sources for generating RWE 9 178

Novel sources for generating RWE 9 179

26 21 11 11 21 11 Carers (n=19) Carers

18 11 14 11 21 25 Stage IV (n=28) IV Stage

19 6 31 6 12 25 Stage III (n=16) III Stage

30 - 10 - 10 50 Stage II (n=10) II Stage day the during chair a

41 - 6 12 24 18 Stage I (n=17) I Stage Need to stay in bed or or bed in stay to Need

28 22 17 - 17 17 Carers (n=19) Carers

21 7 14 14 25 18 Stage IV (n=28) IV Stage

19 12 19 6 19 25 Stage III (n=16) III Stage

30 10 - - 20 40 Stage II (n=10) II Stage house the outside walk

33 17 - - 22 28 short short a taking Trouble Stage I (n=18) I Stage

26 21 21 21 5 5 Carers (n=19) Carers

11 18 29 18 7 18 Stage IV (n=28) IV Stage

19 38 6 12 6 19 Stage III (n=16) III Stage

30 - - - 30 40 Stage II (n=10) II Stage long walk long a

29 24 6 6 24 12 Stage I (n=17) I Stage taking Trouble

21 32 11 21 5 11 Carers (n=19) Carers

11 18 29 18 7 18 Stage IV (n=28) IV Stage

12 31 25 - 12 19 Stage III (n=16) III Stage

30 - - 20 10 40 Stage II (n=10) II Stage strenuous activities strenuous

29 12 12 6 24 18 Trouble doing doing Trouble

Stage I (n=17) I Stage

EORTC QLQ-C30: EORTC Not relevant at all at relevant Not Not relevant Not Neutral Relevant Very relevant Very Does not apply to me to apply not Does

Question in in Question

Relevance

EORTC QLQ-C30 questionnaire in our study population ( population study our in questionnaire QLQ-C30 EORTC the from questions of Relevance - 5 Table subsample). a

11 50 11 11 - 17 Carers (n=19) Carers

18 11 21 11 4 36 Stage IV (n=28) IV Stage

6 19 38 6 6 25 Stage III (n=16) III Stage

30 - 10 - 10 50 Stage II (n=10) II Stage

38 6 19 12 12 12 Stage I (n=16) I Stage Pain

Carers (n=19) Carers 33 22 - 11 11 22

Stage IV (n=28) IV Stage 18 7 29 - 4 43

Stage III (n=16) III Stage 19 19 12 12 6 31

Stage II (n=10) II Stage 40 - - 10 10 40

Stage I (n=17) I Stage breath of Short 35 12 6 - 29 18

21 26 32 11 5 5 Carers (n=19) Carers

11 26 33 7 7 15 Stage IV (n=27) IV Stage

20 7 53 - 7 13 Stage III (n=15) III Stage

leisure time activities time leisure

20 - 10 10 10 50 Stage II (n=10) II Stage your hobbies or other other or hobbies your

18 35 18 12 12 6 Stage I (n=17) I Stage Limitations in pursuing pursuing in Limitations

16 26 21 26 - 11 Carers (n=19) Carers

11 32 25 7 7 18 Stage IV (n=28) IV Stage

6 25 31 12 6 19 Stage III (n=16) III Stage

other daily activities daily other

30 - 10 - - 60 Stage II (n=10) II Stage either your work or or work your either

24 18 18 6 18 18 Stage I (n=17) I Stage Limitations in doing doing in Limitations

26 21 11 - 16 26 Carers (n=19) Carers

29 7 4 4 7 50 Stage IV (n=28) IV Stage

toilet the

25 - 25 - 6 44 Stage III (n=16) III Stage

yourself or using using or yourself

30 - - - - 70 Stage II (n=10) II Stage dressing, washing washing dressing,

47 6 - 12 18 18 Stage I (n=17) I Stage Need help with eating, eating, with help Need

EORTC QLQ-C30: EORTC Not relevant at all at relevant Not Not relevant Not Neutral Relevant Very relevant Very Does not apply to me to apply not Does

Question in in Question

Relevance

- continued - 5 Table

Novel sources for generating RWE 9 180

Novel sources for generating RWE 9 181

‘Family and friends support.’ friends and ‘Family

treatment.’ the of step every through

‘Being surrounded by people who support you you support who people by surrounded ‘Being similar Wording Relevant family my from support emotional get I FACT-M

‘Doctors who don’t take your worries seriously.’ worries your take don’t who ‘Doctors

QLQ-MEL38 (friends, relatives and work).’ and relatives (friends, serious condition? condition? serious a is melanoma

‘Understanding how hard it is to live with cancer cancer with live to is it hard how ‘Understanding differ may Wording relevant Less EORTC EORTC Have you felt able to accept that that accept to able felt you Have

QLQ-MEL38 melanoma site? site? melanoma

NA NA relevant Less EORTC EORTC Have you had swelling near your your near swelling had you Have

‘Up to date knowledge of available treatments.’ available of knowledge date to ‘Up available to you? you? to available

QLQ-MEL38 with me.’ with treatment options options treatment the about think to

‘Having treatment options explained and discussed discussed and explained options treatment ‘Having differ may Wording relevant Less EORTC EORTC Have you been given enough time time enough given been you Have

‘Being able to live without pains.’ without live to able ‘Being QLQ-MEL38 near your melanoma site? site? melanoma your near

‘Worry and fears about future pain and mortality.’ and pain future about fears and ‘Worry differ may Wording Relevant EORTC EORTC Have you had problem with pain at or or at pain with problem had you Have

emotional needs.’ emotional

being informed according to my intellectual and and intellectual my to according informed being

‘Not being treated like like treated being ‘Not passive idiotic patient but but patient idiotic passive a your disease? disease? your

QLQ-MEL38 and knowledge.’ and of (spread) extent the about information

‘More facts and less fantasy. I could need statistics statistics need could I fantasy. less and facts ‘More differ may Wording Relevant

EORTC EORTC Have you received realistic and reliable reliable and realistic received you Have

‘Piece of mind that help is just at at just is help that mind of ‘Piece phone.’ a of end the be available if you needed it? it? needed you if available be

QLQ-MEL38 psychological support service would would service support psychological a personal network).’ personal

‘Care and mental support (professionals and and (professionals support mental and ‘Care similar Wording Relevant

EORTC Have you felt confident that that confident felt you Have

‘Ability to life my life as normal as possible.’ as normal as life my life to ‘Ability QLQ-MEL38 things as normal? normal? as things

Relevant Wording similar Wording ‘Wish to continue life as before.’ as life continue to ‘Wish EORTC EORTC Have you felt able to carry on with with on carry to able felt you Have

patient population patient in wording in Example of patient response patient of Example Questionnaire Question

Difference Difference

Relevance to to Relevance

patients find important in HRQoL in important find patients

- Examples of questions in current HRQoL questionnaires EORTC-QLQ-C30 and FACT-M compared to responses from our study population about what what about population study our from responses to compared FACT-M and EORTC-QLQ-C30 questionnaires HRQoL current in questions of Examples - 6 Table

NA, Not Applicable (e.g. respondents did not discuss anything regarding this question). this regarding anything discuss not did respondents (e.g. Applicable Not NA,

arm or leg or arm

NA NA relevant Less I have good range of movement in my my in movement of range good have I FACT-M

‘Being able to control drug side-effects.’ drug control to able ‘Being

from treatment.’ from treatment

‘I’m very anxious about potential side-effects side-effects potential about anxious very ‘I’m differ may Wording Relevant

I am bothered by side effects of of effects side by bothered am I FACT-M

‘Being able to exercise fully.’ exercise to able ‘Being

impatient because of fatigue.’ of because impatient

lack of energy of lack a have I FACT-M energy to play with my children not be be not children my with play to energy the ‘Have differ may Wording Relevant

about mets.’ about

constant worry and stress stress and worry constant the from free be ‘To

FACT-M I worry that my condition will get worse get will condition my that worry I Relevant Wording similar Wording ‘Worry every time it I have to go for my liver scan.‘ liver my for go to have I it time every ‘Worry

patient population patient in wording in Example of patient response patient of Example Questionnaire Question

Difference Difference

Relevance to to Relevance

- continued - 6 Table

Novel sources for generating RWE 9 182 Novel sources for generating RWE 9 183 use of use of option option United United broader broader marginal marginal a the the the the setting of RCTs of RCTs setting the geographic spread of of spread geographic formal setting in which in which setting formal The The the trusted environment of their of their environment trusted a relevance of questions posed in in posed of questions relevance the scope of information from from of information scope development of individualised HRQoL HRQoL of individualised development the the difference between what patients and carers and carers patients what between difference a impact HRQoLof by:drugs of REA on increasing question of which form of HRQoL questionnaires HRQoL of questionnaires form which of question the the incremental value of cancer- or melanoma-specific or melanoma-specific of cancer- value incremental dynamics of ongoing discussions. Such advantages of of advantages Such discussions. of ongoing dynamics efforts invested by stakeholders such as clinicians and such as clinicians and by stakeholders efforts invested the the varying relevance of questions posed in available cancer- available in posed questions of varying relevance the fact that even they may be unable to distinguish between between distinguish to be unable may they fact even that the the possible explanation for this phenomenon may be that HRQoL HRQoL that be may this phenomenon for explanation possible A future (35;36). future the establishment and conduct of patient/citizen panels at various stages of HTA HTA of stages various at panels conductand patient/citizen of establishment the EQ-5D, considering considering EQ-5D, HRQoL of patients with different disease stages. Provided that innovative, expensive expensive innovative, that Provided stages. disease different with HRQoL of patients Findings from this study illustrated study illustrated this from Findings Meanwhile, findings on findings Meanwhile, the patients reached through social media is considerable, ranging in this study from from this study in ranging is considerable, media social through reached patients collect Serbia media to social Romania. and Employing Norway, to of America (U.S.A.), States trials or point-of-care than multi-centre studies resources less require would such data cervical on Similarly, research cancer(21). previous in confirmed been and has collection data than similar efficient and more be less resource-intensive may social media through decision making. Moreover, data collection through social media allows patients patients allows social media collection data through making.decision Moreover, to provide HRQoL information at their own pace and within pace their own at HRQoL information provide to to hesitate participating may panels patients in patient/citizen In contrast, choice. own their HRQoL to due to particularshare related insights personal or due to conducted are panels collected. of responses candidness increasing and group patient reported important as been HRQoL. aspects regard have Similar findings may for HRQoL validated to and carers patients of responses exploring research in previous Despite questionnaires(30-32). social media use may help increase increase help may use media social widening HTA, of HRQoL for availability data questionnaires that it can thus be argued HRQoL questionnaires, develop to scientists on differences Moreover, and carers. patients across implementable equally be not may findings in Comparable stage. disease important aspects of HRQoL patients’ extended to discussions for enticed have research previous This questionnaires(13;33;34). raises (e.g. women or patients with early with of melanoma)(28;29). stages or patients women (e.g. HTA agencies should resort to within REA’s. Moreover, it raises doubts as to whether current current whether as to doubts raises it Moreover, resort should agencies within REA’s. to HTA with different patients between to distinguish HRQoL of sufficient are questionnaires In of melanoma. stages fact, on several in literature cited needs has been repeatedly patient HRQoL to comparison in areas(13;14;38). disease questionnaires for REA’s may be questionable when compared to more general tools such such tools general more to compared when questionable be may REA’s for questionnaires as or routine practice (5;37). Controversy regarding regarding Controversy practice (5;37). or routine the as as well melanoma), metastatic (i.e. stage III/IV patients at targeted drugs are profiles toxicity drugs and survivalinnovative amongst in overall gains incremental relative stage- necessary separate become develop to therefore it may use, with their associated HRQoL with gains delineate better to and carers for patients HRQoLspecific questionnaires in new treatments specific or melanoma-specific questionnaires to patients may provide insights as to why why to as insights provide may patients to questionnaires or melanoma-specific specific in whether low, remain HRQoL questionnaires for rates completion collection, collection, the interpretation of of interpretation mining of highly of highly mining survey, allowing allowing survey, potential of using of using potential survey, representing representing survey, the the authors. Additionally, Additionally, authors. the the mining of data already already of data mining AD Spectrum for Better Better AD Spectrum for survey to collect patient survey collect patient to the collection hypothesis- of a the the the the moment, little research exists on on exists research little moment, newly-developed questionnaires are are newly-developed questionnaires the physician- or scientific focus whereby whereby focus scientific or physician- a the other hand, it involves it involves other hand, latter approach, little intervention is applied by by applied is intervention little approach, latter medium for spreading spreading medium for the a the topic of ongoing research by by research of ongoing topic two approaches for using social media correlate or differ, differ, or correlate social media using for approaches two a subsequent irrelevance of certain questions, in combination of certain combination in questions, irrelevance subsequent the paucity of HRQoL data for purposes such as REA ensues. If If purposes such as REA ensues. paucity of HRQoL for data The The a “real world”. On world”. “real the opportunity important which aspects them in of HRQoL express to were to same effect (e.g. distribution of questionnaires via mailing lists). Meanwhile, lists). mailing via Meanwhile, distribution of questionnaires (e.g. same effect patient-centred approach which elicits thorough patient input at all stages stages all input at patient thorough elicits which approach patient-centred the a the emphasis is set on aspects such as reliability, validity, and cross-cultural relevance, cross-cultural and relevance, validity, set on aspects is emphasis reliability, as such reasons behind potential discrepancies in such results and thus and thus results in such discrepancies potential behind reasons researcher (e.g. through directed questions), implying that insights gained on HRQoL on HRQoL gained insights that implying questions), directed through (e.g. researcher

In this study, social media was used as used media was social In this study, Open-ended questions regarding HRQoL were posed in HRQoLOpen-ended were regarding questions

Care (IMI-ROADMAP), and Healthcare Alliance for Resourceful Medicines Offensive Against Against Medicines Offensive Resourceful for Alliance Healthcare and (IMI-ROADMAP), Care (IMI-HARMONY) in Hematology Neoplasms exploring currently are such findings. Therefore, it remains unclear whether both approaches complement or or complement whether both approaches unclear remains it Therefore, such findings. themselves lend potentially may they how and subsequently one another contradict is This decision-making. HTA to Across Outcomes projectsongoing Real-World of IMI including disease and haematological Alzheimer’s use for data on big gain insights social media to (41;42). respectively malignancies, unstructured data which was initially not designed to answer specific research questions questions research specific answer to not designed initially which was unstructured data to in relation challenges methodological fundamental raising hence At HTA. for of such data and interpretation analysis by generated results whether the another potential approach for using social media pertains using social for to approach potential another networkingsocial or (micro-)blogs, forums, patient as such channels, different on available HRQoL(16;17;40).on servicesIn data for the represent better may generated, structured data, some may argue that channels other than social media may may other than social media channels that argue some may structured data, generated, be used to with factors such as disease burden and practical difficulties associated with completing completing with practical and associated difficulties factorswith burden disease as such provide to less inclined feeling patients in result may questionnaires, paper-based Consequently, responses. of limitations abovementioned address would new HRQoL of questionnaires developers patients by provided insights use to be worthwhile thus it may questionnaires, current that ensure social media to through and carers of development(13;14;39). of development(13;14;39). deemed relevant to their personal perspectives, thereby encouraging them to complete complete to them encouraging thereby perspectives, personal their to deemed relevant such questionnaires. for allows perspectives on HRQoL. this approach Although respondents respondents questionnaires are conventionally developed with developed conventionally are questionnaires rather than than rather the Strengths to distribute used were media channels social different Three two different forms of social media: Twitter (micro-blogs), Facebook and LinkedIn (social and LinkedIn Facebook (social (micro-blogs), Twitter forms media: of social different two networking sites).

Novel sources for generating RWE 9 184 Novel sources for generating RWE 9 185 relevance relevance issues raised. raised. issues the the selected disease- the survey into languages survey languages into robustness of findings findings of robustness project timelines given project given timelines relevance of findings for for of findings relevance the the the current study does not shed study does not shed current the the context of this feasibility study, study, of this feasibility context respondents in this study, which which study, this in respondents the the three instruments was agreed upon upon agreed was instruments three analysis were resolved by consensus consensus by resolved were analysis relevance of such generic measures for for genericsuch of measures relevance the the the single time-point single per one set of answers using cancer-specific EORTC-QLQ-C30 and melanoma- and EORTC-QLQ-C30 cancer-specific a authors aimed to translate translate aimed to authors feasibility of employing social media to assess patient patient assess to social media of employing feasibility the the cross-sectional analysis of melanoma patient perspectives perspectives of melanoma patient cross-sectional analysis the SF-36 and EQ-5D questionnaires. Provided Provided and EQ-5DSF-36 questionnaires. a composition and constitution of respondents allowed for for allowed of respondents and constitution composition effectiveness of melanoma drugs in practice conventionally conventionally of melanoma drugs in practice effectiveness considerable number of number considerable the research team. Other team. generic which HRQoL exist instruments research a the the the feasibility of long-term, longitudinal data collection through social social collection data through longitudinal long-term, of feasibility study. Initially, Initially, study. the the other hand, it may be argued that that be argued may it hand, other the native language of of language native survey developed to assess perspectives on HRQoL was written in English. This was was This in English. perspectives assess HRQoL on written survey to was developed low number of survey respondents precluded quantitative analysis of differences differences of analysis quantitative precluded of survey number low respondents comparison made would have been predictably lower than for than for predictably been lower have made would comparison the identification of such differences in perspectives on HRQoL and generated hypotheses hypotheses in perspectives on HRQoLand generated of such differences identification aim of this study was to test test to was this study aim of views of their writers accurately and comprehensively. and accurately of their writers views Comparison of patient and carers´ perspectives on HRQoL performed was and carers´ of patient against Comparison Data collection performedData was at The The Inductive content analysis on free text in compiled responses was conducted conducted was responses textin compiled on free analysis Inductive content specific instruments. specific of such generic measures for REA of drugs, this may impact this may of drugs, REA for of such generic measures On HTA. the were not included, such as not included, were by consensus amongst consensus by three validated questionnaires: questionnaires: validated three on HRQoL. Although this information is valuable in is valuable on HRQoL. information this Although or completion questionnaire in rates attrition potential on light decision-making. inform to media Selection of FACT-M. and EORTC-MEL38 specific spoken by all MPNE members yet this proved unfeasible within unfeasible proved this yet all MPNE members by spoken countries. European least 13 at members span that providing thereby respondent, decision-making on HTA further be may studies Therefore, social media. collection data through longitudinal from assess to required requires longitudinal data collection on HRQoL. Therefore, Therefore, collection data on HRQoL. longitudinal requires may have impacted their ability to adequately represent their thoughts on thoughts their represent impactedability their adequately to have may likely more selection English-speaking were also led to since bias It respondents have may participateto in not perspectives on HRQoL, the research. future for Limitations The between patients vs. carers and patients of different stages. However, provided that that provided However, stages. of different patients and carers vs. patients between the independently by 2 researchers. This approach avoids limitations associated with with associated limitations avoids approach This 2 researchers. by independently and slang or misinterpreting words misspelled as missing such approaches computerised to related all discrepancies Moreover, sarcasm. their own terms and length. This ensured that responses compiled were likely to represent represent to likely were compiled responses that ensured This and length. terms own their the amongst both researchers to ensure validity. ensure to both researchers amongst potential potential the similar study on study on similar a potential value of of value potential the medium vs. data mining) mining) data medium vs. a feasibility study, study, feasibility a use of social media to collect longitudinal collect longitudinal to media use of social the potential approaches for using social media using social media for approaches potential methods for combining combining for methods the availability and impact and availability HRQoL of of in REA data the use of social media (i.e. as media (i.e. use of social the the valuable tool to assess patient and carer perspectives on perspectives on carer and patient assess to tool valuable a medium for gathering cross-sectional data on melanoma patient patient on melanoma cross-sectional gathering data medium for a use of data-mining approaches to glean insights on HRQoL from other other on HRQoL glean insights from to approaches data-mining use of the question of how information generated from current cancer- and melanoma- cancer- current from generated information of how the question broader melanoma patient population. Differences emerge between what patients patients what between emerge Differences population. melanoma patient broader two different approaches for approaches different two HRQoL of melanoma patients. HRQoL patients. of melanoma

wide array of other potential uses, such as: such uses, potential of other wide array Furthermore, current knowledge on current Furthermore,

CONCLUSIONS channels (e.g. patient forums) and and forums) patient (e.g. channels data on HRQoL,data the for HTA decision-making. Additionally, since this was was this since decision-making. Additionally, HTA for perspectives on HRQoL through questionnaires, future research should also aim to address address should also aim to research future questionnaires, perspectives on HRQoL through the use of social media as social media use of important of aspects analysis are that to quantitative robust for allow would scale larger the to inform HTA decision-making is sparse. Although this study sheds light on sheds light study this decision-making Although sparse. is HTA inform to specific HRQoL questionnaires could be used for HTA decision-making and whether new, decision-making and whether new, HTA for be used could HRQoLspecific questionnaires reflectpatient stage-specific better that be developed should instruments patient-centred, perspectives HRQoL. on HRQoL, thus potentially increasing increasing HRQoL, thus potentially drugs. However insights gleaned through social media are not easily generalizable to to generalizable easily not are social media through gleaned insights However drugs. the and of varying important HRQoL. consider Cancer- and carers stages for melanoma fully correlate to do not seem available currently HRQoL specific melanoma- questionnaires important as view issues. of patients with what in HRQoL, particularly wording to in relation raises This Social media may provide provide Social may media

Novel sources for generating RWE 9 186 Novel sources for generating RWE 9 187 cross- a crowdsourcing crowdsourcing a Driving Seat: Policy Makers, Health Health Makers, Driving Seat: Policy national pharmacovigilance database. database. pharmacovigilance national the a Feasibility Study in Chronic Lymphocytic Lymphocytic Chronic in Study Feasibility Carr AJ, Higginson IJ. Are quality of life measures quality life of Are AJ, Carr Higginson IJ. Br Med J 2001;322(7298):1357. centred? patient S, Salek Staniszewska R, Wilson KL, Haywood Should Who PROMs in Healthcare: S. Using Be in or Patients? Methodologists Professionals, Springer. 2016. Drug SafDrug 2014;37(8):629-37. K, S, Simacek S, Fleming Bull McCarrier KP, Elicitation Concept al. et D, Cella P, Wicks Networks: Research Patient-Powered Within A 2016;19(1):42-52. Health Value Leukemia. K, Basen-Engquist J, Burzawa T, Zaid Bodurka et J, Ramondetta DC, Brown LM, conduct media to of social Use al. sectional quality and epidemiologic of life IMI-GetReal.IMI-GetReal: objectives. Overall from: Dec Available 2016]. [cited 2017 https://www.imi-getreal.eu/About-GetReal/ Overall-objectives. JS, Menone CM, Brownstein CC, Freifeld Digital al. et T, R, Kass-Hout Filice W, Bao surveillance:drug safety monitoring Drug products twitter. in pharmaceutical Saf2014;37(5):343-50. A, Hill A, Benton Chung S, Ungar L, Mao JJ, drug of Online discussion al. et CE, Leonard among and discontinuation side effects survivors. Pharmacoepidemiol cancer breast SafDrug 2013;22(3):256-62. A, Bondon-Guitton Montastruc E, Pages related effects JL, Bagheri H. Undesirable drugs: comparison antineoplastic oral to narratives internet patientsGÇÖ between and Pigmented Sande SR, MAJ. Piner L, Heijden synovitis: villonodular two and seventy hundred study of two Int Orthoppatients. 2016;1-10. https://www.imi-getreal.eu/Portals/1/ Documents/01%20WP1%20deliverables/ Deliverable%201.5%20Report%20-%20 metastatic%20melanoma%20-%20 webversion.pdf.

19. 20. 21. 17. 18. 13. 14. 15. 16.

High Level HighLevel methods of the the Makady A, Kalf R, Goettsch W., Lees M. M. Lees MakadyW., A, Kalf R, Goettsch on Metastatic Melanoma. Study - Case D1.5 from: DecAvailable 2016]. [cited 2017 Cella D. FACT-M: Functional Assessment of of Assessment Functional FACT-M: D. Cella Melanoma. - [cited 2017 Therapy Cancer http://www.facit. from: Dec Available 2016]. org/FACITOrg/Questionnaires. Garratt Schmidt A, Mackintosh L, A, Fitzpatrick bibliographic measurement: R. Quality of life outcome health assessed of patient study 2002;324(7351):1417. BMJ measures. EuroQoL. About EQ-5D. 2017 [cited Dec Dec [cited 2017 EQ-5D. About EuroQoL. http://www.euroqol. from: Available 2016]. org/about-eq-5d.html. Dec [cited 2017 EORTC-QLQ-C30. EORTC. http://groups.eortc. from: Available 2016]. be/qol/eortc-qlq-c30. National Institute for Health and Care Care and Health Institute for National Guide to Excellence. Institute National 2013. appraisal. technology 3-7-2015. Excellence. Care Health and for Kleijnen S, Lipska I, Alves TL, Meijboom K, ElsadaKleijnen S, Lipska I, Alves effectiveness Relative et al. V, Vervölgyi A, pricing for medicines oncology of assessments decisions in European and reimbursement 2016;mdw233. Oncol Ann countries. Richtlijn voor Nederland. Zorginstituut evaluaties economische van het uitvoeren 17-11-2015. in de gezondheidszorg. Nederland. Zorginstituut Matikas A, Mavroudis D. Beyond CTLA-4: Beyond Matikas D. Mavroudis A, strategies immunotherapy novel Future melanoma. metastatic for Immunotherapy LG. S, Campana Valpione directions. future melanoma: advanced for 2016;8(2):199-209. Immunotherapy HLPF. High Level Pharmaceutical Forum Forum Pharmaceutical High Level HLPF. and Conclusions Final 2005-2008. of Recommendations 2015;11(6):997-1009. Oncology Health Technology Assessment Technology Health Aug [cited 2017 Glossary. HTAi International. http://htaglossary. from: Available 2017]. net/tiki-index.php?page=List+all+terms. Mar 2015]. [cited 2017 Forum. Pharmaceutical

REFERENCE LIST REFERENCE 12. 11. 10. 9. 8. 7. 6. 5. 4. 3. 2. 1. U.K. the John Conley Conley John individual. J J the individual. development of development the The The European Organization Organization European me-too stifles mentalitythat The The a checklist for protocol writers: writers: checklist protocol for Innovative Medicines Initiative. Big Big Medicines Initiative. Innovative a ADDQoL. Qual Life Res 1999;8(1):79-91. ADDQoL. Qual Life randomised study of SIP, SF-36 and SF-36 SIP, study of randomised for Research and Treatment of Cancer of Cancer Treatment and Research for assessment: quality to of life approach questionnaire developing for guidelines Res 1993;2(4):287-95. Qual Life modules. Y. S, Su S, Gholizadeh Tsay Beusterien K, with colorectal experience Real-world forum web patient chemotherapies: cancer 2013;7:361. Ecancermedicalscience analysis. The respiratory and safety medicines data, for IMI Call million of €93 focus disease Dec 2016]. Available [cited 2017 proposals. Fojo T, Mailankody S, Lo A. Unintended Unintended A. Mailankody S, Lo T, Fojo cancer expensive of consequences of marginal pursuit therapeuticsGÇöthe and indications drugs do cancer Why J. Lexchin DW, Light 2015;350:h2068. BMJ ride. such an easy get Harvey A, Girling P, Hopwood PM, Fayers Quality R. Stephens of life Machin D, DJ, in clinical trialsGÇöguidelines assessment and Patients’ GD. Borasio M, Wasner Neudert C, instruments: quality of assessment of life a amyotrophic with patients in SEIQoL-DW Sci 2001;191(1):103-9. J Neurol sclerosis. lateral K, Bjordal A, Groenvold Cull MAG, Sprangers NK.M, Aaronson Bradley C, Todd C, Gorton T, Symonds E, Symonds T, Gorton C, Todd C, Bradley Martin R. A, Plowright of measure questionnaire an individualized impact on quality of diabetes life: of perceived the O’Boyle CA, Kearney M, Moriarty D, Waldron in Quality-of-life measurement D. M, Carney cancer: assessing advanced 1999;17(11):3603-11. Oncol Clin J Eur experience. Council Medical Research 1997;33(1):20-8. Cancer Health-related quality of life in patients with with patients in quality life of Health-related and additional limitations tumors: brain and neurology Current measures. outcome reports 2013;13(7):359. neuroscience innovation and creativity: innovation Lecture. JAMA OtolaryngologyGÇôHead JAMA & Lecture. SurgeryNeck 2014;140(12):1225-36.

39. 40. 41. 36. 37. 38. 35. 34. 33. quality quality economic economic the the feasibility study. study. feasibility a Danish population- Danish A cervix: language and environment and environment language A the role of health care providers and and providers care of health role external validity of randomised external validity of randomised The The nationwide population-based study in in study population-based nationwide evaluation of treatments for AlzheimerGÇÖs AlzheimerGÇÖs for of treatments evaluation 2012;29(1):31-43. aging Drugs & disease. J Clin disease. with chronic of patients of life 2002;55(11):1130-43. Epidemiol NK,Dirven Aaronson JC, L, Reijneveld MJ. Taphoorn BM, A, Uitdehaag Bottomley Shearer J, Green C, Ritchie CW, Zajicek JP. JP. Zajicek C, Ritchie CW, Green J, Shearer in use for values state Health Aaronson MA, Sprangers KC, Sneeuw NK. for statistical computing. R Foundation for for R Foundation computing. statistical for 2015 R Foundation Computing Statistical from: ; Available Computing Statistical for https://www.R-project.org/. can affect that Factors PM. Rothwell the 2006;1(1):e9. Trial Clin PLOS trials. controlled Melanoma. of Erasmus C. Burden Holterheus 2011. Rotterdam; University Hsieh HF, Shannon SE. Three approaches approaches Three SE. Shannon Hsieh HF, Qual Health analysis. content qualitative to Res 2005;15(9):1277-88. R: Team. R Core Bay C, Kejs A, Storm H, Engholm G. Incidence G. Incidence H, Engholm A, Storm C, Kejs Bay and survival with cutaneous in patients anatomical morphology, by melanoma stage: TNM and site Scott N, Fayers P, Aaronson N, Bottomley A, de A, N, Bottomley Aaronson P, N, Fayers Scott QLQ-C30 EORTC M, et al. A, Groenvold Graeff ; July EORTC 2008 EORTC Values. Reference Cancer 1989-2011. study register based 2015;39(1):1-7. Epidemiology MPNE. Melanoma Patient Network Europe. Network Europe. MelanomaMPNE. Patient from: Dec Available 2016]. 28 [cited May 2016 http://www.melanomapatientnetworkeu. org/home.html. E, Mansson-Brahme J, Eriksson Lyth H, M, Ingvar C, C, Lindholm Frohm-Nilsson is associated of education level Low et al. reduced and diagnosis at stage later with survival melanoma: malignant in cutaneous A 2013;49(12):2705-16. J Cancer Eur Sweden. survey of patients with neuroendocrine surveyneuroendocrine with patients of of carcinoma significant others in evaluating evaluating others in significant Gynecol Oncol 2014;132(1):149-53. Oncol Gynecol

32. 31. 29. 30. 28. 27. 26. 25. 24. 23. 22.

Novel sources for generating RWE 9 188 Novel sources for generating RWE 9 189 AD spectrum for better care: Multi- care: better AD spectrum for from: http://www.imi.europa.eu/content/ from: imi-2-call-6-launch. across Outcomes Real world ROADMAP. the [cited 2017 Platform. Access modal data http://www. from: Dec Available 2016]. roadmap-alzheimer.org/index.html.

42.

SECTION General Discussion VI

CHAPTER

Real-World Evidence for Health Technology Assessment of 10 Pharmaceuticals: Perspectives on the status quo and the future

General Discussion 195 10 dialogue dialogue definitions definitions a unanimous unanimous fundamental fundamental a use of RWE in the the use of real-world real-world use of the single definition of of definition single the a sources of data that qualify as RWDas qualify that data of sources research conducted. Subsequently, we we Subsequently, conducted. research the the pre-determined interim analysis) randomization pre-determined randomization analysis) interim a current gap of knowledge on current sub-analysis of HTA agencies, regulatory agencies and regulatory and agencies agencies, sub-analysis of HTA the definitions available in current literature (both academic academic (both literature current in available definitions a field of RWE use in decision making and their relevance field of makingdecision in use RWE relevance and their general lack of consensus of what RWD precisely is. RWD is. of what lack of consensus precisely general final chapter of this thesis, we will summarize and briefly briefly and summarize we will thesis, of this final chapter value of RWD for decision making is hindered by these these of RWD making decision value by for hindered is the a the main findings of main findings the concept of RWD and its associated evidence sources. of RWDevidence concept its associated and field of healthcare assign to this concept during interviews. during interviews. concept this to assign of healthcare field concepts of Real-World Data (RWD), as well as (RWD), Data as well of Real-World concepts the the the the the non-RCT setting). Yet for others stakeholders, who believe that RWD that believe who stakeholders, others for Yet non-RCT setting). practical and cultural obstacles associated with practical associated obstacles and cultural OF RESEARCH QUESTION RESEARCH OF a , we explored explored we , the definitions for RWD identified in literature or cited in interviews could be interviews in could or cited for in literature definitions identified RWD strictly non-experimental setting, PCTs would not. Therefore, Therefore, not. would strictly PCTs non-experimental setting, a later point in time (after point later same extent as RCTs. For some stakeholders, PCTs would thus qualify as RWDthus qualify would PCTs some stakeholders, For RCTs. as extent same the a implications of of implications the FINDINGS & IMPLICATIONS & FINDINGS results demonstrated demonstrated results the implications of such findings are far-reaching, since they comprise they since far-reaching, of such findings are implications main findings of this thesis. Finally, we will discuss potential steps towards more more towards steps potential will discuss we Finally, thesis. of this main findings first section The The The The Although we recognize that agreement by all stakeholders on all stakeholders by agreement that recognize we Although the the MAIN MAIN RE-STATEMENT RE-STATEMENT understanding of what RWE is, whether RWE is used by HTA agencies, in which contexts in which contexts agencies, RWE RWE whether of what understanding HTA is, is used by RWE and is used to making.and decision use of RWEoptimal in HTA evidence (RWE) in health technology assessment (HTA) and decision making for drugs. making decision and drugs. for (RWE) (HTA) assessment evidence technology health in have pertained aspects, gap whether stakeholders including: This several to HTA and decision making. and In HTA discuss in initiatives new will highlight In Diverging understanding of RWE and differing policies for RWE use among policies understanding of RWE and differing Diverging agencies HTA In this thesis, we set out to address address set out to we In thesis, this and non-academic) for for and non-academic) understanding of stakeholders regarding RWD, RWD, regarding stakeholders of understanding Trials Clinical Pragmatic consider we an example, making. use in decision and their As criteria inclusion/exclusion based on less stringent included are patients In PCTs, (PCTs). arms. treatment to randomized initially and are trials (RCTs) controlled than randomized at However, and blinding of treatment allocation may be ended (1). Such PCTs involve elements whereby whereby elements involve Such PCTs be ended (1). may allocation of treatment and blinding procedures allocation criteria and treatment intervene with inclusion/exclusion researchers not to yet different stakeholders in stakeholders different Qualitatively, classified into different categories, each category imposing different requirements on what what on requirements category each different imposing categories, different into classified In does or does not qualify as RWD. (i.e. data generated in generated data (i.e. pharmaceutical industry, two-thirds of interviewees could not provide an official definition official definition an not provide two-thirds of interviewees could industry, pharmaceutical is there that clear became it Therefore, RWD level. institutional an on adopted for is which RWDis. what on agencies) precisely HTA (including stakeholders between no consensus in is collected amongst stakeholders on stakeholders amongst in underlying differences RWD may seem far-fetched, we strongly encourage consensus-seeking on this topic. consensus-seeking on this topic. encourage strongly we RWD seem far-fetched, may

lack of lack of the evidence is is evidence harmonized harmonized A ability to plan ability plan to the the context of use and use and of context same assessors would would assessors same basis of limited phase II phase II limited of basis platform for discussions discussions for platform the number of HTA agencies, agencies, of HTA number a the a the use and appraisal of RWE by of RWEuse and appraisal by case in the increasing trend of new (oncology or or (oncology new of trend increasing the the current policies of 6 European HTA agencies agencies HTA policies of 6 European current policies for for policies the disease, treatment-related costs, resource use and and use resource costs, treatment-related disease, RWE submitted differently depending on whether whether on depending RWEdifferently submitted the the the opportunity for CRSs, since opportunitysince CRSs, for separate assessors. Another factor to this problem presents presents problem this factor Another to assessors. separate good starting point for further dialogue amongst different startinggood further for point different amongst dialogue the context of IRD or CRS. Meanwhile, inter-agency differences in in inter-agency differences or CRS. Meanwhile, of IRD context a the the research question at hand determines whether whether determines hand at question research , we explored explored , we result, RWE submitted as part of an HTA submission may be assessed assessed be may submission RWE as partresult, submitted HTA of an the product life cycle (3-5). product life a aforementioned points, it may be useful for HTA agencies in Europe to to Europe in agencies HTA for useful be may it points, aforementioned the use of RWE for treatment effect estimates to compensate for compensate to estimates effect use of RWE treatment for parameters for which RWE is used. In general, three distinct contexts could distinct could three which RWE contexts In for general, is used. parameters envisioned purpose. However, as is However, purpose. envisioned RWE they accept in HTA submissions, as well their appraisal of RWE their appraisal in as well submissions, RWEin HTA they accept the the the latter theoretically yielding outcomes more generalizable for HTA purposes purposes HTA for generalizable more outcomes yielding theoretically latter the European Network of HTA (EUnetHTA) may provide provide may (EUnetHTA) Network of HTA European the second sectionsecond Innovative Medicines Initiative (IMI)-GetReal consortium in 2016 (1). We believe believe (IMI)-GetReal MedicinesWe Initiative Innovative consortium (1). 2016 in the results demonstrated that that demonstrated results The The prevalence and incidence of incidence and prevalence the the International Society for Pharmacoeconomic and Outcomes Resarch (ISPOR) in 2007 (2) 2007 (2) in (ISPOR) Resarch Outcomes and Society Pharmacoeconomic International for dossier is submitted in submitted dossier is In In all three contexts, RWE use for parameters of (relative) effectiveness of drugs was was of drugs effectiveness (relative) of RWE parameters for use contexts, In all three The The Inter-context variation in how RWE is used in HTA decision making may seem logical at making decision at seem logical may RWE in how Inter-context variation in HTA used is Inof light the In recent years, several definitions have been developed by different initiatives like like initiatives different by developed been have definitions several years, Inrecent the up to that such definitions provide provide definitions such that stakeholders on this topic. on this stakeholders regarding associated with some degree of controversy. Moreover, agencies differed in how acceptable acceptable in how differed agencies Moreover, controversy. of degree with some associated they deemed be sketched whereby RWE use is considered, namely: RWE Initial use is considered, whereby Discussions Reimbursement be sketched Schemes Reimbursement (CRS). and Conditional (PEA) Analysis Pharmacoeconomic (IRD), HTA submissions. HTA decision makingagencies in main factors; on two hinges HTA subsequently relating RWE parameters this, use for to In diseases). contrast rare for (e.g. RCTs for evidence to first sight; ultimately, ultimately, sight; first adherence to treatment was largely accepted. accepted. largely was treatment to adherence relevant for for relevant and effectiveness relative on evidence assess to usually assigned are assessors separate As cost-effectiveness. by differently and appraised offer which agencies in itself set of policies on RWD use by HTA agencies would provide MAHs with with MAHs provide would agencies on RWDset of policies HTA by use real-world from data more provide next RCTs, that, to pathways generation evidence studies; data or surrogate outcomes rather than confirmatory rather phase III RCT outcomes (6-8). surrogate data or data policies on RWE accepted and RWE appraisal may present Marketing Authorization Holders Holders Marketing Authorization RWE and present on RWEpolicies may appraisal accepted evidence for strategies questions when developing of challenging array with an (MAHs) across generation on practical guidance aspects policies on RWE align of RWE and and provide generation important is especially This of in light analysis. marketing on authorization conditional orphan) drugs granted be obliged to assess and appraise appraise and assess to obliged be (9-11). (9-11). the

General Discussion

196 10 General Discussion 197 10

more more a lack of of lack CF drugs drugs CF the lack of RWE the setting of CEA of CEA setting the the procedure for CF. An An CF. for procedure initial HTA report (T=0) HTA initial the main reason for inclusion inclusion for main reason the The The implementation of Conditional of Conditional implementation the relatively minor role in HTA, especially especially in HTA, minor role relatively Netherlands, an example of Managed of Managed an example Netherlands, a efforts undertaken by EUnetHTA later in in later efforts undertakenby EUnetHTA fixed 4-year window, may provide provide may window, 4-year fixed possible reason could be could reason possible the a evidence generated (14;15). Moreover, one one Moreover, (14;15). generated evidence A the time needed to set up registries required for for required set up registries time needed to inclusion, analysis and interpretation of RWEof interpretation and analysis inclusion, the product of these ongoing collaborations will be will be product ongoing collaborations of these the the use of RWE in IRD (i.e. standard HTA) of innovative innovative of HTA) use of RWE standard (i.e. in IRD elapsed time between time between elapsed the the the experiences gained in in gained experiences drug, there might be insufficient time for MAHs to collect to MAHs for time be insufficient might there drug, the a 5 agencies; some citing RWD only for prevalence and/or and/or RWDprevalence citing some agencies; 5 for only number of shortcomings related to to of shortcomingsnumber related a fact that question why RWE why question plays the 4-year period is applicable to all indications for which which for indications all period to applicable is 4-year the the a , we examined examined , we use of tailored approaches for determining required time-frames required to determining for approaches use of tailored , we explored how and in which contexts RWE is currently used in HTA RWE used in HTA contexts in which and is currently how explored , we The The time of initial HTA assessments. Since these assessments take place soon soon place take these assessments Since assessments. initial HTA time of , we explored explored , we myriad of factors, including including of factors, myriad the questions raised at T=0, rather than T=0, rather at raised questions a In Chapter 4 In Chapter study highlighted study highlighted the third sectionthird In general, RWD was more often included in cost-effectiveness assessments (CEAs) than than (CEAs) assessments RWD often cost-effectiveness in included In more was general, The The These results raise raise results These In Chapter 5 In Chapter the in REA was estimating prevalence and/or incidence of melanoma and in of melanoma and/or incidence prevalence estimating was in REA in relative effectiveness assessments (REAs) of HTA reports. HTA of (REAs) assessments effectiveness in relative this chapter. practice. agencies. HTA European 5 melanoma by metastatic for drugs In fragmented reality of RWE use in HTA practice: standard HTA and HTA standard practice: of RWE reality HTA use in fragmented The (MEA) Managed Entry Agreements on aligning RWD policies. We will return to to will return We RWD on aligning policies. Financing (CF) of expensive hospital drugs in of expensive (CF) Financing Entry Agreements (MEA) scheme. (MEA) EntryAgreements MEAs. future for design appropriate available at at available of after regulatory approval important is example for parameters relating to drug effectiveness. drug effectiveness. to relating parameters for for extrapolating long-term effectiveness of new drugs. If reports, drugs. of new included in was effectiveness RWD long-term extrapolating for critical When identified, often not identified. were appraisal its regarding clear statements Inclusion of RWD differed mostly unknown was REAs in or negative. outcome appraisal between substantially incidence and others for drug effectiveness and safety. effectiveness drug for and others incidence answer answer guidance on systematic approaches for for approaches systematic on guidance begun collaborating only recently have agencies HTA In purposes. this context, HTA for RWD generating and for study designs of appropriate understanding on strengthening through sources further RWE of synthesis different developing from for methods analytic amongst HTA dialogue Further (12;13). such as IMI-GetRealinitiatives EUnetHTA and is necessary that agencies ensure to chapter. this in later this to return will We decision-makers. useful by deemed This may report reassessment and final all drugs. for almost years 4 extended(T=4) beyond be due to approved. were data collection, and subsequently assess collection, assess data and subsequently whether wonder may RWE through registries or observational studies. Another factor could be factorobservational or Another could RWEregistries studies. through time. time. third of of third dossiers dossiers national national the a context of of context majority of the the stakeholders stakeholders feasibility and and feasibility ever-changing ever-changing the the The The the the timelines projected. projected. timelines the fact that methodological standpoint. standpoint. methodological a the RWE generated between T=0 and T=0 and between RWE generated the necessary feedback rounds from HTA HTA from necessary rounds feedback functioning of CF, possible improvement improvement possible functioning of CF, the validity and relevance of outcomes research research of outcomes validity and relevance the binary decision system on reimbursement and and binary on reimbursement decision system low quality of RWE generated, quality low of RWE generated, the the the policy tool. a value of RWE for eventual decision making of RWE eventual at value for opinions of different stakeholders on CF. CF. on stakeholders different of opinions the reimbursement of controversial drugs through drugs through of controversial reimbursement the scheme in practice. scheme the stakeholders deemed CF successful in achieving all of its aims aims all of its in achieving CF successful deemed stakeholders the proposed outcomes research, research, outcomes proposed the CF procedure for drugs, thereby avoiding waste of valuable time and and time valuable of waste avoiding thereby drugs, for procedure CF insights provided by stakeholders confirmed those from those from confirmed stakeholders by provided insights the , we explored explored we , necessary improvements relating to procedure, governance, methodology methodology governance, procedure, to relating necessary improvements the mid-term reporting of outcomes research progress and interim results results interim and progress research reportingmid-term of outcomes future need for CF as CF need for future often poor quality of RWE generated by outcomes research meant that much much that meant research often quality poor outcomes by of RWE generated the the other hand, all stakeholders alluded to positive aspects of CF. For example, it it example, For aspects positive CF. of to alluded stakeholders all other hand, the The The the study also highlighted shortcomings in CF from shortcomings from in CF also highlighted study emergence of innovative, yet expensive medications is occurring rapidly. Moreover, Moreover, is occurring rapidly. medications expensive yet of innovative, emergence the conduct of value of information (VoI) analyses at T=0 to highlight highlight T=0 to (VoI) at conduct of information analyses value of In Chapter 6 InChapter The The Many of Many On The “back alleyway” to allow allow to “back alleyway” Secondly Secondly accelerated/conditional approval pathways (6). Consequently, HTA agencies increasingly agencies increasingly HTA (6). Consequently, pathways approval accelerated/conditional uncertainties with more submissions on aspects health such as long-term encounter on MEAs reliance increasing to practice leading in clinical effectiveness and outcomes intrinsic value of data collection for specific parameters within within parameters specific collection of data for value intrinsic money for all stakeholders involved. Both design aspects may be essential for future design design future for aspects Both be essential design may involved. all stakeholders money for (18). literature in previous iterated as has been of MEAs, between T=0 and T=4 could have led to more timely decisions regarding adjustment or or adjustment regarding decisions timely more to led have T=4 could T=0 and between of termination on aspects insights their such as provided stakeholders were skeptical of skeptical were stakeholders marketing gain based on less authorization increasingly treatments oncology novel studies) within I/II phase or efficacy(e.g. on safety evidence conclusive For example, it was apparent that despite despite that apparent it was example, For on committees quality answer to insufficient of deemed was research outcomes through T=4 uncertainty remained unresolved regarding key reimbursement criteria, including cost- criteria, including key reimbursement regarding uncertainty unresolved remained challenges as such findings, for these reasons numerous to alludes Literature effectiveness. making decision for RWE Safeguards (16;17). generated and interpreting with analyzing such shortcomings Firstly, in hindsight. prevented have could in ZIN guidelines proposed the questions. questions. analyzed in Chapter 5. Notable new insights gained included gained 5. Notable new insights in Chapter analyzed points and points Therefore, none of Therefore, and only half indicated that its goals were partially met. Moreover, some considered it it considered some partially Moreover, goals were its met. that indicated half and only a served important to as an alternative reimbursement package. reimbursement CF boosted Furthermore, collaboration. inter-stakeholder lessons for valuable generated impactbudget and cost-effectiveness of healthcare. of drugs in awareness societal CF should be reintroduced, that agreed unanimously all of stakeholders almost Therefore, with albeit of and implementation reality of clinical practice often affected practice clinical often of reality affected with InT=0. combination at planned

General Discussion

198 10 General Discussion 199 10

model model a potential potential the inaccessibility inaccessibility focus of future of future focus US FDA-Sentinel US FDA-Sentinel the the potential benefit of of benefit potential the the successes, failures, strengths strengths failures, successes, main advantage of such of such main advantage the acceptability of RWE acceptability and use in CER robustness of findings based on of findings robustness RWE The The the the effectiveness of treatments for all patients in in patients all for treatments of effectiveness obstacles associated with accessing and using using and accessing with associated obstacles the lack of experience with using RWE in currently RWE with using lack of experience in currently wide lack of trust among decision-makers regarding regarding decision-makers wide lack of trust among a the reasons behind such controversy generally hinged hinged generally such controversy behind reasons lack of concrete examples demonstrating demonstrating examples lack of concrete future. Another example relates to to relates example Another future. the consortium’s experiences with inaccessibility of IPD RWE of IPD inaccessibility with experiences consortium’s lack of trust in lack of the The The the The The a wide-spread issue. Bearing in mind that RWE is generated by wide-spreadby BearingRWE that in mind issue. generated is a remaining case studies, access to IPD was denied. Reasons for Reasons for denied. IPD was to access case studies, remaining sentinel initiative without having to relinquish complete access access complete relinquish to having without initiative sentinel , we examined examined , we the the case studies, IPD was accessed from registries and observational and registries from accessed IPD was case studies, the decentralized network of participating databases (22). In this model, databases network participating of databases (22). In databases model, decentralized this Netherlands and elsewhere. a fourth section the worthwhile endeavour for for endeavour worthwhile publication of similar IPD for RWE generated through observational studies and may observational through RWE studies and may for of similar IPD generated publication EU Clinical Trial Directive was recently adopted whereby sponsors of RCTs conducted conducted of RCTs sponsors whereby adopted recently was Directive Trial EU Clinical robustness of findings based on based of findings robustness RWE. An array of potential solutions lend themselves to overcoming this persistent persistent this overcoming to themselves lend solutions of potential array An IMI-GetReal’s experiences in accessing IPD from RWE repositories were disparate. disparate. RWE were from IPD repositories in accessing experiences IMI-GetReal’s Furthermore, stakeholders’ views varied on varied views stakeholders’ Furthermore, a the negative consequences on changing on changing consequences of RWEthe negative and Accessibility longstanding paradigms In by agencies worldwide (19;20). One may argue that without systematic evaluations of of evaluations systematic without that argue may One (19;20). worldwide agencies by ones. as previous caveats similar suffer to likely are schemes novel MEAs, established risk, potential this knowledge regarding counter To research, in order to avoid repeating historical mistakes when setting up new schemes up new schemes setting when mistakes historical repeating avoid to in order research, within and weaknesses of established MEAs in other jurisdictions in and weaknesses MEAs established of should be for marketing authorization applications agreed to provide access to all patient-level all patient-level to access provide to agreed marketing applications authorization for for exists initiative no equivalent reportsclinical of trial subjects Presently, online (21). the be of out or can opt-in queries. external run still research yet their IPD, to inaccessibility to RWE and maximizing societal gain from its use in CER. For example, example, RWE CER. For use in its to inaccessibility from gain societal and maximizing the added value of RWE of added value in CER and use the multiple queries to data standardized can send external researchers whereby initiative, nodes of using RWE to provide critical insights on on critical insights RWEusing provide to aspects: inter-related on two RCT with as compared well as data, For half of For individual patient data (IPD) from RWE repositories in comparative effectiveness research research effectiveness RWE in comparative from repositories (IPD) data patient individual REA). for term an equivalent (CER; for However, studies. project within timelines research-ready not being datasets to mostly related inaccessibility data. share to or unwillingness of lack that implying external stakeholders, many by echoed purposes were research for RWEIPD to remains access diminishes access this barrier data to practice, clinical in patients making. decision subsequent Arguably, effectiveness. drug (comparative) address to methods available healthcare practice. practice. healthcare IPD RWE both to to contributes novel novel use of use of a the drawbacks drawbacks valuable tool tool valuable the a Biobank Standardisation Standardisation Biobank the scoping review of academic of academic review scoping formal setting in which panels in which panels setting formal use of social media to collect collect use of social media to a posts requires manual work since since manual work posts requires the the potential source of RWD for REA, RWD of REA, for source potential a the lack of guidance on managing managing on guidance lack of a , we conducted conducted , we possibilities for RWE generation through RWE through for possibilities generation Healthy Obese Project and and Obese Project Healthy feasibility study on study feasibility the a the sensitive aspects relating to full-fledged access to IPD while IPD while to access full-fledged to aspectssensitive relating In Chapter 8 In Chapter use of social media may be less resource-intensive and more and more less resource-intensive be may social media use of the the extraction, interpretation and subsequent use of social media in social media in use of and subsequent extraction, interpretation development of clear and uniform methodological guidance is thus thus is guidance methodological clear and uniform of development we explored explored we the short period of time and from audiences not usually included in RCTs short in RCTs not usually included audiences period from of time and The The a , we conducted conducted , we availability and impact of HRQoL data in REA of drugs. Responses can be and impact Responsesavailability can be of HRQoL in REA of drugs. data required information for furthering scientific pursuits. Similar frameworks furtheringSimilar frameworks for pursuits. scientific information required the the different note, note, different lack of clear methodological guidance. Standardized approaches regarding regarding approaches Standardized guidance. clear methodological lack of results indicated that Social media may be Social may that media indicated results fifth section, a limited acceptability of such data. data. of such acceptability limited The The On One caveat of using social media to collect health data that requires special attention attention special requires collect that data health to media social of using One caveat Findings from this study indicated that social media may provide provide may social media that indicated this study from Findings In Chapter 9 In Chapter the the and Harmonisation for Research Excellence Project (BioSHaRE-EU) (23). (BioSHaRE-EU) Project Excellence Research for and Harmonisation source, namely social media. namely social source, In delivering delivering conduct to as DataSHIELD such research, other fields of in been implemented have as part research of international RWE collection but within confines for potential Social media demonstrates is its ability to circumvent abilityis its circumvent to and grey literature to examine whether social media was being used to inform (relative) (relative) inform to being used media was social whether examine to literature and grey research. effectiveness efficient than data collection through patient/citizen panels at various stages of HTA HTA various of stages at panels patient/citizen collection data than through efficient share to hesitate participating panels may in patient/citizen making.decision Patients their HRQoL due to to particular related insights personal (e.g. women or patients with early stages of melanoma) (27;28). Employing social media media social with early Employing (27;28). of melanoma) stages or patients women (e.g. trials or point-of-care than multi-centre less resources require collect would such data to cervical on research (29). cancer in previous confirmed and has been studies gathered within gathered to quickly and efficiently assess patient and carer perspectives on HRQoL, potentially perspectives on HRQoL, potentially quicklyand carer to patient assess and efficiently increasing is and necessary comparability ensure are to and interpretation collection,data analysis be may posts media social although example, For studies. between reproducibility of interpretation processes, extracted automated by particularly on aspects such as adverse events (AEs), occurrence of disease-specific of particularly occurrence (AEs), on aspects events adverse as such potential This (HRQoL). quality of life health-related and behaviour, adherence symptoms, accompany that limitations fully acknowledged been methodological to due has not yet as as well and selection bias bias, information as such data, health media-generated social the automated processes cannot successfully interpret aspects or such as sarcasm interpret successfully cannot processes automated is currently there Additionally, (24-26). slang selection bias and of authenticity, validation regarding above of social media highlighted bias. information perspectives HRQoL on in melanoma. and carers’ patients’ critical to improve improve critical to decision making.decision

General Discussion

200 10 General Discussion 201 10 trusted trusted findings findings a number of of number conduct of conduct of “real world”. world”. “real the a the pre-registration pre-registration the survey collect to publicly-accessible publicly-accessible a the two approaches for for approaches two other hand, through through hand, other a IMI-Big Data for Better Better for Data IMI-Big the the the topic of ongoing research by by of ongoing research topic collection and use of RWE in a researcher (e.g. through directed directed through (e.g. researcher relevance thereof for for thereof relevance the the the potential of using social media to gain gain to media social of using potential the medium for spreading spreading medium for authors explicitly call for call for explicitly authors a the reasons behind potential discrepancies in such in such discrepancies potential behind reasons the mining of unstructured data which was initially not not initially was which unstructured of mining data the confidence of health care decision makers to factor these factorto these decision makers health care of confidence dynamics of ongoing discussions. On discussions. of ongoing dynamics collection, analysis and interpretation of such data for HTA. At At HTA. for data such of and interpretation collection, analysis number of these initiatives have provided (or will soon provide) (or will soon provide) provided have initiatives of these number the A the the mining of data already available on different channels, such as such as channels, on different available already of data mining interpretation of such findings. This is This findings. of such interpretation International Society for Pharmacoepidemiology (ISPE) have recently recently have (ISPE) Society Pharmacoepidemiology International for the research conducted for this thesis between 2014 and 2018, several several and 2018, 2014 between thesis this for conducted research first report first et al., by Berger the the the Special Task Force (ISPOR-ISPE STF) on RWE. This task force published two two published force task This RWE. (ISPOR-ISPE STF) on Force Task Special the a other hand, it involves it involves other hand, latter approach, little intervention is applied by by is applied intervention little approach, latter ISPOR and ISPOR and authors. Additionally, initiatives such as IMI-WEB-RADR, and such as IMI-WEB-RADR, initiatives Additionally, authors. moment, little research exists on whether results generated by by generated results on whether exists research little moment, the As partAs of In this study, social media was used as media was social In this study, the INITIATIVES ON RWE & RELEVANCE TO MAIN FINDINGS MAIN TO RELEVANCE & RWE ON INITIATIVES ONGOING website, particularly for studies aiming to confirm pre-established hypotheses on (relative) particularly pre-established confirm to studies aiming on (relative) hypotheses for website, of RWE study protocols (including data analysis protocols) on on protocols) analysis data (including of RWE protocols study studies into their decision making (34); and good practices to structurally improve reporting decision making their structurally into studies practices to and good improve (34); in confidence instilling hereby reproducible, become studies database healthcare so that RWE (35). insights on adverse effects of drugs or to focus on big data use for cardiovascular disease, disease, cardiovascular for use data focus on big to drugs or of effects adverse on insights (32;33). respectively malignancies, and haematological disease Alzheimer’s Outcomes (IMI-BD4BO) are currently exploring currently are (IMI-BD4BO) Outcomes patient perspectives on HRQoL. Meanwhile, another potential approach for using social social using for approach potential another perspectives on HRQoL. Meanwhile, patient media pertains to patient forums, (micro-)blogs, or social networking or social (micro-)blogs, on HRQoL services data (24;30;31). forums, for patient In RWE enhance that studies using social media correlate or differ, or differ, correlate media using social and thus results the the environment of their choice. of their environment represent better on HRQoL gained insights may that implying questions), On methodological fundamental raising hence questions research specific answer to designed to in relation challenges established established namely: practices for topics, two reports procedural good addressed that themes these initiatives have (or aim to) address and address (or aim to) have themes these initiatives social media, patients can provide candid answers at their own pace and within and within pace own their at answers candid provide can patients social media, HTA and decision making. and decision HTA complimentary insights to those presented in this thesis. Below, we highlight highlight we Below, in this thesis. complimentary presented those to insights The The Good practices for the conduct of RWE studies, analyzing RWE the conduct and reporting of RWEGood studies, for practices RWEdecision making use in barrier to the cultural Addressing findings: In parallel to In to parallel to relation in been instigated have initiatives reputable of this thesis. of this are conducted or due to to due or conducted are work set of of set a the initiative will will initiative second report second the the robustness of such such of robustness ISPOR-ISPE STF and STF and ISPOR-ISPE the the PAtient REgistry INiTiative REgistry INiTiative PAtient transparency, reproducibility reproducibility transparency, the study. life-cycle approach to improve life-cycle improve to approach later in this chapter), chapter), this in later face validity of results from such such from validity results of face the A the the the longstanding collaboration aiming to create create to aiming collaboration longstanding a Work Packages (WP) Packages Work 3 (JA3, Action Joint within 2016- same datasets and decision-makers should be able to to able be should decision-makers and datasets same the the comprehensive list of analytical choices and data parameters parameters data and choices analytical of list comprehensive a checklist tool for registries to assess their fit for informing HTA HTA informing for their fit assess to registries checklist for tool initiative hopes to diminish hesitancy amongst drug regulators, hesitancy diminish hopes to regulators, drug amongst initiative work group plans to conduct pilot projects whereby registries registries conduct pilot projects to plans whereby group work current state of replicability of current RWE from studies on large RWE on large studies current of of replicability from state current (36), RECORD (37), PCORI Methodology Report PCORI (37), and EnCePP (38) RECORD (36), cultural reluctance amongst decision makers regarding RWE use regarding makers decision amongst reluctance cultural a the the on early dialogues (initial evidence generation) and Strand B on post- B on and Strand generation) on early evidence (initial dialogues the the A result of such reporting, different research teams should be able to reproduce reproduce to be able should teams research different of such reporting, result validity of analytic choices made throughout made throughout validity choices of analytic a sustainable system of HTA knowledge sharing and promote good practice in HTA practice good HTA in knowledge sharing promote and of HTA system sustainable a the European network for HTA (EUnetHTA) is (EUnetHTA) HTA network for European findings generated using generated findings collaboration has since striven to: facilitate efficient use of resources available for HTA, HTA, for available resources use of efficient facilitate to: striven has since collaboration With regards to Strand B (we will return to Strand Strand to will return B (we Strand to regards With Bearing in mind Another initiative worth is Reproducible noting initiative Another Evidence: to Practices Enhance launch evidence generation & registries. & generation evidence launch generate aims to group performed previously work is based upon by tool This (13). with MemberUnion States European provided which (42) JA) (PARENT Action Joint guidelines, recommendations and tools to support establishment of interoperable patient patient support to tools and interoperable of establishment recommendations guidelines, Moreover, registries. will be set up for RWE collection based on evidence gaps identified in HTA reports. This This reports. RWEin HTA for identified will be set up gaps collection based on evidence by Wang et al. establishes establishes al. et Wang by conducting RWEteams observational based on large studies research databases that report. reporting aim to should It previous from worth is on work builds this list that noting STROBE (e.g. guidelines (39)). As As (39)). the an effective and sustainable network for HTA across Europe (41). Established in 2005, in 2005, Established (41). Europe across for HTA networkand sustainable an effective the create to WP5, is dedicated specifically of EUnetHTA, 2020) activityWP5, two strands Within (13). devices medical of drugs and generation evidence Strand defined, are methods and processes. One of One and processes. methods treatment effects. In doing so, decision-makers’ trust in trust decision-makers’ so, In doing effects. treatment studies would be enhanced and concerns regarding data-mining and their impact data-mining study on regarding concerns and be enhanced would studies In concert be diminished. recommendations, these with would results assess assess quantitatively The The Harmonizing RWE requirements and registry standards for HTA decision HTA for and registry standards Harmonizing RWE requirements collaborative the EUnetHTA making in Europe: RWE (40). In doing so, RWE In (40). doing so, high profile due to databases from trust evidence to and payers clinicians, patients, with overturnedresults. conflictingand controversies efforts that hope such as 7, we 3 to in chapters demonstrated in particularly confidence to (relative) RWE, increased relation to in will contribute REPEAT drugs. of effectiveness and Achieve Transparency (REPEAT) which aims to improve improve which aims to (REPEAT) and Achieve Transparency and validity of longitudinal healthcare database research. In particular, In particular, research. database healthcare and validity of longitudinal healthcare databases and subsequently quantitatively evaluate evaluate quantitatively subsequently and databases healthcare

General Discussion

202 10 General Discussion 203 10

AP The The Shaping Shaping AP concept AP concept the the EMA has recently has recently EMA the medicine (43). medicine conduct of successful conduct of successful a potential of RWEof potential in use the the quality of data generated, there there quality generated, of data drug to supplement RCT supplement drug to and data; the the vital tool for for tool vital pilot projectpilot during which a a drug development pathway. Between March Between March pathway. drug development balance between expedited patient access to to access patient expedited between balance NEW Drug Development ParadIGmS (NEWDIGS) (NEWDIGS) ParadIGmS NEW Drug Development central role for RWE studies to supplement RCT’s. RWE RCT’s. supplement for to studies role central scientific concept for drug development and data and data development for drug concept scientific the a a the the efficacy trial (RCT) (44).Moreover, implementation of robust, standardized, inter-operable inter-operable standardized, of robust, implementation Netherlands in Chapter 6, whereby simple registries with with registries simple whereby 6, NetherlandsChapter in the The The the EMA also completed also completed EMA core common protocol (13). protocol common core a the European Medicines Agency (EMA; pan-European regulatory authority) pan-European Medicines Agency (EMA; European common research question and minimum data set for registries, and, if if and, registries, for set data minimum and question research common negative experiences stakeholders have cited with regards to to regards with cited have stakeholders experiences negative focus of ongoing projects. Building on progress made within within made on progress Building projects. of ongoing focus a the Massachusetts Institution of Technology (MIT), for proposals in resulting Technology Massachusetts Institution of negative impact on expansion of such negative the the need for guidance on core data sets which truly matter to decision making decision to in matter truly sets which data on core guidance need for a the the national and international level. This is much needed, provided HTA agencies’ agencies’ HTA provided needed, much is This level. international and national definition of definition Netherlands but across Europe, provided there is agreement among HTA agencies agencies among HTA is agreement there provided Europe, Netherlands across but restricted patient population with subsequent expansion based on evidence on evidence based expansion subsequent with population patient restricted a a third principle of AP on early dialogue between regulatory agencies, HTA agencies agencies HTA principle of AP on early regulatory dialogue between agencies, third second principle of AP indicates principle of AP indicates second a the implementation of CF in in CF of implementation The The The The Recalling and patients is patients and published scientific guidance on post-authorization effectiveness studies to deliver to deliver studies effectiveness post-authorization on guidance scientific published RWE(45). 2014 and August 2016, 2016, and August 2014 was implemented in practice(43). in implemented was This concept has inspired work within within work has inspired concept This at program effectiveness commence trial would an in which trials (RWE) “efficacy-to-effectiveness” of upon completion seamlessly early involvement of all stakeholders including patients, healthcare providers, HTA agencies agencies HTA providers, healthcare patients, including of all stakeholders early involvement discussions on in scientific and payers concept rests on three fundamental principles: iterative drug development (e.g. approval approval (e.g. development drug principles:iterative fundamental three on rests concept within generation); gathering evidence on real-life use of real-life on evidence gathering generation); began work on Adaptive Pathways (AP), (AP), Pathways on Adaptive work began Coinciding with efforts amongst HTA agencies to explore explore to agencies effortswith HTA amongst Coinciding Adaptive Pathways and the Product Life-Cycle Approach: Improving synergy synergy Improving Approach: Life-Cycle and the Product Pathways Adaptive RWE generation agencies regarding regulatory agencies and HTA between decision making,decision to access patient early and progressive for which allows generation possible, possible, entails producing producing entails the limited sets of parameters exponentially expanded as more research questions were raised raised were questions research as more expanded exponentially of parameters sets limited in time and MEAs on MEAs making(3-5). increasing reliance on MEAs to strike to MEAs on reliance increasing HTA. Not only would work by WP5 Strand B thus inform better implementation of registries of registries implementation better inform B thus Strand WP5 by work only would Not HTA. within protocols. common on core seems to be be seems to deliver could Europe RWE across generation agreement Furthermore, (19;20). generation evidence adequate versus drugs innovative harmonization an implicit suggests agencies HTA amongst protocols common on core obstacles minimize significantly would on this front Streamlining of RWE requirements. decision HTA for evidence valuable generating for MAHs including stakeholders for EMA- effects effects the the generation generation workgroup workgroup trial begins, trial begins, new drug) is is drug) new A development development intervention is is intervention a the the the the conventional approach approach conventional the regulatory pathway include include regulatory pathway effect of effect CRCT. Before Before CRCT. EMA and HTA agencies include include agencies HTA and EMA the a the intervention (e.g. (e.g. intervention the EUnetHTA WP5 WP5 Strand EUnetHTA the one-way cross-over design, where all all where design, one-waycross-over a the outputs of outputs needs of MEAs. needs sustainable system for early dialogues with early with dialogues for system sustainable the a the EMA and EUnetHTA WP5A&B thus complement one one complement thus WP5A&B EUnetHTA and EMA post-marketing setting to supplement phase III RCT supplement setting to post-marketing the the potential benefit of pragmatic trials for generating relevant relevant generating for trials pragmatic of benefit potential road to better synergy between HTA agencies and regulatory and regulatory agencies HTA between synergy better to road seminal paper by Shwartz et al. in 1967 (9;48). Moreover, Moreover, (9;48). in 1967 Shwartz et al. by paper seminal the the the SW-CRCT resembles thus parallel scientific advice between between advice scientific parallel drug development landscape shifts from shifts landscape from development drug the treatment eventually but at different time intervals (50). Consequently, Consequently, intervalstime (50). different at but eventually treatment the design maximizes statistical power since since power statistical maximizes design the the the EMA on initial (pre-authorization) evidence generation for new drugs (13). Naturally, Naturally, (13). drugs new for (pre-authorization) initial on generation EMA evidence temporal and financial burden for evidence generation for HTA decision making decision when HTA for generation evidence for burden and financial temporal Stepped Wedge (SW-CRCT) Wedge institutions healthcare (e.g. Stepped clusters In design. approach, this Therefore, as Therefore, Clustered Randomized Controlled Trials (CRCT)Trials examine often used to are Randomized Controlled Clustered wealth of literature is present on pragmatic clinical trials and their relevance for decision decision for clinical trials and their relevance on pragmatic is present of literature wealth clear RWE generation pathway that meet that RWEclear pathway generation of RWE for new products (e.g. in of RWE new products (e.g. for which at time to allocated randomly is cluster each deliberation on RCT evidence to be developed for new products. However, However, new products. on RCT for deliberation be developed to evidence duringdiscussed also be earlydialogues. may data) based on geographical area) are identified to take part identified are in area) on geographical based EUnetHTA work plan (47). It is also our hope that such initiatives will inform inform will It (47). plan work such initiatives that is also our hope EUnetHTA whereby MEAs, for systems coherent of more a trials such of are variant clinical practice (49). One in routine interventions of healthcare the Essentially, applied. A with making, beginning of phases I-IV development stages to an iterative, product life-cycle will be there an iterative, approach, to stages I-IV development of phases and regulatory agencies HTA between RWE harmonize need to requirements an increasing AP at as such Initiatives agencies. pave and may well another RWE for deliver to approaches and trial designs (pragmatic) Alternative decision making HTA HTA. for evidence reduce may This both between-clusters and within-clustersmeasured (51). comparisons the with several associated are SW-CRCTs that also aware are we However, RCTs. to compared their made for are recommendations before should be considered which drawbacks conduct (49;52). agencies on RWE use in decision making. This collaboration has been formalized in formalized has been collaboration This on RWEagencies making. decision use in interviewees several of this thesis, and 5-6 2-3 Chapters for conducted research throughout indicated have authors and Early European Dialogues (SEED) consortium (SEED) Dialogues (46), Early European discussions within within discussions represents an European effort establish to European an represents the clusters receive receive clusters all since equipoise clinical to relating considerations ethical circumvent SW-CRCTs be particularly may This important when treatment. receive participating eventually clusters be unethical. may conduct of RCTs whereby on orphan therapies, evidence HTA developing Additionally,

General Discussion

204 10 General Discussion 205 10 tool tool a context issue of of issue findings findings placement placement practice of practice of the the the the the 29 jurisdictions. 29 jurisdictions. complex interplay interplay complex the the single nation or continent. or continent. nation single need for multi-stakeholder multi-stakeholder need for potential decisions in trial in trial decisions potential implementation of PCTs to to of PCTs implementation a the use of RWEuse makingdecision in the the United Kingdom (57). Notably, KingdomUnited (57). Notably, quality and efficiency of clinical European and Dutch perspectives perspectives and Dutch European the authors recognize that that recognize authors the outputs of IMI-GetReal on this front IMI-GetReal of outputs front on this the the the the authors would have preferred to investigate investigate to preferred have would authors similar note, van Staa et al provide examples examples et al provide Staa van note, similar entire drug lifecycle (i.e. from early scientific early scientific from (i.e. lifecycle drug entire the boundaries of discussion sectiondiscussion above. a IMI-GetReal initiative has conducted research IMI-GetReal research conducted has initiative the CTTI to increase specifically aims the the use of RWE in HTA practice was fragmented. For For fragmented. practice was use of RWE in HTA the the policies for RWE use amongst HTA agencies were were agencies RWE for policies use amongst HTA reasons behind such strategic choices are discussed in in discussed are choices strategic such behind reasons the the series of scientific articlesseries of scientific discussing The The a research conducted examines examines conducted research authors make to address them, resonate well with well them, resonate address make to authors the governance of data access and use. access of data governance the OF RESEARCH OF experiences of implementing RWE collection and use for MEAs were only only RWE were of implementing MEAs experiences collection for and use Netherlands. Netherlands. the Registry Trial Project of Project Trial Registry publication of publication the the corresponding chapters and include practical considerations and practicalinclude and considerations chapters corresponding associated challenges and practical solutions to overcome such challenges (54). such challenges and practical overcome challenges to solutions associated post-marketing setting. Ideally, Ideally, setting. post-marketing The The the the the operational and ethical challenges associated with with associated challenges and ethical operational the research committee within Europe. However, However, Europe. within committee research approach chosen to assess to chosen approach focus of research conducted alternated between between alternated conducted of research focus authors are also aware that considerable research efforts on similar topics are occurring occurring efforts are topics on similar research considerable that aware also are authors Clinical Trials Transformation Initiative (CTTI) Initiative seeking develop ongoing initiative an to is Transformation Trials Clinical With regards to such drawbacks, such drawbacks, to regards With Another promising approach to expedited and less cumbersome generation of high- generation cumbersome less and expedited to approach promising Another the the The The LIMITATIONS LIMITATIONS reviewed for only 6 agencies in Europe, representing only 6 of only 6 of representing in Europe, 6 agencies only for reviewed Furthermore, Furthermore, for analyzed in detail of The in these to referred and have worldwide presented in Section D of this thesis, once again highlighting again highlighting Section in once D of this thesis, presented on collaboration example, we only analyzed RWE use in standard HTA (i.e. REA and CEA) and in and in CEA) REA and (i.e. RWE HTA only analyzed use in standard we example, of MEAs. Therefore, all Therefore, MEAs. of within collection methods, pertainingissues and gaps requirements, evidence defining to RWE throughout interpretation and analysis

in most chapters. For example, example, For in most chapters. The The RWE making in decision use extends beyond whereby RCTs have been embedded within EHRs in been embedded within EHRs have RCTs whereby literature from both initiatives points to challenges related to governance of individual of individual governance to related challenges to points both initiatives from literature along with challenges, Such (56;57). EHRs or within registries whether (IPD), data patient recommendations provide RWE in early drug development. One of One RWE in earlyprovide drug development. has been on leveraging registries to facilitate high-quality clinical trials at lower costs and has recently high-quality has recently and costs facilitate to lower clinical trials at registries leveraging registry-based of examples RCTs numerous To date, (56). effect this to guidance published On (56). completed been successfully have trials (55). quality RWE lies in embedding RCTs in patient registries or Electronic Health Records (EHRs). (EHRs). Electronic or Records Health registries patient in quality RWE RCTs embedding in lies The will increase practices of adoption that and drive between: pragmatic design options, study feasibility, stakeholder acceptability of trial of trial acceptability stakeholder feasibility, study options, design pragmatic between: In (53). this, to addition generalizability and of outcomes precision validity, outputs, through guide teams to (PragMagic) has been developed design, burden burden subject FDA has has FDA AP project AP project the the the the importance of highly structured structured highly the a assessment of such of such assessment full pathway of new of new full pathway post-marketing setting setting post-marketing the the regulation and assessment and assessment regulation the the same evidence requirements for for requirements same evidence more adaptive framework, whereby framework, whereby adaptive more the a ability to follow ability follow to other hand, other hand, the the finalization of MEA commitments). This may be be This may commitments). of MEA finalization near future (63). Again, Again, (63). near future role of non-RCT evidence (i.e. RWE) is prominent of non-RCT RWE)role (i.e. is prominent evidence the use of RWE for regulatory applications for medical medical for use of RWE regulatory applications for the impact of RWE generated for both stakeholders (5). (5). stakeholders both impact for of RWE generated broad variety of health technologies such as medical as medical such variety of health technologies broad the a future. the the the number of initiatives such as EUnetHTA (13), such as EUnetHTA of initiatives number a use of RWE in HTA and decision making, we propose propose making, and decision we RWEuse of in HTA IMI-ADAPTSMART consortium to allude (58) IMI-ADAPTSMART the the needs of decision-makers. needs more prominent role for RWE for making decision within role processes. prominent more regulation and assessment of drugs is embedded in embedded is drugs of assessment and regulation a future should researchers have have should researchers future outcomes of further research on this particular theme may have tangible tangible of further on this particularoutcomes have research theme may RWE landscape in the Food and Drug Administration (FDA; USA)) and HTA agencies on agencies HTA and USA)) (FDA; Drug Administration and Food The The context of AP. context the the STEPS CAN BE TAKEN TOWARDS MORE OPTIMAL USE OF OF USE OPTIMAL MORE TOWARDS TAKEN BE CAN STEPS similar note, NICE seeks to publish guidance for HTA of health applications on on applications of health HTA for guidance publish to NICE seeks note, similar the the a the discipline, HTA encompasses encompasses HTA discipline, a As Little attention was devoted to potential synergies amongst regulatory agencies (e.g. amongst regulatory (e.g. agencies synergies potential to devoted was attention Little WHICH WHICH RWE IN HTA DECISION MAKING? RECOMMENDATIONS BASED BASED RECOMMENDATIONS MAKING? DECISION HTA IN RWE FINDINGS ON (43), NEWDIGS(43), (44) and CTTI as part conducted research of IMI-GetReal and Previous (55). as partdiscussions of In order to advance advance to In order recommendations related to policy as well as methodological research. Ultimately, both both Ultimately, policy research. to as methodological as well related recommendations founded on and one another;aspects be policies should and inform should influence should research and methodological methodology in scientific advances harness recent inform aim to advice prior to regulatory approval until until regulatory prior approval to advice possible in possible effects on effects drugs in drugs an is this above, cited makingdecision As RWEof for thesis. use this and within generation within research active of field reduce to order agencies in regulatory amongst HTA and agencies such synergies of RWE increase collection and to Therefore, (60). Each phase is also associated with well-documented evidential requirements, often often requirements, evidential well-documented with associated is also Each phase (60). (60). On guidelines in institutional published in embedded is conventionally devices of medical on draft issued guidance recently smartphones in devices and wearable RCT evidence is not always mandatory (or feasible) for marketing authorization purposes marketing purposes authorization for RCT mandatory feasible) (or always not is evidence that argued be such, it may As (61). devices medical RWE different for for use examples historic numerous cited which devices On (62). framework; drug development begins with phase I trials, proceeds through to phase III phase III to through proceeds with phase I trials, begins development drug framework; IV in phase to and subsequently in human patients RCTs pivotal In fact, in this thesis. attention warranted should have and medical devices for on based not is conventionally technologies health devices, surgical procedures or even health applications on smartphones and wearable smartphones on health applications or even and wearable procedures surgical devices, (59). devices EMA and and EMA implying drugs,

General Discussion

206 10 General Discussion 207 10

Canadian Canadian European European use of RWE the the the latter, we recall recall we latter, the USA (44). implementation of such such of implementation the common responsibility of of responsibility common the the efforts of EUnetHTA cited above. above. cited efforts of EUnetHTA the post-marketing setting. To this effect, we we effect, this To setting. post-marketing results from RWE. To this effect, we recall recall we effect, this To RWE. from results main learnings from previous schemes. This This schemes. previous main learnings from product life-cycle approach, HTA agencies agencies product life-cycle HTA approach, USA (22), BioSHaRE-EU in Europe (23) and and (23) in Europe USA (22), BioSHaRE-EU the the the the the EMA on AP (43), of EUnetHTA and EMA on early dialogues and early on and dialogues EMA and of EUnetHTA on AP (43), EMA following with regards to recommendations for future future for recommendations to regards with following interpretation of interpretation main policy recommendations and briefly elaborate on each each on brieflyand elaborate policy main recommendations the REPEAT initiative (40) and others (34;35) cited above. cited others (34;35) (40) and initiative REPEAT the the the the efforts of the learnings discovered should be taken into consideration when designing and and designing when consideration should be taken into learnings discovered distribution of roles, funding of data collection and regulation of access to data data to of access collection data of funding and regulation distribution of roles, FAIR guiding principles for scientific data management and stewardship (66). and stewardship management data scientific principles guiding for FAIR efforts of impact of strategic choices regarding study parameters and analysis methods on on methods analysis and parameters study regarding choices impact of strategic implementation of MEAs to distill distill to of MEAs implementation initiatives such as FDA sentinel in sentinel as FDA such initiatives the recall recall as NEWDIGSAP in on as well (13;65), consultations parallel with coverage as price-volume, such formsof MEAs, different should cover research performance-based and development Importantly, evidence schemes. outcomes the MEAs. future implementing be Bearing RWE that in mind may generation industry pharmaceutical and regulatory agencies, HTA (e.g. stakeholders several transparency on achieve seek to should stakeholders providers), and healthcare the (international) structures for governance alternative Moreover, RWEfor repositories. To barriers. access minimize to be explored may access data HTA agencies should aim to harmonize policies regarding RWE generation and use RWE and use generation regarding policies harmonize aim to agencies should HTA recall we this effect, To making. in decision equally important it is beyond harmonization seek to However, both regulatory with agencies on RWE harmonize seek to should requirements during pre-clinical and in development Investigate which sources of RWE could be collected, using which methods and for and for methods using which of RWE be collected, could sources which Investigate IMI-GetReal upon example, touched has which policy For questions. implement to (67;68), network in meta-analyses research effectiveness (relative) for beyond outcomes long-term extrapolate (69) and to techniques scoring propensity further raised has that questions methodological research this However, (70). RCTs on guidance current Moreover, unanswered. remain the As reliance on MEAs increases worldwide, HTA agencies should invest in research on on research in invest should agencies HTA worldwide, increases MEAs on reliance As the making in decision sparse. methodologies remains understand to RWE,in trust imperative is it develop to makers decision for Inorder the impact such quantify seek to should research using RWE. studies Future from results on effect its and context. We believe that recent guideline developments citing RWE citing by developments guideline recent that believe We context. further (64). for example an collaboration provide agency (CADTH) HTA for attention In of increasing light • • • • • • Finally, we propose propose we Finally, To begin with, we list we with, begin To point below:point methodological research: methodological methodology methodology challenges cited cited challenges the outcomes of such such of outcomes the the potential implementation of RWE implementation potential the data source, guidance on methods for RWE generation and RWE and for on methods generation guidance source, data a number of issues, including: unresolved differences amongst amongst differences unresolved including: of issues, number product lifecycle and stimulate joint action amongst all relevant action joint all relevant amongst and stimulate product lifecycle a the definition of RWE, dissimilarities in policies for RWE use amongst HTA of for definition dissimilarities in policies HTA amongst RWE, RWE use number of challenges complicate complicate of challenges number a the full potential healthcare systems can bear for all citizens. for can bear systems healthcare potential full fragmented reality of RWE use in standard HTA and MEAs, inaccessibility to RWE to inaccessibility MEAs, and of RWE reality HTA in standard use fragmented other hand, numerous initiatives are ongoing to address address to ongoing are initiatives numerous other hand, a the need for alternate governance structures. governance alternate for need the use from social media for (relative) effectiveness research is lacking. Topics where where Topics is lacking. research effectiveness (relative) media for social use from authenticity, responder be of impactwould of validation include: research scientific media and cross- social from collected bias in data selection bias and information social media. collection data from sectionallongitudinal versus Despite its potential as potential its Despite challenges pertainchallenges to the • On CONCLUSION in HTA and subsequently for decision making on resource allocation in healthcare systems. systems. healthcare in allocation making decision for resource on and subsequently in HTA The on stakeholders agencies, agencies, Real-World Evidence bears promise for HTA and decision making on reimbursement of of decision making and reimbursement on HTA for bears promise Evidence Real-World However, drugs. and behind RWE collection and analysis, create awareness for alternative approaches in RWE in approaches alternative for awareness RWEbehind create collection and analysis, throughout generation and decision making. It is use of RWE optimal HTA for progressively to lead may initiatives to closer ever inch will makers decision and agencies HTA so, doing in that aspiration our realizing stakeholders for developing appropriate governance structures for RWE repositories. RWE structuresfor repositories. governance appropriate developing for stakeholders above, made recommendations policyAlongside research and above. Many of these will provide critical insights that may help improve improve help may that critical insights provide of these will Many above.

General Discussion

208 10 General Discussion 209 10 PHAROS registry. PHAROSregistry. the practical guide for using using practical guide for Development and Use of of Use and Development A the practical guide. 2015 [cited Jul Jul [cited 2015 practical guide. A Leveraging of Real-World Data to to Data Real-World of Leveraging cost effectiveness of new cancer drugs: drugs: cancer new of effectiveness cost case of oxaliplatin for treatment of stage III of treatment for oxaliplatin case of Mohseninejad L, van Gils C, Uyl-de Gils C, Mohseninejad CA, van L, Groot of patient Evaluation T. Feenstra E, Buskens supportingdecisions: registries reimbursement the 2015;18(1):84-90. Health Value cancer. colon Improve Improve 2015;18(1):127-30. Health Value Medicines. IMI-GetReal. IMI-GetReal: objectives. Overall from: DecAvailable 2016]. [cited 2017 https://www.imi-getreal.eu/About- GetReal/Overall-objectives. cycle 5 - Life Package Work EUnetHTA. Generation. Evidence improve to approach http://eunethta.eu/ from: Available 2017 activities/eunethta-joint-action-3-2016-20/ work-package-5-life-cycle-approach- improve-evidence-gener. JM, MardekianAlvir Riaz MPP, Ali D, Alemayehu of Examination et al. EM, Perfetto Mullins CD, J, Methods Proposed and Issues Analytical Data, Effectiveness Conducting Comparative for J Manag Data”. “Real-World Using Research suppl.):S3-S37. 2011;17(9 Pharm Care Infrastructure Mardekian J. D, Alemayehu Secondary in for Sources Data Requirements J Manag Research. Effectiveness Comparative A):S16-S21. Supp 2011;17(9 Pharm Care MG, Uyl-de HM, Franken Blommestein CA. Groot Freemantle N, Strack T. Real-world Real-world T. Strack N, Freemantle be should new medicines of effectiveness clinical designed appropriately by evaluated 2010;63(10):1053-8. J Clin Epidemiol trials. World Real with Value Demonstrating ABPI. Data: http://www.abpi.org. from: Available 2015]. uk/our-work/library/guidelines/Pages/real- world-data.aspx. Axelsen A, Gutteridge C, Lipset ML, Berger Optimizing Madigan D. K, P, Subedi the about decisions inform registry to data the learnedlessons from Pharmacoecon 2015;33(6):551-60. Pharmacoecon

15. 16. 17. 13. 14. 11. 12. 9. 10.

British the Academy of Medical of Academy joint meeting held on meeting held joint International Society International a the Association of Association current drug development development drug current the the LIST The The conditional marketing authorizationconditional ISPOR Real World Data Task Force Force Task Data World ISPOR Real journalof Academy of Medical Sciences. Real world of Medical Sciences. Academy pathway for oncology medicines in Europe. in Europe. medicines oncology for pathway 1;98(5):534-41. 2015 Nov Ther Clin Pharmacol Hoekman J, Boon WPC, BouvyWPC, JC, Ebbers HoekmanBoon J, of De ML. Use Bruin JP, HC, de Jong the A, Sherwood A. Does Troubat Kaaniche A, Marketing Authorisation Conditional France, in Access Influence Market in health: Value and Germany? England, the Lipska I, Hoekman J, McAuslane N, Leufkens Lipska Leufkens N, I, Hoekman McAuslane J, Does conditional AM. H+Âvels HGM, drugs in Europe oncology new for approval technology health in differences lead to Pharmacol Clin decisions? assessment 1;98(5):489-91. Nov 2015 Ther Makady A, Goettsch W. Review of Policies Review Policies of W. MakadyGoettschA, 2015 Data. on Real-World and Perspectives 2017]. Apr [cited paradigm: responding to US and European US and European to responding paradigm: of comparative evidence for demands effectiveness. relative and effectiveness 2014. Economics Office of Health The Summaryevidence: of Mestre-Ferrandiz J, Deverka P, Pistallato M, M, Pistallato Deverka J, P, Mestre-Ferrandiz E. Rosenberg Garrison LP, Neumann PJ, Erickson P, Erickson P, Neumann PJ, Garrison LP, World Real Using Mullins CD. Marshall D, Decisions: and Payment Coverage for Data The 2007;10(5):326-35. Health Value Report. Makady A, Goettsch W, Hummel N, Jonsson N, Jonsson Hummel W, Makady A, Goettsch Updated C. GetReal. C, Zuidgeest Nordon P, glossaryterms. common of of definitions https://www. from: Available 14 June 2016 imi-getreal.eu/Portals/1/Documents/ Publications/D1.3%20GetReal%20 Glossary%20-%20Update.pdf. for Pharmacoeconomics and Outcomes and Outcomes Pharmacoeconomics for 2015;18(7):A559. Research 17 September 2015 by by 2015 September 17 Sciences and Sciences Pharmaceutical Industry. 2015 September 17 September 2015 Industry. Pharmaceutical https://www. from: Oct[cited 2016]. Available acmedsci.ac.uk/viewFile/56cab22108cf9.pdf.

REFERENCE REFERENCE 8. 7. 6. 5. 4. 3. 2. 1.

cross- a Reporting of feasibility study. study. feasibility a the cervix: the joint ISPOR-ISPE Special Task Task ISPOR-ISPE Special joint Strengthening Strengthening the Innovative Medicines Initiative. Big Big Medicines Initiative. Innovative AD spectrum for better care: Multi- care: better AD spectrum for The The modal data Access Platform. 2017 [cited [cited 2017 Platform. Access modal data http://www. from: Dec Available 2016]. roadmap-alzheimer.org/index.html. Pocock M, Egger DG, Altman E, Elm Von et JP, Vandenbroucke PC, G+©tzsche SJ, al. Mao JJ, Chung A, Benton A, Hill S, Ungar L, A, Hill A, Benton Chung S, Ungar L, Mao JJ, drug of Online discussion al. et CE, Leonard among discontinuation and side effects survivors. Pharmacoepidemiol cancer breast SafDrug 2013;22(3):256-62. The respiratory and safety medicines data, for IMI Call of €93 million focus disease Dec 2016]. Available [cited 2017 proposals. http://www.imi.europa.eu/content/ from: imi-2-call-6-launch. across Outcomes Real world ROADMAP. the Brixner Eichler DL, RJ, Willke H, ML, Sox Berger real- Good practices for et al. W, H, Goettsch and/or studies of treatment data world Recommendations effectiveness: comparative from in health care evidence on real-world Force Drugdecision making. Pharmacoepidemiol Saf 2017;26(9):1033-9. ML, S, Berger Schneeweiss SV, Wang Reporting Douglas al. I, et F, Vries J, Brown and facilitate reproducibility improve to database healthcare for validityassessment Drug Pharmacoepidemiol 0. V1. studies Saf2017;26(9):1018-32. Holterheus C. Burden of Melanoma. Erasmus Melanoma. of Erasmus C. Burden Holterheus 2011. Rotterdam; University K, Basen-Engquist J, Burzawa T, Zaid Bodurka et J, Ramondetta DC, Brown LM, conduct to of social media Use al. sectional quality and epidemiologic of life surveyneuroendocrine with patients of of carcinoma Y. S, Su S, Gholizadeh Tsay Beusterien K, with colorectal experience Real-world forum web patient chemotherapies: cancer 2013;7:361. Ecancermedicalscience analysis. Observational in Epidemiology Studies Gynecol Oncol 2014;132(1):149-53. Oncol Gynecol

35. 36. 33. 34. 30. 31. 32. 28. 29.

analysis European European American American the review of of review analysis. Int Janalysis. The The United States: States: United the a the the data to to data framework for coverage coverage for framework A the United States: Lessons learned Lessons States: United the data, not data, external validity of randomised external validity of randomised review. Journal of biomedical Journal biomedical of review. opportunities and challenges. Br J Clin opportunities Br J Clin and challenges. the policy framework for public health uses of of uses health public policy for framework to approach adaptive payer pathways. Clin pathways. payer adaptive approach to Ther Pharmacol 2016;100(6):730-42. Union and journal of managed care 2015;21(9):632-40. journal care managed of Garrison Jr LP, Carlson JJ, Bajaj PS, Towse A, A, Towse PS, Bajaj JJ, Carlson Garrison LP, Jr sector Private al. et SD, Sullivan PJ, Neumann in agreements risk-sharing and prospects. barriers, trends, Faulkner SD, Lee M, Qin D, Morrell L, Xoxi E, Xoxi L, Morrell M, Qin D, Lee SD, Faulkner and reimbursement Pricing A, et al. Sammarco in and insights experiences Walker S, Sculpher M, Claxton K, Palmer Sculpher S, Claxton M, K, Palmer Walker development, with evidence S. Coverage or patient risk sharing, only in research, scheme? access 2012;15(3):570-9. Health Value decisions. Rothwell PM. Factors that can affect can affect that Factors PM. Rothwell the 2006;1(1):e9. Trial Clin PLOS trials. controlled Sloane R, Osanlou O, Lewis D, Bollegala D, Bollegala D, D, Lewis Osanlou O, Sloane R, M. Social media Maskell Pirmohamed S, and pharmacovigilance: Freifeld CC, Brownstein JS, Menone CM, Bao W, W, MenoneJS, Bao CM, Brownstein CC, Freifeld drug safety Digital et al. T, R, Kass-Hout Filice surveillance: pharmaceutical monitoring Saf Drug 2014;37(5):343-50. products in twitter. Sarker A, Ginn R, Nikfarjam A, OGÇÖConnor K, Utilizing et al. S, K, Smith Jayaraman pharmacovigilance: for data media social A the 2015;80(4):910-20. Pharmacol Gaye A, Marcon Y, Isaeva J, LaFlamme P, Turner A, Turner P, LaFlamme J, Isaeva Y, A, Marcon Gaye taking DataSHIELD: Jones EM, et al. European Commission. Clinical trials - Clinical trials - Commission. European Mar 2017 [cited EU No 536/2014. Regulation https://ec.europa. from: Available 2017]. eu/health/human-use/clinical-trials/ regulation_en#ct4. K, Rosati B. Evans D, McGraw A Pharmacoepidemiol electronic health data. SafDrug 2012;21(S1):18-22. to 2015;54:202-12. informatics Epidemiol 2014;43(6):1929-44. Epidemiol

18. 27. 26. 25. 24. 23. 22. 21. 20. 19.

General Discussion

210 10 General Discussion 211 10

pragmatic, pragmatic, ANNALS of of ANNALS a The The stepped wedge trial trial wedge stepped The The Sustainable, Multi-stakeholder Sustainable, a systematic review. BMC medical BMC medical review. systematic a Hospital Elder Life Program to to Program Elder Life Hospital American Academy of Political and and of Political American Academy meeting held March 30th, 2016. 2016. 2016. 2016. 30th, held March meeting 2012;344:e55. BMJ test. Mdege ND, Man MS, Taylor CA, Torgerson DJ. DJ. Torgerson CA, Taylor Man MS, Mdege ND, cluster wedge of stepped review Systematic is design that trials shows randomized particularly interventions evaluate used to J Clin implementation. during routine 2011;64(9):936-48. Epidemiol RJ. CA, Lilford Brown McDougall C, D, Torgerson D, Pearson one agencies, two of Parable R. Bowles randomizes. of which IMI-ADAPT SMART. Accelerated Accelerated SMART. IMI-ADAPT Patient of Appropriate Development Therapies: Treatment- to Research from Approach Dec 1 [cited December 2017 outcomes. http://adaptsmart.eu/. from: Available 2017]. Heim N, van Stel HF, Ettema RG, van der der RG, van Ettema HF, Stel Heim N, van Mast RC, SK, Inouye Schuurmans MJ. in executing Problems HELP! trial on wedge stepped randomized, the delirium patients. in older prevent 2017;18(1):220. Trials Goetz MG, Groenwold I, Zuidgeest DE. Grobbee GJ, Thiel RH, Irving van E, world trials and real Series: Pragmatic 1. Introduction. J Clin Paper evidence: 2017;88:7-13. Epidemiol Oct 2017 [cited IMI-GetReal. PragMagic. http://www.imi- from: Available 2017]. getreal.eu/Tools/PragMagic. CTTI. Initiative. Transformation Trials Clinical from: Oct [cited 2017 2017]. Available https://www.ctti-clinicaltrials.org/. Multi- CTTI. Project Trials Registry ExpertStakeholder Meeting: Summary of the M, Gulliford B, Goldacre TP, Staa van A, et al. Taweel M, Pirmohamed J, Cassell trials using routine randomised Pragmatic putting them to electronic records: health the design: design: 2006;6(1):54. methodology research the Social 2010;628(1):11-29. Science

56. 57. 58. 53. 54. 55. 51. 52. 49. 50. PCORI PCORI The The Cross-Border Patient Patient Cross-Border The The REPEAT Initiative. 2017 [cited [cited 2017 Initiative. REPEAT The The REporting using Conducted of studies Schwartz D, Lellouch J. Explanatory and J. Lellouch Schwartz D, J trials. in therapeutical attitudes pragmatic Dis 1967;20(8):637-48. Chronic EUnetHTA, European Medicines Agency. Medicines Agency. European EUnetHTA, 2017- plan: three-year work EMA-EUnetHTA Dec 2017]. 13 [cited November 2017 2020. http://www.ema.europa. from: Available eu/ema/index.jsp?curl=pages/news_and_ events/news/2017/11/news_detail_002850. jsp&mid=WC0b01ac058004d5c1#. European Medicines Agency. Scientific Scientific Medicines Agency. European efficacy on post-authorisation guidance 12-10-2016. studies. Early for Dialogues European Shaping SEED. Oct [cited 2017 2017]. technologies. health http://www.eunethta.eu/seed. from: Available European Medicines Agency. Adaptive Adaptive Medicines Agency. European Oct [cited 2017 Available 2017]. Pathways. http://www.ema.europa.eu/ema/ from: index.jsp?curl=pages/regulation/general/ general_content_000601.jsp. NEWDIGS. us: MIT NEWDIGS. About 2017 https:// from: Oct[cited 2017]. Available newdigs.mit.edu/about. EUnetHTA. About us: EUnetHTA. 2017 [cited [cited 2017 us: EUnetHTA. About EUnetHTA. http://www. from: Oct Available 2017]. eunethta.eu/about-us. JA. PARENT Mar 2017]. 2016 [cited Registries Initiative. http://patientregistries.eu/. from: Available Observational Routinely-collected health medicine PLoS statement. (RECORD) Data 2015;12(10):e1001885. Guide on Methodological ENCePP MJ-M. Takac 2011. Pharmacoepidemiology. in Standards REPEAT. http://www. from: Oct Available 2017]. repeatinitiative.org/about.html. Benchimol EI, Smeeth L, Guttmann A, A, BenchimolGuttmann L, Smeeth EI, al. et I, Petersen K, Harron Moher D, The PP-CORIM. Committee (STROBE) Statement: guidelines for reporting for guidelines (STROBE) Statement: observational Journal International studies. of Surgery 2014;12(12):1495-9. Methodology Report. 2013.

38. 37. 48. 47. 46. 45. 44. 43. 42. 41. 39. 40. data management and stewardship. and stewardship. management data 2016 Scientific Data 15;3:160018. Mar NICE, Novartis, University of Leicester. of Leicester. Novartis, University NICE, Case 1.6 (Multiple Sclerosis Deliverable https://www. from: Available 2016 Study). imi-getreal.eu/Portals/1/Documents/01%20 deliverables/Deliverable%201.6%20 Report%20-%20multiple%20sclerosis%20 case%20study_webversion.pdf. M. Lees MakadyW., A, Kalf R, Goettsch Review: Study Case WP1 D1.6 Deliverable Mar Metastatic Melanoma. [cited 2016 https://www.imi- from: Available 2017]. getreal.eu/Portals/1/Documents/01%20 WP1%20deliverables/Deliverable%20 1.6%20Report%20-%20Rheumatoid%20 Arthritis_webversion.pdf. Ericson A, Gsteiger S. Deliverable D1.6 D1.6 S. Deliverable Ericson Gsteiger A, Review: Study Case WP1 Rheumatoid Arthritis. Mar Available 2017]. [cited 2016 https://www.imi-getreal.eu/ from: Portals/1/Documents/01%20deliverables/ Deliverable%201.6%20Report%20-%20 Rheumatoid%20Arthritis_webversion.pdf. Efthimiou EM, Didden TPA, Debray N, Hummel Methodological C, et al. Fletcher M, Egger O, and illustrative recommendations guidance, (network) meta-analysis for studies case predict real-world to and modelling participant individual using effectiveness May [cited 2017 data. aggregate and/or https://www.imi- from: Available 2017]. getreal.eu/Portals/1/Documents/01%20 deliverables/2017-03-30%20-%20WP4%20 -%20Methodological%20guidance%2C%20 recommendations%20and%20 illustrative%20case%20studies.pdf.

69. 70. 68. 67.

Drug The The Economic Evaluation Economic the efficacy of medical and safety the FAIR Guiding Principles for scientific scientific for Principles Guiding FAIR of Health Technologies: Canada - 4th Edition. Canada Technologies: of Health from: Oct [cited March 2017 Available 2017]. https://www.cadth.ca/guidelines-economic- evaluation-health-technologies-canada- 4th-edition. Wilkinson MD, Dumontier M, Aalbersberg M, Aalbersberg Dumontier MD, Wilkinson Axton G, Baak A, et al. M, Appleton IJ, The CADTH. Guidelines for for Guidelines CADTH. Medicines Agency. European EUnetHTA, EMA- consultation parallel for Guidance Dec 2017]. 30 [cited 2017 June EUnetHTA. http://www.eunethta.eu/ from: Available outputs/ema-eunethta-parallel-consultation. Food & Drug Administration. Use of Real-World of Real-World Use Administration. & Drug Food Support to Evidence Regulatory Decision- 27-7-2016.Making Medical Devices. for Introduce to NICE MulryneB. Clemence J, from Process Assessment App HTA-style [cited 28 February 2017 2017. March https://www. from: Oct Available 2017]. digitalhealthdownload.com/2017/02/ nice-introduce-hta-style-app-assessment- process-march-2017/. Food and Drug Administration. Administration. Drug and Food Oct 24 [cited June 2015 Process. Development https://www.fda.gov/ from: Available 2017]. ForPatients/Approvals/Drugs/default.htm. What PR, Armeni P. R, Boscolo Tarricone assess type is needed to evidence of clinical Respiratory European devices? medical ReviewSep 2016 1;25(141):259. Banta HD, Behney CJ, Andrulid DP. DP. Andrulid Behney CJ, HD, Banta Assessing of Office Washington: technologies. 1978. Assessment Technology

64. 63. 62. 61. 60. 59. 66. 65.

General Discussion

212 10

CHAPTER Appendices 11

Appendices 217 11 - PubMed search strategy search - PubMed

I 2 - APPENDIX - 2 CHAPTER Figure Figure ((((Perspective[tiab] OR “guideline”[tiab] OR “regulation”[tiab] OR approach*[tiab] OR OR OR approach*[tiab] “regulation”[tiab] OR OR “guideline”[tiab] ((((Perspective[tiab] agency” OR “Regulatorypolicy*[tiab]) agency” AND (“HTA OR industry[tiab] OR “healthcare OR data” world payer”[tiab] AND (“real provider”[tiab] OR stakeholder*[tiab]) OR “healthcare “hospital OR data” effectiveness OR “clinical outcome” world OR “real evidence” world “real “alternative OR registry” OR “patient OR “effectiveness”[tiab] health records” OR “electronic data” OR “post-marketing design” trial” OR “observational clinical AND (“Pragmatic study design”) effectiveness OR observ*[tiab]study”(“comparative OR design*[tiab]) AND OR comparative “evidence”[tiab] assessment” effectiveness OR OR “relative research” OR “outcomes research” AND (“2005/01/01”[PDat]: effectiveness”[tiab]))) OR “comparative making”[tiab] OR “decision “2016/12/31”[PDat]))

Federal Join Committee (G-BA) Committee Join Federal European of Committee Standing Stakeholder and Care Health Institute for National (NICE) Excellence Nederland Zorginstituut Sante de Autorite Haute Quality EfficiencyInstitute and for (IQWiG) in Health Italaina Agencia de Farmaco in Technologies Drugs and Agency for Canadian (CADTH) Health Technology and Practice for Centre (USA) Assessment GlaxoSmithKline Pfizer Merk, Sharp Dohme & (MSD) Novartis Genzyme Medicines Agency (EMA) European (FDA) Drug Administration and Food The (HOPE) Federation & Healthcare Hospital European The Doctors (CPME) (ESIP) Social Platform Insurance European Nederland Zorgverzekeraars Maladie des l’Assurance de nationale Caisse salaries (CNAMTS) travailleurs Funds Insurance Health of Standing Association (GKV Spitzerband) (IAPO) Organisations of Patient Alliance International (EPF) Forum Patients’ European Research Outcomes Patient-Centered Institute (PCORI) and Society Pharmacoeconomics International for (ISPOR) Research Outcomes (CPRD) Datalink Research Practice Clinical (NEWDIGS) Paradigms Development New Drug Institute of Medicine (IOM) (HTAi) International Assessment Technology Health – HTA (NIHR) Research Health Institute for National Program Quintiles McKinzie PriceWaterhouseCooper (USA) Council Pharmaceutical National - Websites searched locate to grey literature across stakeholder 8 groups. Stakeholder Group Stakeholder Agencies (HTA) Assessment Technology Health Industry Pharmaceutical Regulatory Agencies Providers Healthcare Healthcare Payers/ Insurers Payers/ Healthcare Organisations Patient Initiatives Table I

Appendices

218 11 Appendices 219 11 summary abstract or a context of pharmaceutical of pharmaceutical context the GalenInstitute drug development, drug regulation and drug and drug drug regulation drug development, assessment. Exclusion criteria Exclusion Data World on Real focus does not Document (RWD) use in of RWD on methodology only focuses Document or synthesis, of evidence best practices analysis, synthesis. evidence full document). to no access (i.e. Document only comprises only comprises Document Stakeholder RAND Corporation Ernst & Young PatientsLikeMe (CMTP) Policy Technology Medical for Centre and Medicaid Services Medicare for (CMS) Centre Pharma for Eye Corporation Sciences Computer Industry of British Pharmaceutical Association Medicine (EAPM) Personalised for Alliance European Paraxel The document document the peer-reviewed peer-reviewed a official website of website official the scientific article,scientific article opinion or or report,guideline scientific article,scientific article, opinion a a a - Inclusion and exclusion criteria for selection of documents from PubMed and grey literature - continued - case of case of case of the the recognised institute/organisation. institute/organisation. recognised Document is Document editorial, report or guideline. editorial, Inclusion criteria in English is published Document In In editorial, it must be published in be published it must editorial, journal. Stakeholder Group Stakeholder must be published on published must be a Table I searches. Table II Table

quality of information qualityinformation of the methodology and/or software used to software and/or methodology used to lifecycle of your products? products? of your lifecycle added benefits of using added benefits of in drug RWD the the quality of RWD? the limitations of collecting and using RWD for drug of collectinglimitations using RWD and drug for the the timing? quality of the type of product? term real-world data (RWD)? (RWD)? data real-world term the possible solutions to such limitations? to solutions possible the the the quality during collected of RWD studies? are that specific definition, in your opinion, of opinion, your in definition, specific RWD? a methodology available for evidence synthesis of relative effectiveness effectiveness of relative synthesis evidence for available methodology the the type of RWD cases? in such collected evidence for relative effectiveness assessments? Do you have any suggestions suggestions any have you Do assessments? effectiveness relative for evidence real-world setting? real-world a available methodology directly be implemented? directly methodology be implemented? available the person believes RWD can positively contribute to drug development licensing and and licensing development drug to contribute positively RWDperson believes can the the collection of RWD mostly connected to mandatory obligations from EMA (e.g. risk risk collection (e.g. of RWD EMA mandatory to connected mostly from obligations disease area? disease How do you value value do you How Can Can Is RWD presently included in submission files to regulators and reimbursement agencies? reimbursement and regulators to files submission included in Is RWD presently Do you have any suggestions to improve improve to suggestions any have Do you Are there other reasons for your company to collect RWD, for example, for relative relative for example, for collect RWD, to company your for other reasons there Are assessments? effectiveness RWD be to for be likely it data would form, in what and If conditions, what under not, Does your company only collect only RWD company after marketingDoes or also premarketing authorization your specify you Could authorization? Doesit vary on depending types which of for specify you could products, not collect do If RWD your all you for collect RWD?products you is What for e.g. generally, products or more of your comparators for collected also data Is real-life a made public? Could you provide provide you Could what extent do you use RWD data in relative effectiveness modelling performed by your modelling performedyour effectiveness by use RWD in relative do you extent data what usefulness the

applicable in applicable To what extent is extent what To a. a. treatment mixed as meta-analysis, such strategies, synthesis familiar with evidence you Are value you do how and networkor meta-analysis, comparisons opinion regarding is your What What are, according to your perceptions, perceptions, your to according are, What a. b. To institution? forecast RWD and/or to marketCan in pre-licensing applications be used as evidence studies, effectiveness? clinical What are, according to your perceptions, perceptions, your to according are, What RCT example, for data? to, comparison in development, analyses? such from resulting synthesise improvements? for a. peer-reviewed in publication by instance RWD with studies for from public, made results Are journals? a. What is timing of collection of RWD in relation to collection of timing is to of RWD What relation in a. Is or evidence with (coverage part or requirements management) reimbursement of national reimbursement)? conditional Do you collect RWD for all your licensed products? Why or why not? or why Why products? licensed collect RWDyour Do all you for a. b. c. d. What is your understanding of understanding your is What a. development. And what are are what And development.

7. 5. 6. 2. 3. 4. 1. 5. 3. 4. 2. 1. market access market Perceived Perceived Extent which a to RWD

Appendices

220 11 Appendices 221 11

available available the implementation of of implementation the required data typically not typicallydata not required the pharmaceutical industrypharmaceutical group the collectionRWDof as well as credibility of predictions regarding relative effectiveness? relative of predictions regarding credibility the the decision-making process of drug development? development? of drug decision-making process the role of structured decision aids such as multiple criteria decision of structured criteria as multiple decision such aids decision role the implementation (or assessment) of statistical/mathematical models for for models statistical/mathematical of assessment) (or implementation available RWD data sources? If so, what types what of structural uncertainty If RWD so, sources? available data the the use of RWDuse of in current obstacles faced in faced obstacles current role of software in enabling efficient use of RWD (for example, efficient analysis of analysis efficient example, of use of software(for efficient in enabling role RWD routine in-house task or do you frequently need external expertise? frequently task or do you in-house routine the a is your opinion of such software? Are there any important any functionality gaps in there software? of such opinion Are or is your the the there key software that you use or that you feel is needed but currently missing? missing? currently needed but is feel you use or that you key software that there software do you currently use (if any) for evidence synthesis and/or predictive modelling? predictive and/or synthesis evidence for any) use (if currently software do you methodology directly be implemented, or is some of is some or directlymethodology be implemented, Is Do you have any suggestions for improvements? for suggestions any have Do you Is this usability software? of such Which data sources may enhance enhance may sources data Which What Are sufficient sensitivity analyses performed relative to key assumptions being made? assumptions to key sensitivity performed sufficient analyses relative Are If not, which types of required data are typically not at hand? (i.e. can typically (i.e. hand? at not are data Iftypes which not, of required at hand? at underlying structured data sets and/or statistical models (in electronic models format)? underlyingstatistical and/or sets structured data ease of use ease of use

- Interview of stakeholders to sent questionnaire a. What are are What a. is challenging How assessment? effectiveness relative in synthesis data a. is What results)? of communication efficient data, What (if any) should be (if any) What effectiveness? relative decisions on in (MCDA) analysis b. reports text-based with satisfied for evidence of you input used as an evidence RWD Are these reports be supported to by prefer you would or modelling, predictive synthesis/ the What a. What is your opinion regarding uncertainty arising from synthesising evidence for relative relative uncertainty for evidence synthesising arising opinion regarding from is your What definitions? assumptions made or parameter example, for to, due are that assessment effectiveness a. b. is that effectiveness performconsider/ to be willing of relative an assessment you Would from predicted be addressed? regarding, for example, assumptions made or parameter definitions, should primarily primarily should definitions, parameter or made assumptions example, for regarding, policies for for policies to which effortto collect and use to RWD is needed

1. 2. 3. 12. 10. 11. 9. 8. Figure II Figure Perceived Perceived Degree

available available the quality of information qualityinformation of the required data typically hand? data not at required collection RWD of post- from data the the methodology and/or software used to and/or softwaremethodology used to the quality of RWD? added benefits of using RWD for marketing of using added benefits marketing for RWD limitations of collecting and using RWD for drug of collectinglimitations using RWD and drug for the the the post-marketing setting (e.g. PASS, PAES or other or other PAES PASS, (e.g. setting post-marketing quality of a term real-world data (RWD)? (RWD)? data real-world term the possible solutions to such limitations? to solutions possible the role of structured decision aids such as multiple criteria decision criteria decision of structured as multiple such aids decision role the quality during collected of RWD studies? are that specific definition, in your opinion, of opinion, your in definition, specific RWD? the a collection of RWD officially linked to official regulatory requirements of collection regulatory of RWDrequirements to official officially linked methodology available for evidence synthesis of relative effectiveness effectiveness of relative synthesis evidence for available methodology the use of RWD as supportive in marketing applications? authorisation evidence the the the evidence for relative effectiveness assessments? Do you have any suggestions suggestions any have you Do assessments? effectiveness relative for evidence real-world setting? real-world policies of your organisation governing governing organisation of your policies a examples? types of products/ areas? disease available methodology directly be implemented? directly methodology be implemented? available the person believes RWD can positively contribute to drug development licensing and and licensing development drug to contribute positively RWDpersoncan believes is your opinion of such software? Are there any important any gaps in functionality there software? of such opinion Are or is your the sort of RWD is ideally preferred and requested for clinical efficacy clinical for assessments? sort requested and of RWD preferred ideally is requirements? ideal to sortin comparison of RWD available, currently is the you publish any guidelines on this subject?guidelines on any publish you this particularly relevant for orphan diseases? this particularly for relevant software do you currently use (if any) for evidence synthesis and/or predictive modelling? predictive and/or synthesis evidence for any) use (if currently software do you What What usability software? of such Can Can can typically hand? (i.e. not at are data If types which not, of required methodology directly be implemented, or is some of of or is some directlymethodology be implemented, How do you value value do you How improve to suggestions any have Do you Specific Specific Is Relevant Did Could you please specify? specify? please you Could What What Could you provide provide you Could what extent do you use RWD data in relative effectiveness modelling performed by your modelling performedyour effectiveness by use RWD in relative do you extent data what usefulness

your institution? your marketing studies? applicable in applicable a. be should (if any) What To what extent is extent what To a. b. What authorization submissions in comparison to, for example, RCT example, for data? to, comparison in submissions authorization a. b. What are, according to your perceptions, perceptions, your to according are, What perceptions, your to according are, What c. d. e. are What a. To which extent is extent which To a. request Do you a. b. What is your understanding of understanding your is What a. Are you familiar with evidence synthesis strategies, such as meta-analysis, mixed treatment treatment mixed as meta-analysis, such strategies, synthesis familiar with evidence you Are value you do how and networkor meta-analysis, comparisons regarding opinion is your What To institution? in RWD generated Can currently efficiency of drugs? predictreal-world observational used to be approaches) resulting from such analyses? such from resulting synthesise analysis (MCDA) in decisions on relative effectiveness? relative decisions on in (MCDA) analysis development. And what are are what And development. for improvements? for

1. 4. 2. 3. 1. 9. 7. 8. 5. 6. 3. 4. 2. RWD market access market Perceived Perceived Extent which a to

Appendices

222 11 Appendices 223 11 implementation of implementation the regulatory group agencies the collection of RWD as as well the decision-making process of drug assessment at your institution? your at assessment of drug decision-making process the implementation (or assessment) of statistical/mathematical models for for models statistical/mathematical of assessment) (or implementation the use of RWD in current obstacles faced in in faced obstacles current role of software in enabling efficient use of RWD (for example, efficient analysis of analysis efficient example, of use of software(for efficient in enabling role RWD routine in-house task or do you frequently need external expertise? frequently task or do you in-house routine the a the the there key software that you use or that you feel is needed but currently missing? missing? currently is needed but feel you use or that you softwarekey that there Is Do you have any suggestions for improvements? for suggestions any have Do you Is this ease of use use ease of - Interview of stakeholders to sent questionnaire

data, efficient communication of results)? results)? of communication efficient data, a. a. is challenging How a. is What What are are What assessment? effectiveness relative in synthesis data policies for policies for to which effortto collect use and to RWD is needed

2. 3. 1. Figure III Figure Perceived Perceived Degree

available available the quality of information qualityinformation of the required data typically not at typically data at not required purposes of decision- making for purposes making of decision- for the the methodology and/or software used to software and/or methodology used to limitations of submitting RWD for HTA RWD of submitting HTA limitations for the quality of RWD? submitted evidence, or would you prefer these these prefer you or would evidence, submitted use of RWD for reimbursement decision-making?use of RWD reimbursement for the the added benefits of using RWD for HTA submissions, submissions, added benefits of using for HTA RWD the the the quality of use of RWD data in HTA submissions at your organization? your at submissions RWDuse of HTA in data term real-world data (RWD)? (RWD)? data real-world term the underlying structured data sets and/or statistical models (in (in models statistical and/or underlying sets structured data the possible solutions to such limitations? to solutions possible the the the quality of RWD that are submitted to you? quality to submitted of RWD are that specific definition, in your opinion, of opinion, your in definition, specific RWD? a methodology available for evidence synthesis of relative effectiveness effectiveness of relative synthesis evidence for available methodology the use of RWD in HTA submissions for for submissions RWDuse of HTA in the the evidence for relative effectiveness assessments? Do you have any suggestions suggestions any have you Do assessments? effectiveness relative for evidence real-world setting? real-world a policies governing governing policies examples? types of products/ areas? disease available methodology directly be implemented? directly methodology be implemented? available person believes RWD can positively contribute to drug development licensing and and licensing development drug to contribute positively RWDperson believes can the this particularly relevant for orphan diseases? this particularly for relevant sort of RWD is ideally preferred and requested for HTA assessments? HTA for sort requested and RWD of preferred ideally is requirements? ideal to sortin comparison RWD of available, currently is the the Is this expected in reimbursement files from manufacturers? from files this expected reimbursement in this related to Coverage with Evidence Development (CED) or conditional reimbursement reimbursement or conditional (CED) Development with Evidence Coverage to this related yes, how is this assessed by your organization? your by is this assessed how yes, RWD be used as evidence in pre-licensing studies, market applications and/or to forecast forecast RWD and/or to market in pre-licensing applications used as evidence be studies, methodology directly be implemented, or is some of or is directlymethodology be implemented, hand? Can Can can typically (i.e. hand? at not are data If which types not, of required Is If How do you value value do you How improve to suggestions any have Do you Relevant Relevant regarding guidelines any publish Did you What What What Is after market authorization? Specific • Could you provide provide you Could what extent do you use RWD data in relative effectiveness modelling performed by your modelling performedyour effectiveness by use RWD in relative do you extent data what usefulness

reports supported be to by electronic format)? applicable in applicable a. b. submissions. And what are are what And submissions. is extent what To What are, according to your perceptions, perceptions, your to according are, What a. b. What are, according to your perceptions, perceptions, your to according are, What e. are What a. reportstext-based with satisfied of you Are RCT example, for data? to, in comparison Do you request request Do you a. b. c. d. What is your understanding of understanding your is What a. What is your opinion regarding opinion regarding is your What synthesise Can Can effectiveness? clinical a. b. treatment mixed as meta-analysis, such strategies, synthesis familiar with evidence you Are value you do how and networkor meta-analysis, comparisons To institution? resulting from such analyses? such from resulting reimbursement? for improvements? for

3. 4. 2. 1. 6. 7. 4. 5. 2. 3. 1. market access market Extent which a to Perceived Perceived RWD

Appendices

224 11 Appendices 225 11 implementation of of implementation the HTA agencies group agencies HTA the credibility of predictions regarding relative relative of predictions regarding credibility collectionRWDof as well as the the decision-making process of your institution? institution? of your decision-making process the role of structured decision aids such as multiple criteria decision criteria decision of structured as multiple such aids decision role the implementation (or assessment) of statistical/mathematical models for for models statistical/mathematical of assessment) (or implementation available RWD data sources? If so, what types what of structural uncertainty If RWD so, sources? available data the the use of RWDuse of in current obstacles faced in faced obstacles current role of software in enabling efficient use of RWD (for example, efficient analysis of analysis efficient example, of use of software(for efficient in enabling role RWD routine in-house task or do you frequently need external expertise? frequently task or do you in-house routine the a is your opinion of such software? Are there any important any functionality gaps in there software? of such opinion Are or is your the the there key software that you use or that you feel is needed but currently missing? missing? currently needed but is feel you use or that you key software that there software do you currently use (if any) for evidence synthesis and/or predictive modeling? predictive and/or synthesis evidence for (if any) use currently software do you usability software? of such What What Is this Is this Is Do you have any suggestions for improvements? for suggestions any have Do you Are sufficient sensitivity analyses performed relative to key assumptions being made? assumptions to key sensitivity performed sufficient analyses relative Are enhance may sources data Which effectiveness? ease of use ease of use - Interview of stakeholders to sent questionnaire

data synthesis in relative effectiveness assessment? assessment? effectiveness relative in synthesis data a. is What results)? of communication efficient data, a. analysis (MCDA) in decisions on relative effectiveness? relative decisions on in (MCDA) analysis are What a. is challenging How What What a. should be (if any) What What is your opinion regarding uncertainty arising from synthesising evidence for relative relative uncertainty for evidence synthesising arisingopinion regarding from is your What or parameter made assumptions example, for to, due are that assessment effectiveness definitions? a. b. Would you be willing to consider/ perform an assessment of relative effectiveness that is is that effectiveness perform consider/ to be willing of relative an assessment you Would from predicted regarding, for example, assumptions made or parameter definitions, should primarily be primarily should be definitions, or parameter made assumptions example, for regarding, addressed? policies for for policies to which effortto collect and use to RWD is needed

3. 1. 2. 10. 11. 9. 8. Figure IV Figure Perceived Perceived Degree lysis, lysis, D) use D) use drug drug drug drug i.e. noi.e. i.e. noi.e. ynthesis. ynthesis. E2: n=5 E3: n=1 E1: n=41 E1: n=449 Exclusion Criteria: Exclusion Criteria: access to full document). access to full document). regulation and drug assessment. drug and regulation regulation and drug assessment. drug and regulation Dueto Exclusion Criteria (N=455) Dueto Exclusion Criteria (N=455) in the context of pharmaceutical drug development, development, drug pharmaceutical of context the in in the context of pharmaceutical drug development, development, drug pharmaceutical of context the in best practices of evidence synthesis, or evidence s evidence or synthesis, evidence of practices best best practices of evidence synthesis, or evidence s evidence or synthesis, evidence of practices best E3: Document only comprises a summary or abstract ( abstract or a summary comprises only Document E3: E3: Document only comprises a summary or abstract ( abstract or a summary comprises only Document E3: E1: Document does not focus on Real World Data (RW Data on Real World not focus Document E1: does E1: Document does not focus on Real World Data (RW Data on Real World not focus Document E1: does E2: Document only focuses on methodology of RWD ana RWD of on focuses methodology Document only E2: E2: Document only focuses on methodology of RWD ana RWD of on focuses methodology Document only E2: (N=47) 66 Hits (N=462) 496 Hits Excluded Excluded Excluded Excluded Included (N=34) Included (N=19) journal. journal. nstitute/ nstitute/ ent must be be must ent ent must be be must ent I2: n=6 I1: n=7 organisation. organisation. Inclusion Criteria: Inclusion Criteria: report or guideline. report or guideline. Dueto Inclusion Criteria (N=6) Dueto Inclusion Criteria (N=7) I1: Document publishedis in English I1: Document publishedis in English I3: In the case of a scientific article, opinion article opinion or article, a scientific of case the In I3: I3: In the case of a scientific article, opinion article opinion or article, a scientific of case the In I3: editorial, it must be published in a peer-reviewed peer-reviewed a in published be must it editorial, editorial, it must be published in a peer-reviewed peer-reviewed a in published be must it editorial, published on the official website of a recognised i recognised of a website official the on published published on the official website of a recognised i recognised of a website official the on published I2: Document ais scientific article, opinion article,editorial, I2: Document ais scientific article, opinion article,editorial, I4: In the case of a guideline or report, the docum the report, or a guideline of case the In I4: I4: In the case of a guideline or report, the docum the report, or a guideline of case the In I4:

- Inclusion and exclusion of documents retrieved from grey literature. grey from retrieved documents of - Inclusion and exclusion

31.12.2016) - Inclusion and exclusion of documents retrieved through PubMed search. PubMed through retrieved documents of - Inclusion and exclusion

Exclusion Applied Screening Applied Screening

(01.01.2005 – – (01.01.2005 Search Results Search Reasons for Exclusion for Reasons VI

Results of of Results Reasons for for Reasons Inclusion & Exclusion Criteria Criteria Exclusion & Inclusion Results of of Results Inclusion & Exclusion Criteria Criteria Exclusion & Inclusion Search Results Results Search Figure Figure Figure V Figure

Appendices

226 11 Appendices 227 11 evolution of of evolution practical guide. practical guide. A the regulator’s regulator’s a next in step case for action. case for European Parliamant and of and of Parliamant European future of drug development. drug development. of future the A pharmaceutical business model. business pharmaceutical Pharmaceutical IndustryPharmaceutical Perspective the Federal Food, Drug, and Cosmetic Act. and Cosmetic Drug, Food, Federal the the the Conduct of Pharmacoepidemiological and and Conduct of Pharmacoepidemiological the growing Need. growing the Leveraging of Real-World Data to Improve Improve to Data Real-World of Leveraging A efficacy-effectiveness gap: the the ENCePP Code of Conduct for Scientific Independence and Scientific and Independence of Conduct for Code ENCePP ISPOR Real-World Data Task Force Report Force Task Data ISPOR Real-World Development and Use of Medicines Use and Development Council flexible blueprint for blueprint flexible framework for understanding cancer comparative comparative cancer understanding for framework A Document Title Document Methods Proposed and Issues Analytical of Data, Examination “Real- Using Research Effectiveness Conducting Comparative for Data” World Data: Real-Life drug approval Bridging Demonstrating Value with Real World Data: Data: World with Real Value Demonstrating Research Effectiveness Comparative Optimizing the A needs data research effectiveness Republic 2008 Czech in data Real-world from CER Looking at taking Licensing: Adaptive perspective on addressing variability response of drug perspective addressing on practices (GPP) pharmacoepidemiology good for Guidelines Legislature. 2014-2019 Paper Briefing MEP’s of 2010/84/EU Directive the The The in Transparency Pharmacovigilance Studies. Studies. Pharmacovigilance 2020 Report: for in Europe health Redesigning Force Task eHealth are data World Report, Data Real How 2013-2014: World Real change used to being Real-World Data Research: Research: Data Real-World Best Practices for Conducting Reporting and for BestPractices Electronic Using Studies Safety Pharmacoepidemiologic Data. Healthcare Implementation of - Trials and Clinical Studies Postmarketing Section of 505(o)(3) Real-world effectiveness of new medicines should be evaluated be evaluated should medicines of new effectiveness Real-world clinical trials designed appropriately by Research: Effectiveness Comparative for Data Medicare Using Opportunities Challenges and Decisions: and Payment Coverage for Data Real-World Using The Date of of Date Publication 2011 2007 2011 2011 2010 2014 2010 2012 2008 2012 2012 2007 2014 2010 2010 2012 2014 2013 2013 2011 2010 2011 2007 - List of documents selected from academic and grey literature search for analysis. Primary Author D. Alemayehu, L. Annemans, Eichler, H.G. Eichler, Association of British of British Association Pharmaceutical Industry M. Berger, M. Berger, Barker, R. Barker, W. Carpenter, Doležal,T R. Dubois, H. G. Eichler, Epstein, M. Epstein, Alliance European Personalised for Medicine European Commission European Medicines Medicines European Agency European Union European Pharma for Eye Foltz, D. Foltz, Food and Drug Drug and Food Administration Drug and Food Administration Freemantle, N. Freemantle, V. Fung, Garrison, L. Table III Table

use of use of the

Italian registries: Italian registries: The The key imperative for improving improving for imperative key A health decision maker perspective. maker perspective. decision health evidence for decisions about decisions for evidence the use of Observationaluse of Data the tight timeframe: stakeholder perspectives on perspectives stakeholder on timeframe: tight the Clinical Utility of Comparative UtilityClinical of Comparative a Value of Observational of Research. Value methods of technology appraisal 2013 appraisal of technology methods the the Slovak republic Slovak the the Reality of Real World Data and its Use in Health Care Care in Health and its Use Data World Reality of Real efficacy of relative and effectiveness of comparative future tall order on order tall model to evaluate appropriateness and effectiveness appropriateness evaluate model to in Poland A Document Title Document Data± Health Federal to Accesss care. health and health evaluations of economic transferability and data Real-world comparative effectiveness research using electronic using clinical data research effectiveness comparative provision healthcare for financing patient-focused Towards Modelling Paper: Working MethodsGeneral in Hungary Evaluation Pharmacoeconomic for Data World Real The Decisions in Europe research Real world and public and for drug development for of big data Use and care health personal partner organizations with manufacturers care managed Can research effectiveness comparative for Real- Unique Leverage to Healthcare Maccabi and Israel’s Merck Health Approaches Novel Inform to Database World The perspective. an international drugs± Perspective Medicare’s from CER Looking at Effectiveness Comparative Understanding Guide to Practical (CER) Research Guide to Leaders in Clinical Trial Data Transparency Data Trial in Clinical Leaders Introduction to worth? they are much How New drugs: a Unlocking Evaluations: in Pharmacoeconomic Data World Real Using OpportunitiesChallenges, and Approaches on De testimonio: therapeutic interventions. therapeutic from data Real-world What are we talking we about? are What Maintopics Sanofi CSR on positions decision pricing and reimbursement for data real-world Using within and research medicine for its promise and Dara World Real Maximizing Research Effectiveness 2012 Date of of Date Publication 2014 2008 2013 2009 2013 2008 2011 2010 2016 2008 2013 2015 2012 2013 2013 2014 2013 2012 2012 2013 2008 2013 2013 2008 2014 2010 - continued - Holve Primary Author Healthcare Council Leadership T. Heranowski, HOPE IQWiG IQWiG Kaló,Z. P. Keohane, J. Knottnerus, L. Leyens B. Luce, Merck D. Messner, P. Mohr, J.G. Neely, NICE Novartis N. Olson, A. Palozzo, Paraxel A. M. Pleil, M. Rawlins, Romio, S. Romio, Sanofi T. Tesar, G. M. Turner, C.A. Umscheid, Table III Table

Appendices

228 11 Appendices 229 11 Number of interviewees 2 3 2 2 1 2 2 3 1 2 1 1 1 1 1 1 1 1 1 1 Stakeholder Agency A HTA Agency B HTA Agency C HTA Agency D HTA Agency E HTA Industry A Industry B Industry C Industry D Regulatory Agency A Regulatory Agency B A Academia B Academia C Academia A Insurer Payer/ A Provider A Organisation Patient B Organisation Patient A Initiative B Initiative Document Title Document and reimbursement in information effectiveness of relative Use pricing Europe in decisions practice learning and from data Real-life 8.4 Paper Background innovation advance to Medicaid into research effectiveness comparative Translating policy:payment pharmacy and medical from views directors. Date of of Date Publication 2013 2013 2015 - Overview of interviews conducted per stakeholder group. stakeholder per conducted Overview interviews - of - continued - Primary Author F. Nooten, van P. T. Staa, van J. Weissman, Health Technology Assessment (HTA) Agencies (HTA) Assessment Technology Health IndustryPharmaceutical Regulatory Agencies Academia Insurers Payers/ Healthcare Provider Healthcare Organisations Patient Initiatives Stakeholder Group Stakeholder Table III Table Table IV Table

purposes makingof decision- the limitations of submitting RWD for HTA RWD of submitting HTA limitations for quality of RWD? submitted evidence, or would you prefer these these prefer you or would evidence, submitted use of RWD for reimbursement decision-making?use of RWD reimbursement for the the added benefits of using RWD for HTA submissions, submissions, added benefits of using for HTA RWD the the the use of RWD data in HTA submissions at your organization? your at submissions RWDuse of HTA in data term real-world data (RWD)? (RWD)? data real-world term underlying structured data sets and/or statistical models (in (in models statistical and/or underlying sets structured data the possible solutions to such limitations? to solutions possible the the the quality of RWD that are submitted to you? quality to submitted of RWD are that specific definition, in your opinion, of opinion, your in definition, specific RWD? a the use of RWD in HTA submissions for submissions for use of RWD in HTA the policies governing governing policies examples? types of products/ areas? disease person believes RWD can positively contribute to drug development licensing and and licensing development drug to contribute positively RWDperson believes can this particularly relevant for orphan diseases? this particularly for relevant sort of RWD is ideally preferred and requested for HTA assessments? HTA for sort requested and RWD of preferred ideally is requirements? ideal to sortin comparison RWD of available, currently is the Is this expected in reimbursement files from manufacturers? from files this expected reimbursement in this related to Coverage with Evidence Development (CED) or conditional reimbursement reimbursement or conditional (CED) Development with Evidence Coverage to this related yes, how is this assessed by your organization? your by is this assessed how yes, RWD be used as evidence in pre-licensing studies, market applications and/or to forecast forecast RWD and/or to market in pre-licensing applications used as evidence be studies, Is If How do you value value do you How improve to suggestions any have Do you Relevant Relevant regarding guidelines publish any Did you What What What Is after market authorization? Specific • Could you provide provide you Could what extent do you use RWD data in relative effectiveness modelling performed modelling effectiveness by use RWD in relative do you extent data what usefulness

reports supported be to by electronic format)? in comparison to, for example, RCT example, for data? to, in comparison perceptions, your to according are, What are what And submissions. a. b. What are, according to your perceptions, perceptions, your to according are, What for reimbursement? for e. are What a. reportstext-based with satisfied of you Are a. b. c. d. What is your understanding of understanding your is What a. request Do you Can Can effectiveness? clinical a. b. To your institution? your strategy used for searching PubMed. searching used for strategy

3. 4. 1. 2. 4. 2. 3. 1. CHAPTER 3 - APPENDIX - 3 CHAPTER market access market Extent which a to Perceived Perceived Search Search (((((Zorginstituut Nederland[Affiliation]) OR Agenzia Italiana del Farmaco[Affiliation]) OR Haute Autorite Autorite Haute OR Farmaco[Affiliation]) Agenzia OR Italiana del Nederland[Affiliation]) (((((Zorginstituut Wirtschaftlichkeit im Gesundheitswesen[Affiliation]) OR und fur Institut Qualitat de Sante[Affiliation]) Tandvards-lakemedel OR Excellence[Affiliation])) Care AND Health[Affiliation] Institute for OR (National : "2016/06/21"[PDat])) AND (("2006/01/01"[PDat] sformansverket[Affiliation] RWD Interview Questionnaire

Appendices

230 11 Appendices 231 11 available available the implementation of of implementation the quality of information qualityinformation of the required data typically hand? data not at required the methodology and/or software used to and/or softwaremethodology used to the collectionRWDof as well as credibility of predictions regarding relative effectiveness? relative of predictions regarding credibility the quality of the the decision-making process of your institution? institution? of your decision-making process the role of structured decision aids such as multiple criteria decision criteria decision of structured as multiple such aids decision role the implementation (or assessment) of statistical/mathematical models for for models statistical/mathematical of assessment) (or implementation methodology available for evidence synthesis of relative effectiveness effectiveness of relative synthesis evidence for available methodology the the use of RWDuse of in evidence for relative effectiveness assessments? Do you have any suggestions suggestions any have you Do assessments? effectiveness relative for evidence real-world setting? real-world a current obstacles faced in faced obstacles current available methodology directly be implemented? directly methodology be implemented? available role of software in enabling efficient use of RWD (for example, efficient analysis of analysis efficient example, of use of software(for efficient in enabling role RWD routine in-house task or do you frequently need external expertise? frequently task or do you in-house routine the the a is your opinion of such software? Are there any important any gaps in functionality there software? of such opinion Are or is your available RWD data sources? If so, what types of structural uncertainty regarding, for for types what structuralof uncertainty If RWD so, regarding, sources? available data the the the there key software that you use or that you feel is needed but currently missing? missing? currently needed but is feel you use or that you key software that there software do you currently use (if any) for evidence synthesis and/or predictive modeling? predictive and/or synthesis evidence for (if any) use currently software do you the methodology directly be implemented, or is some of of some or is directlymethodology be implemented, being made? assumptions to key sensitivity performed sufficient analyses relative Are enhance may sources data Which What usability software? of such If not, which types of required data are typically not at hand? (i.e. can can typically hand? (i.e. not at are data If types which not, of required Is this Is this Is Do you have any suggestions for improvements? for suggestions any have Do you Can Can ease of use ease of use

- Interview interviews. guide used to questionnaire data, efficient communication of results)? results)? of communication efficient data, How challenging is is challenging How a. is What a. What are are What a. assessment? effectiveness relative in synthesis data a. b. What a. should be (if any) What effectiveness? relative decisions on in (MCDA) analysis Would you be willing to consider/ perform an assessment of relative effectiveness that is predicted that effectiveness perform consider/ to be willing of relative an assessment you Would from primarily should be addressed? definitions, made or parameter assumptions example, relative uncertainty for evidence synthesising regarding opinion arising from is your What definitions? made or parameter assumptions example, for to, due are that assessment effectiveness To what extent is extent what To in applicable a. b. Are you familiar with evidence synthesis strategies, such as meta-analysis, mixed treatment treatment mixed as meta-analysis, such strategies, synthesis with evidence familiar you Are value you do how and networkor meta-analysis, comparisons opinion regarding is your What synthesise improvements? for resulting from such analyses? such from resulting policies for for policies to which effortto collect and use to RWD is needed

2. 3. 1. 10. 11. 8. 9. 7. 5. 6. Figure I Figure Perceived Perceived Degree positive positive a appraisal? appraisal? (please describe(please per study) the recommendations section identifying section identifying recommendations effectiveness-assessment? effectiveness-assessment? cost-effectiveness-assessment? different parameters (observational, registries, registries, (observational, parameters different the the the role of data derived from RWD. from derived data of role recommendations section identifying recommendations the real-world data? data? real-world the the ICER? ICER? the RWD decision-making? for the role of data derived from RWD. from derived data of role the the reason for inclusion? for inclusion? reason for inclusion? reason (prospective/retrospective, controlled, randomised, blinded, etc.) blinded, randomised, controlled, (prospective/retrospective, the the impact of characteristics of life (improves morbidity or mortality) (improves life the the (patient, intervention, comparison and outcome) and comparison intervention, (patient, indication was used under assessment? under assessment? used was indication negative opinion regarding regarding opinion negative model used to estimate estimate to model used a Positive impact, statement in impact, statement Positive in impact, statement Negative Study-design PICO of patients Number Extend QoL or symptoms Improve Other(prophylaxis) opinion regarding regarding opinion so, for which parameters was RWD used? (e.g. prevalence, effectiveness, cost- effectiveness, prevalence, RWD (e.g. used? was parameters which for so, a Ű Ű Ű Ű Ű Ű Ű Ű PCTs, etc.)? PCTs, What was was What Ű Ű Is information originating from RWD in from mentioned originating Is information Are real-world data (RWD) data in included real-world Are what was If so, Is If costs) and/or effectiveness used for Which were type of sources Ű Ű Ű Are real-world data (RWD) data in included real-world Are what was If so, are What Which Which Goalof treatment: Ű Ű Ű Topic (drugs) Topic Country of reportURL (?) of extractionDate/data recommendation HTA of Date

1. 2. 1. 2. 3. 6. 7. 1. 2. 3. 4. 5. 2. 1. 3. 4. 5. 4 - APPENDIX - 4 CHAPTER Effectiveness Assessment 1: Data-extraction Appendix (DEF) form information General Appraisal Effectiveness Cost-effectiveness

Appendices

232 11 Appendices 233 11 neutral neutral positive positive negative negative a a a neutral opinion opinion neutral a primary reason for primary for reason the appraisal? appraisal? the assessment. assessment. recommendations section, regarding section, regarding recommendations section, regarding recommendations dossier for effectiveness? effectiveness? for dossier cost-effectiveness? for dossier the the the the the the recommendations section that cannot clearly section clearly cannot that recommendations recommendations section identifying recommendations the recommendations section identifying recommendations the recommendations section identifying recommendations recommendations section cannot clearly be that recommendations RWD decision-making? for recommendations section identifying recommendations the the the the role of data derived from RWD. from derived data of role RWD. from derived data of role role of data derived from RWD. from derived data of role the the the the role of data derived from RWD. from derived data of role negative recommendation, what was was what recommendation, negative appraisal of appraisal of recommendation final of recommendation final the a the the the benefit or added benefit not proven benefit not or added benefit RWD. RWD. negative recommendation? recommendation? negative regarding regarding Neutral, statement in statement Neutral, Negative cost-effectiveness Negative Clinical Cost/cost-effectiveness Both Other Not included, RWD included in not Not included, was benefit or added Positive Equal benefit or lesser Negative cost-effectiveness Positive identified as positive, negative or neutral. negative as positive, identified be identified as positive, negative or neutral. or negative positive, as be identified in statement Positive, regarding opinion in statement Negative, regarding opinion Impact, in statement in no statement Not identified, Neutral impact, statement in impact,Neutral statement Impact in unknown, statement in statement no Not identified, the RWD in included not Not included, was the opinion regarding regarding opinion Ű Ű Ű Ű Ű Ű Ű Ű Ű Ű Ű Ű Ű Ű Ű Ű Ű Ű Ű the Ű Ű Ű In case of Ű Ű Ű Ű Ű Ű was What Ű Ű was What Ű Ű Is information originating from RWD in from mentioned originating Is information was What Ű Ű Ű Ű Ű Ű

3. 2. 1. 1. 2. Final Cost-effectiveness

Fleiss’ Fleiss’ the score of 1 indicates indicates 1 of score Date of publication of Date 12-12-2012 09-03-2015 publication 12-11-2015 14-12-2011 of Date 23-07-2014 12-12-2016 11-12-2015 03-10-2012 a remaining 44 reports were 44 reportsremaining were the

Drug Ipilimumab Pembrolizumab Nivolumab Vemurafenib Drug Vemurafenib Dabrafenib Pembrolizumab Ipilimumab score of 0 indicates poor agreement and agreement poor indicates 0 of score Scotland (HIS) (NICE) Excellence and Care Health Institute for National MedicinesScottish Consortium(SMC), Improvement Healthcare (IQWiG) Scotland Wirtschaflichkeit im Institut und für Qualität Gesundheitswesen (HAS) Santé de Autorité Haute Zorginstituut Nederland (ZIN) Zorginstituut a

inter-rater reliability (IRR) was calculated twice in 2 different rounds. In each round, round, In each rounds. twice different in 2 calculated (IRR) was reliability inter-rater

round round round round third, independent researcher. Once IRR was established, established, IRR was Once researcher. independent third, nd st perfect agreement[11]. Authors’ extraction for open-ended questions was compared by by extraction open-ended compared for questions was perfect Authors’ agreement[11]. a authors. both amongst divided equally kappa method, whereby kappawhereby method, d’Andon Anne Wieseler Beate Hedi SchellemanHedi Jonsson Pall Moseley Owen 3: Health Technology Assessment (HTA) panel members (HTA) Assessment Technology 3: Health Appendix List of panel-members IQWiG HAS Agency NICE SMC HAS 2 NICE SMC IQWiG 1 Agency List of reports used to calculate IRR List of reports calculate used to 2: Inter-Rater Reliability2: Inter-Rater Appendix The reports 4 randomly-selected for extracted list below (see independently authors from data using compared extraction were closed questions for reports Authors’ per round).

Appendices

234 11 Appendices 235 11 13-06-2012 11-12-2013 10-09-2012 09-12-2013 24-02-2014 12-12-2012 Date of publication of Date 27-04-2012 13-03-2014 26-03-2014 16-05-2014 22-06-2016 26-10-2016 18-02-2016 27-07-2016 01-10-2015 25-11-2015 14-05-2012 08-04-2013 10-11-2014 09-03-2015 12-09-2016 07-03-2016 08-08-2016 07-11-2016 09-11-2015 09-11-2015 12-12-2016 10-02-2012 12-12-2012 23-07-2014 22-10-2014

Ipilimumab (Second Addendum) (Second Ipilimumab Vemurafenib Vemurafenib Ipilimumab (First Addendum) (First Ipilimumab Ipilimumab Ipilimumab Pembrolizumab (treated) Pembrolizumab IQWiG Pembrolizumab (treated) (untreated) Pembrolizumab Pembrolizumab Nivolumab Nivolumab with ipilimumab) (combined Nivolumab Vemurafenib Dabrafenib with dabrafenib) (combined Trametinib Ipilimumab Ipilimumab Vemurafenib SMC Ipilimumab Pembrolizumab (treated) (untreated) Pembrolizumab Pembrolizumab Cobimetinib (combined with vemurafenib) with (combined Cobimetinib Nivolumab with ipilimumab) (combined Nivolumab Dabrafenib with dabrafenib) (combined Trametinib Ipilimumab Ipilimumab Vemurafenib Vemurafenib NICE ZIN Ipilimumab 4: Health Technology Assessment (HTA) reports included (HTA) Assessment Technology 4: Health Appendix reportsList of included 03-10-2012 Date of publication of Date 21-11-2016 12-11-2015 14-01-2016 14-12-2011 19-11-2014 07-05-2014 20-01-2016 16-03-2016 13-01-2016 16-03-2016 23-12-2013 14-03-2014 28-12-2015 24-02-2016 28-12-2015 11-03-2016 12-05-2016 13-10-2015 11-12-2016 12-09-2016

Cobimetinib (Combined with vemurafenib) (Combined Cobimetinib Nivolumab Pembrolizumab Vemurafenib Dabrafenib dabrafenib) with combined or (Alone Trametinib HAS Ipilimumab Ipilimumab Pembrolizumab (Addendum) Pembrolizumab Nivolumab (Addendum) Nivolumab Nivolumab Pembrolizumab Nivolumab (Addendum) vemurafenib) with (combined Cobimetinib Nivolumab Nivolumab Dabrafenib (Combined with trametinib – Addendum) with trametinib (Combined Dabrafenib dabrafenib) with combined or (Alone Trametinib Cobimetinib Dabrafenib (Addendum) Dabrafenib with trametinib) (Combined Dabrafenib List of included reportsList of included Dabrafenib

Appendices

236 11 Appendices 237 11 No Yes No Yes No Yes No Yes 9) 9)

= =

(n (n (n (n 19) 19) 2016 2016 5) 2016 5) 2016

= =

(n (n (n (n 10) 10) 2015 2015 4) 2015 4) 2015

= =

CEAs)

REAs) CEAs) REAs)

(n (n of of of

of 10)

10)

2) 2014 4) 2014 2) 2014 4) 2014

number = number number = = number

= = = =

(n (n (n (n (n (n (n

Year Year Year Year 5) 2013 8)2013 5) 2013 8)2013

= = = =

(n (n (n (n 0) 2012 1)2012 0) 2012 1)2012

= = = =

(n (n (n (n 2011 2011 2011 2011 0% 0% 0% 0% – Inclusion of Real-World Data (RWD) in cost-effectiveness assessments (CEAs) over time. over (CEAs) (RWD) assessments Data in cost-effectiveness – Inclusion of Real-World - Inclusion of Real-World Data (RWD) in relative effectiveness assessments (REAs) over time. over (REAs) assessments effectiveness (RWD) Data relative in - Inclusion of Real-World

90% 80% 70% 60% 50% 40% 30% 20% 10% 90% 80% 70% 60% 50% 40% 30% 20% 10%

100% 100%

100% 100% Percentage Percentage Percentage Percentage Figure S2 - Inclusion of RWD in CEAs over time Figure S2 - Inclusion of RWD in CEAs over Figure S1 - Inclusion of RWD in REAs over time. - Inclusion of RWD Figure S1 Figure S2 - Inclusion of RWD in CEAs over time Figure S2 - Inclusion of RWD in CEAs over Figure S1 - Inclusion of RWD in REAs over time. - Inclusion of RWD Figure S1 Figure S2 Figure Figure S1 Figure 5: Supplementary Figures S1 & S2 and Table S1 Table S1 & S2 and 5: Supplementary Appendix Figures https://www.nice.org.uk/ guidance/ta268/documents/ melanoma-stage-iii-or- iv-ipilimumab-updated- analysis-with-revised- patient-access-scheme- from-bristol-myerssquibb2 Reference http://file.scirp.org/pdf/ JCT_2014102311333330.pdf

clinical clinical McKesson McKesson the Prospective Prospective observational cohort; non- study. randomised Study Design Study Retrospective Retrospective electronic health (EHR) record observational cohort study of treated patients with first-line ipilimumab monotherapy in practices of practices of the Specialty Health/ US Oncology Network (MSH/USON). hospital/ a

iKM source source iKM

the

primary of measure the time from starttime from of ipilimumab the Overall survivalOverall Outcome(s) measured Outcome(s) 1) Variables to describe patient describe patient to Variables 1) characteristics and disease demographic primary race, age, gender, included initial time since diagnosis, at site at disease stage diagnosis, melanoma location start treatment, of ipilimumab performance status ECOG of metastases, serum metastases, brain of presence (PS), and (LDH) level, lactate dehydrogenase status. BRAF mutation of care patterns evaluate to Variables 2) treatment dosing, ipilimumab included for reasons and discontinuations, delays and concomitant termination, treatment supportive and subsequent and therapy. anticancer was 3) OS defined was and effectiveness treatment as was Safety cause. any from death until reported AEs. by assessed 4) Because AE grade/intensity treatment- causality (e.g. information, occurrence of date and or not), related in available not were reported irrespective are AE data data, any However, or causality. of grade during AE reported occurred that Any captured. was treatment ipilimumab as serious in AEs (SAEs) AEs documented charts to or leading patients’ emergency room visit or death were were or death visit emergency room as SAEs. defined Comparator(s) 2 different 2 different doses of ipilimumab N/A

(1) ipilimumab 10mg/kg 10mg/kg (1) ipilimumab 4 doses every for 3 weeks by (induction), followed ipilimumab doses of additional every (maintenance) 12 weeks every 3mg/kg (2) ipilimumab 3 (induction), doses 4 for weeks re-induction by followed if necessary. Intervention(s) ipilimumab monotherapy ipilimumab entire entire the clinical trial, trial, clinical a first-line setting during April 1, duringApril setting 1, first-line the Italian subset of European patient patient Italian European subset of setting. care routine from population Eligible patients were required to have have to required were patients Eligible started have old, years be ≥18 AM, mg/kg 3 at monotherapy ipilimumab in Patient Population Patient 2011 to September 30, 2012 and received and received 2012 30, September to 2011 Comprehensive an MSH/USON at care used that Network site Alliance Strategic over iKM EHR capabilities full study period. Ex-clusion criteria included Ex-clusionstudy period. criteria included current AM, for treatment prior systemic in enrollment or pending and treatment for other cancers. cancers. other for and treatment

primary community community a Summary of RWD study characteristics The The to objectives were describe 1) patient demographic and disease characteristics of care 2) patterns of patients 4) AEs with treatment- who AM naïve with treated were 3 mg/ ipilimu-mab kg monotherapy in practice setting 3) survival outcomes Early Access EarlyAccess (Italian Programme find to subset) dosing optimal Aim – Studies used provide to real-world data on (RWD) effectiveness and safety of new drugs in relative Study Name Study CA184-332 CA184-089 effectiveness assessments(REAs). Table S1

Appendices

238 11 Appendices 239 11 https://www.nice.org.uk/ guidance/ta268/documents/ melanoma-stage-iii-or- iv-ipilimumab-updated- analysis-with-revised- patient-access-scheme- from-bristol-myerssquibb2 http://file.scirp.org/pdf/ JCT_2014102311333330.pdf Reference

Study Design Study Prospective Prospective observational cohort; non- study. randomised Retrospective Retrospective electronic health (EHR) record observational cohort study of treated patients with first-line ipilimumab monotherapy in the clinical practices of the McKesson Specialty Health/ US Oncology Network (MSH/USON). a hospital/

iKM source source iKM

the

primary of measure the time from starttime from of ipilimumab Outcome(s) measured Outcome(s) 1) Variables to describe patient describe patient to Variables 1) characteristics and disease demographic primary race, age, gender, included initial time since diagnosis, at site at disease stage diagnosis, melanoma start location treatment, of ipilimumab performancestatus ECOG of metastases, serum metastases, brain of presence (PS), and (LDH) level, lactate dehydrogenase status. BRAF mutation Overall survivalOverall of care patterns evaluate to Variables 2) treatment dosing, ipilimumab included for reasons and discontinuations, delays and concomitant termination, treatment supportive and subsequent and therapy. anticancer reported irrespective are AE data data, any However, or causality. of grade during AE reported occurred that Any captured. was treatment ipilimumab serious as in AEs (SAEs) AEs documented charts to or leading patients’ 3) OS was was 3) OS 4) Because AE grade/intensity treatment- causality (e.g. information, occurrence of date and or not), related in available not were were or death visit emergency room as SAEs. defined treatment effectiveness and was defined defined was and effectiveness treatment as the was Safety cause. any from death until reported AEs. by assessed Comparator(s) N/A ipilimumab doses of doses of 2 different 2 different

(1) ipilimumab 10mg/kg 10mg/kg (1) ipilimumab 4 doses every for 3 weeks by (induction), followed ipilimumab doses of additional every (maintenance) 12 weeks every 3mg/kg (2) ipilimumab 3 (induction), doses 4 for weeks re-induction by followed if necessary. Intervention(s) ipilimumab monotherapy ipilimumab the entire the entire clinical trial, trial, clinical a first-line setting during April 1, duringApril setting 1, first-line the Italian subset of European patient patient Italian European subset of setting. care routine from population have to required were patients Eligible started have old, years be ≥18 AM, mg/kg 3 at monotherapy ipilimumab in Patient Population Patient 2011 to September 30, 2012 and received and received 2012 30, September to 2011 Comprehensive an MSH/USON at care used that Network site Alliance Strategic over iKM EHR capabilities full Ex-clusionstudy period. criteria included current AM, for treatment prior systemic in enrollment or pending and treatment for other cancers. cancers. other for and treatment

Summary of RWD study characteristics objectives were to to objectives were describe 1) patient demographic and disease characteristics of care 2) patterns 3) survival outcomes Early Access EarlyAccess (Italian Programme find to subset) dosing optimal of patients 4) AEs with treatment- who AM naïve with treated were 3 mg/ ipilimu-mab kg monotherapy in a community practice setting Aim – Studies used provide to real-world data on (RWD) effectiveness and safety of new drugs in relative Study Name Study CA184-332 The primary CA184-089 Table S1 effectiveness assessments(REAs). Us A Pigment Cell & & Cell Pigment Reference http://www.has-sante.fr/ portail/upload/docs/evamed/ CT-12741_YERVOY_PIC_ REEV_Avis3_CT12741.pdf Margolin, K., Wong, S. S. Wong, K.,Margolin, J., Song, R., J. Penrod, L., B., D. Johnson, I. F., Chang, (2013). D. & Mcdermott, ... safety and Effectiveness for ipilimumab of first-line melanoma- advanced retrospective multisite study. 26(6), Melanoma Research, 986-987.

clinical clinical the Study Design Study N/A; non- N/A; study randomised Retrospective Retrospective electronic health (EHR) record observational cohort of study treated patients with first-line ipilimumab monotherapy in practices of practices of specialized (Cytokine centers Group Working or [CWG] CWG-affiliated) hospital/ a

iKM source source iKM

the

primary of measure the time from starttime from of ipilimumab the Outcome(s) measured Outcome(s) Drug-relatd adverse effects adverse Drug-relatd 1) Variables to describe patient describe patient to Variables 1) characteristics and disease demographic primary race, age, gender, included initial time since diagnosis, at site at disease stage diagnosis, melanoma location start treatment, of ipilimumab performance status ECOG of metastases, serum metastases, brain of presence (PS), and (LDH) level, lactate dehydrogenase status. BRAF mutation of care patterns evaluate to Variables 2) treatment dosing, ipilimumab included for reasons and discontinuations, delays and concomitant termination, treatment supportive and subsequent and therapy. anticancer was 3) OS defined was and effectiveness treatment as was Safety cause. any from death until reported AEs. by assessed 4) Because AE grade/intensity treatment- causality (e.g. information, occurrence of date and or not), related in available not were reported irrespective are AE data data, any However, or causality. of grade during AE reported occurred that Any captured. was treatment ipilimumab as serious in AEs (SAEs) AEs documented charts to or leading patients’ emergency room visit or death were were or death visit emergency room as SAEs. defined Comparator(s) N/A N/A ipilimumab monotherapy ipilimumab Intervention(s) ipilimumab monotherapy ipilimumab entire entire the clinical trial, trial, clinical a first-line setting during April 1, duringApril setting 1, first-line the As present in clinical practice across practice across in clinical present As Europe Eligible patients were required to have have to required were patients Eligible started have old, years be ≥18 AM, mg/kg 3 at monotherapy ipilimumab in Patient Population Patient 2011 to September 30, 2012 and received and received 2012 30, September to 2011 Comprehensive an MSH/USON at care used that Network site Alliance Strategic over iKM EHR capabilities full study period. Ex-clusion criteria included Ex-clusionstudy period. criteria included current AM, for treatment prior systemic in enrollment or pending and treatment for other cancers. cancers. other for and treatment

safety of of safety primary community community the a Summary of RWD study characteristics Periodic update update Periodic of use in ipilimumab practiceclincal 4) AEs of patients of patients 4) AEs with treatment- who AM naïve with treated were 3 mg/ ipilimu-mab kg monotherapy in practice setting 3) survival outcomes The The to objectives were describe 1) patient demographic and disease characteristics of care 2) patterns Aim – continued – Study Name Study Periodic Periodic Safety Update Report data) (PSUR CA184-338 Table S1

Appendices

240 11 Appendices 241 11 Us A Pigment Cell & & Cell Pigment http://www.has-sante.fr/ portail/upload/docs/evamed/ CT-12741_YERVOY_PIC_ REEV_Avis3_CT12741.pdf Margolin, K., Wong, S. S. Wong, K.,Margolin, J., Song, R., J. Penrod, L., B., D. Johnson, I. F., Chang, (2013). D. & Mcdermott, ... safety and Effectiveness for ipilimumab of first-line melanoma- advanced retrospective multisite study. 26(6), Melanoma Research, 986-987. Reference

Study Design Study N/A; non- N/A; study randomised Retrospective Retrospective electronic health (EHR) record observational cohort of study treated patients with first-line ipilimumab monotherapy in the clinical practices of specialized (Cytokine centers Working Group Group Working or [CWG] CWG-affiliated) CWG-affiliated) a hospital/

iKM source source iKM

the

primary of measure the time from starttime from of ipilimumab Outcome(s) measured Outcome(s) Drug-relatd adverse effects adverse Drug-relatd 1) Variables to describe patient describe patient to Variables 1) characteristics and disease demographic primary race, age, gender, included initial time since diagnosis, at site at disease stage diagnosis, melanoma start location treatment, of ipilimumab performancestatus ECOG of metastases, serum metastases, brain of presence (PS), and (LDH) level, lactate dehydrogenase status. BRAF mutation of care patterns evaluate to Variables 2) treatment dosing, ipilimumab included for reasons and discontinuations, delays and concomitant termination, treatment supportive and subsequent and therapy. anticancer reported irrespective are AE data data, any However, or causality. of grade during AE reported occurred that Any captured. was treatment ipilimumab serious as in AEs (SAEs) AEs documented charts to or leading patients’ 3) OS was was 3) OS 4) Because AE grade/intensity treatment- causality (e.g. information, occurrence of date and or not), related in available not were were or death visit emergency room as SAEs. defined treatment effectiveness and was defined defined was and effectiveness treatment as the was Safety cause. any from death until reported AEs. by assessed Comparator(s) N/A N/A ipilimumab monotherapy ipilimumab Intervention(s) ipilimumab monotherapy ipilimumab the entire the entire clinical trial, trial, clinical a first-line setting during April 1, duringApril setting 1, first-line the Eligible patients were required to have have to required were patients Eligible started have old, years be ≥18 AM, mg/kg 3 at monotherapy ipilimumab in practice across in clinical present As Europe Patient Population Patient 2011 to September 30, 2012 and received and received 2012 30, September to 2011 Comprehensive an MSH/USON at care used that Network site Alliance Strategic over iKM EHR capabilities full Ex-clusionstudy period. criteria included current AM, for treatment prior systemic in enrollment or pending and treatment for other cancers. cancers. other for and treatment

safety of of safety Summary of RWD study characteristics objectives were to to objectives were describe 1) patient demographic and disease characteristics of care 2) patterns 3) survival outcomes of patients 4) AEs with treatment- who AM naïve with treated were 3 mg/ ipilimu-mab kg monotherapy in a community practice setting update Periodic of the use in ipilimumab practiceclincal Aim – continued – Study Name Study Periodic Periodic Safety Update Report data) (PSUR CA184-338 The primary Table S1 BRAF BRAF the Long-term Long-term 2010; 363:809-19. 2010; Reference Ascierto, P. A., Minor, Minor, A., P. Ascierto, C., Lebbe, A., Ribas, D., O’Hagan,Arya, A., ... & N., II Phase (2013). O. Hamid, of trial (BREAK-2) Flaherty KT, Puzanov I, Kim I, Kim Puzanov Flaherty KT, Ribas A, McArthurKB, GA, Inhibition SosmanJA, et al. BRAF activated of mutated, melanoma. in metastatic JournalNew England of Medicine. KimSosman K,J, Schuchter A, L, Gonzalez Pavlick R, et al. J, Weber V600 of BRAF follow-up metastatic mutated patients melanoma with vermrafenib treated survival. prolonged reveals JournalNew England of 366:707-14. 2012; Medicine. inhibitor dabrafenib dabrafenib inhibitor in patients (GSK2118436) melanoma. with metastatic Journal oncology, of clinical 3205-3211. 31(26),

Study Design Study Phase II, single-Phase arm, multicentre study Phase 1 dose- 1 Phase non- ranging; study randomised open II Phase – single armlabel

Outcome(s) measured Outcome(s) Primary: investigator-assessed overall overall Primary: investigator-assessed rate response Secondary: survival (PFS) progression-free survival (OS) and overall 1) Safety/Adverse events 1) Safety/Adverse activity2) Pharmacodynamic in tumour tissue Primary: (ORR) rate response Overall Secondary: survival Overall (OS) Comparator(s) N/A N/A N/A vemurafenib 960 mg twice mg twice 960 vemurafenib daily mg orally 960 vemurafenib twice daily Intervention(s) dabrafenib 150 mg twice 150 mg daily dabrafenib

Previously treated BRAF mutation- treated Previously Inclusion patients. melanoma positive criteria were: 18 > Male or female melanoma stage IV proven Histologically BRAF V600-positive mutation one least at following disease Progressive treatment prior systemic 0-1 status ECOG least at for metastasis controlled Brain 3 months 5 years in last cancer invasive No other and renal haematological, Adequate functionhepatic Previously treated BRAF mutation positive positive BRAF mutation treated Previously tumours (advanced with solid patients colorectal). and metastatic melanoma Inclusion criteria were: 18 > Male or female and renal haematological, Adequate functionhepatic Solid tumours Solidtumours care refractoryHistologically standard to available care or no 0-1 status ECOG expectancyLife > 3 months of or knownof progressing Absence metastases brain unstable Patient Population Patient Patients with histologically confirmed confirmed with histologically Patients melanoma BRAFV600E/K metastatic IV)(stage Additional enrolled. were eligibility measurable criteria included Response Evaluation to according disease 1.1 and version Tumors in SolidCriteria Group Oncology Cooperative an Eastern performance Patients 0 or 1. of status they had not or whether eligible were for therapy prior systemic received other of exclusive disease, metastatic History BRAF/MEK inhibitors. or evidence exclusionary. was metastases of brain and renal, liver, bone marrow, Adequate function cardiac normal and clotting required. also were parameters safety safety the Summary of RWD study characteristics Assess Assess Long-term Long-term of follow-up vemurafenib- patients treated with BRAF-V600 mutation positive Phase 1 dose dose 1 Phase study ranging Aim and clinical activityand clinical in of dabrafenib BRAF(V600E/K) mutation-positive metastatic melanoma – continued – Study Name Study BREAK-2 BRIM2 BRIM1 Table S1

Appendices

242 11 Appendices 243 11 the BRAF Long-term Long-term 2010; 363:809-19. 2010; Ascierto, P. A., Minor, Minor, A., P. Ascierto, C., Lebbe, A., Ribas, D., O’Hagan,Arya, A., ... & N., II Phase (2013). O. Hamid, of trial (BREAK-2) dabrafenib inhibitor in patients (GSK2118436) melanoma. with metastatic Journal oncology, of clinical 3205-3211. 31(26), Flaherty KT, Puzanov I, Kim I, Kim Puzanov Flaherty KT, Ribas A, McArthurKB, GA, Inhibition SosmanJA, et al. BRAF activated of mutated, melanoma. in metastatic JournalNew England of Medicine. KimSosman K,J, Schuchter A, L, Gonzalez Pavlick R, et al. J, Weber V600 of BRAF follow-up metastatic mutated patients melanoma with vermrafenib treated survival. prolonged reveals JournalNew England of 366:707-14. 2012; Medicine. Reference

Study Design Study Phase II, single-Phase arm, multicentre study Phase II open open II Phase Phase 1 dose- 1 Phase non- ranging; study randomised – single armlabel

Outcome(s) measured Outcome(s) Primary: investigator-assessed overall overall Primary: investigator-assessed rate response Secondary: survival (PFS) progression-free survival (OS) and overall Primary: (ORR) rate response Overall Secondary: survival Overall (OS) 1) Safety/Adverse events 1) Safety/Adverse activity2) Pharmacodynamic in tumour tissue Comparator(s) N/A N/A N/A vemurafenib 960 mg orally mg orally 960 vemurafenib twice daily mg twice 150 daily dabrafenib vemurafenib 960 mg twice mg twice 960 vemurafenib daily Intervention(s)

Previously treated BRAF mutation- treated Previously Inclusion patients. melanoma positive criteria were: 18 > Male or female melanoma stage IV proven Histologically BRAF V600-positive mutation least one at following disease Progressive treatment prior systemic 0-1 status ECOG least at for metastasis controlled Brain 3 months 5 years in last cancer invasive No other and renal haematological, Adequate functionhepatic confirmed with histologically Patients melanoma BRAFV600E/K metastatic IV)(stage Additional enrolled. were eligibility measurable criteria included Response Evaluation to according disease 1.1 and version Tumors in SolidCriteria Group Oncology Cooperative an Eastern performance Patients 0 or 1. of status they had not or whether eligible were for therapy prior systemic received other of exclusive disease, metastatic History BRAF/MEK inhibitors. or evidence exclusionary. was metastases of brain and renal, liver, bone marrow, Adequate function cardiac normal and clotting required. also were parameters Previously treated BRAF mutation positive positive BRAF mutation treated Previously tumours (advanced with solid patients colorectal). and metastatic melanoma Inclusion criteria were: 18 > Male or female Solidtumours care refractoryHistologically standard to available care or no 0-1 status ECOG expectancyLife > 3 months of or knownof progressing Absence metastases brain unstable and renal haematological, Adequate functionhepatic Patient Population Patient Summary of RWD study characteristics follow-up of of follow-up vemurafenib- patients treated with BRAF-V600 mutation positive activityand clinical in of dabrafenib BRAF(V600E/K) mutation-positive metastatic melanoma Phase 1 dose dose 1 Phase study ranging Aim – continued – Study Name Study BRIM2 Long-term BREAK-2 Assess the safety BRIM1 Table S1 phase phase A Reference Kirkwood, J. M., Long, G. G. Long, M., Kirkwood, J. A., M. Davies, U., Trefzer, V., Chapman, A., P. Ascierto, L. V. Goodman, ... & B., P. BREAK-MB: (2012). overall assessing II study response intracranial dabrafenib to (OIRR) rate in patients (GSK2118436) V600E/k with BRAF(pts) mutation-positive with brain melanoma (mets). metastases Horn, S. N., L., Gettinger, R., D. Gandhi, L., Spigel, Rizvi, S. J., Antonia, ... & N. A., Overall (2015). V. Sequist, L. survival and long-term of nivolumab safety death (anti–programmed BMS-936558, 1 antibody, ONO-4538)with patients in advanced treated previously non–small-cell cancer. lung Journal oncology, of clinical 2004-2012. 33(18), Study Design Study Phase II, single- II, Phase arm study dose- I Phase cohort;escalation non-randomised study

Outcome(s) measured Outcome(s) Primary: investigator-assessed overall overall Primary: investigator-assessed (OIRR) rate response intracranial Primary: events adverse drug-related (ORR; rate objective response (AEs), confirmed with of patients percentage or partial all among complete responses patients) treated Secondary: survival Overall Comparator(s) N/A N/A dabrafenib 150 mg twicemg 150 daily dabrafenib 3-, or 10-mg/kg 1-, nivolumab Intervention(s) Patients eligibility criteria were as as eligibility criteria were Patients confirmed pathologically follows: Eastern years, age ≥ 18 NSCLC, advanced (ECOG) Group Oncology Cooperative performance (before 0 or 1 of status in 4 amendment of implementation performanceOctober status ECOG 2010, organ adequate allowed), 2 was to of 0 prior systemic function, five to and one NSCLC. advanced for regimens treatment experienced have also had to Patients one least at through progression or taxane-basedplatinum- and regimen lesion by measurable least one at have with Patients 1.0).14 (version RECIST at for stable metastases brain treated Exclusion eligible. were 8 weeks least disease, autoimmune criteria included T cell–modulating with prior therapy anti–CTLA-4, (eg, antibodies anti–PD-1, and anti–PD-L1), requiring conditions history medications, immunosuppressive and active infection HIV, by of infection C viruses. B or hepatitis by Stage IV pts with ≥ 1 intracranial met met 1 intracranial ≥ pts with IV Stage without MRI) by cm–4 cm assessed (0.5 with or A) (Cohort therapy prior brain therapy prior brain following progression V600E/K with eligible B) were (Cohort mutation. Patient Population Patient Summary of RWD study characteristics Phase I dose- I Phase cohortescalation trial expansion safety evaluating activityand clinical of nivolumab with in patients NSCLC, advanced and melanoma, colorectal, kidney, and castration- prostate resistant cancer. Assess overall overall Assess intracranial response to (OIRR) rate in dabrafenib with patients BRAF V600E/k mutation-positive with melanoma metastases. brain Aim – continued – Study Name Study Checkmate- 003 BREAK-MB Table S1

Appendices

244 11 Appendices 245 11 A phase A phase Gettinger, S. N., Horn, L., Horn, S. N., L., Gettinger, R., D. Gandhi, L., Spigel, Rizvi, S. J., Antonia, ... & N. A., Overall (2015). V. Sequist, L. survival and long-term of nivolumab safety death (anti–programmed BMS-936558, 1 antibody, ONO-4538)with patients in advanced treated previously non–small-cell cancer. lung Journal oncology, of clinical 2004-2012. 33(18), Kirkwood, J. M., Long, G. G. Long, M., Kirkwood, J. A., M. Davies, U., Trefzer, V., Chapman, A., P. Ascierto, L. V. Goodman, ... & B., P. BREAK-MB: (2012). overall assessing II study response intracranial dabrafenib to (OIRR) rate in patients (GSK2118436) V600E/k with BRAF(pts) mutation-positive with brain melanoma (mets). metastases Reference Study Design Study Phase I dose- I Phase cohort;escalation non-randomised study Phase II, single- II, Phase arm study

Outcome(s) measured Outcome(s) Secondary: survival Overall Primary: drug-related adverse events Primary: events adverse drug-related (ORR; rate objective response (AEs), confirmed with of patients percentage or partial among all complete responses patients) treated Primary: investigator-assessed overall overall Primary: investigator-assessed (OIRR) rate response intracranial Comparator(s) N/A N/A dabrafenib 150 mg twice 150 daily dabrafenib 3-, or 10-mg/kg 1-, nivolumab Intervention(s) Patients eligibility criteria were as as eligibility criteria were Patients confirmed pathologically follows: Eastern years, age ≥ 18 NSCLC, advanced (ECOG) Group Oncology Cooperative performance (before 0 or 1 of status in 4 amendment of implementation performanceOctober status ECOG 2010, organ adequate allowed), 2 was to of 0 prior systemic function, five to and one NSCLC. advanced for regimens treatment experienced have also had to Patients one least at through progression or taxane-basedplatinum- and regimen lesion by measurable least one at have with Patients 1.0).14 (version RECIST at for stable metastases brain treated Exclusion eligible. were 8 weeks least disease, autoimmune criteria included T cell–modulating with prior therapy anti–CTLA-4, (eg, antibodies anti–PD-1, and anti–PD-L1), requiring conditions history medications, immunosuppressive and active infection HIV, by of infection C viruses. B or hepatitis by Stage IV pts with ≥ 1 intracranial met met 1 intracranial ≥ pts with IV Stage without MRI) by cm–4 cm assessed (0.5 with or A) (Cohort therapy prior brain therapy prior brain following progression V600E/K with eligible B) were (Cohort mutation. Patient Population Patient Summary of RWD study characteristics Phase I dose- I Phase cohortescalation trial expansion safety evaluating activityand clinical of nivolumab with in patients NSCLC, advanced and melanoma, colorectal, kidney, and castration- prostate resistant cancer. Assess overall overall Assess intracranial response to (OIRR) rate in dabrafenib with patients BRAF V600E/k mutation-positive with melanoma metastases. brain Aim – continued – Study Name Study Checkmate- 003 BREAK-MB Table S1

Reference Hamid O et al (2013) Safety Safety et al (2013) Hamid O with responses and tumor (Anti-PD-1) lambrolizumab in melanoma NEJM 134-144. 369:2 Study Design Study Expansion cohortExpansion 1 dose-of phase study;escalation non-randomised study. use of use of Response Response the the

National Cancer Institute Common Institute Common Cancer National Efficacy end-points: overall responses responses overall Efficacy end-points: data, investigator-reported from derived immune- to according with assessment criteria; overall and response related independent, from derived responses with review, radiologic blinded central, to according assessment Outcome(s) measured Outcome(s) Evaluation Criteria in Solid Tumors Tumors in Solid Criteria Evaluation (RECIST), 1.1 version with graded effects: Toxic the Events, Adverse for Criteria Terminology 4.0. version Comparator(s) Pembrolizumab Pembrolizumab mg/kg every10 2 weeks Pembrolizumab mg/kg every10 3 weeks Pembrolizumab 2 mg/kg every 2 mg/kg Pembrolizumab 3 weeks Intervention(s) Patients with measurable metastatic or or metastatic with measurable Patients unresectable melanoma, advanced locally prior received had those who both those who and with ipilimumab therapy had not. Patient Population Patient

safety profile profile safety Summary of RWD study characteristics To evaluate evaluate To the of pembrolizumab called (formerly lambrolizumab) tumour assess every response 12 weeks Aim – continued – Study Name Study Keynote-001 B1) (Part Table S1

Appendices

246 11 Appendices 247 11

National Cancer Institute Common Institute Common Cancer National Outcome(s) measured Outcome(s) Toxic effects: graded with graded effects: Toxic Efficacy end-points: overall responses responses overall Efficacy end-points: data, investigator-reported from derived immune- to according with assessment criteria; overall and response related independent, from derived responses with review, radiologic blinded central, to according assessment Tumors in Solid Criteria Evaluation (RECIST), 1.1 version the Events, Adverse for Criteria Terminology 4.0. version Comparator(s) 3 weeks 10 mg/kg every10 mg/kg Pembrolizumab Pembrolizumab 2 weeks 10 mg/kg every10 mg/kg Pembrolizumab Pembrolizumab Pembrolizumab 2 mg/kg every 2 mg/kg Pembrolizumab 3 weeks Intervention(s) Patients with measurable metastatic or or metastatic with measurable Patients unresectable melanoma, advanced locally prior received those who had both those who and with ipilimumab therapy had not. Patient Population Patient

outcomes research proposal research outcomes the Number of critical comments made made critical comments Number of %) comments; of total (percentage 2 (3%) 2 (3%) 8 (12%) 5 (7%) 13 (19%) 7 (10%) 2 (3%) 8 (12%) 14 (21%) 0 (0%) 7 (10%) 68 Number of critical comments made made Number of critical comments %) comments; of total (percentage 0 (0%) 0 (0%) 9 (16%) 3 (5%) 7 (12%) 6 (10%) 4 (7%) 7 (12%) 10 (17%) 12 (21%) 58

research proposal (T=0) proposal research - Report sections for which critical commentary was collected for - Report sections for which critical commentary was collected for appropriate use assessment

A Outcomes Outcomes report section question Research Indication population Patient treatment Comparator (effects) Outcomes (costs) Outcomes horizon Time collectionData method Model feasibility Study bottleneck(s) Anticipated comments of number Total report section question Research Indication population Patient treatment Comparator (effects) Outcomes collectionData method (generalisability) patients of included Representativeness clinical practice use in Appropriate effectiveness Clinical Quality of Life comments of number Total Appropriate use assessment (T=4) use assessment Appropriate Table (T=0) and thetotal number of comments per section. (T=4) and the total number of comments per section. Table B 5 - APPENDIX - 5 CHAPTER

Appendices

248 11 Appendices 249 11 Number of critical comments made made critical comments Number of %) comments; of total (percentage 11 (9%) 2 (2%) 14 (11%) 20 (16%) 15 (12%) 11 (9%) 0 (0%) 2 (2%) 1 (0%) 0 (0%) 0 (0%) 10 (8%) 10 (8%) 5 (4%) 2 (2%) 12 (10%) 8 (7%) 123

- Report sections for which critical commentary was collected for cost-effectiveness assessment Patient population Patient treatment Comparator (effects) Outcomes (costs) Outcomes Model structure Input parameters technique Analysis perspective Study Horizon Time Discounting Assumptions Sensitivity (planned) analyses costs & total Incremental effects & total Incremental ICER Sensitivity (results) analyses Other comments of number Total report section Cost-effectiveness assessment assessment Cost-effectiveness (T=4) Table C (T=4) and the totalnumber of comments per section.

Budget Budget Impact Analysis Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Cost- effectiveness assessment Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes T=4 Dossier Report on outcomes research Report research on outcomes use appropriate for Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No Pharmacotherapeutic Pharmacotherapeutic assessment Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes a Value of of Value Information analysis No No No No No No No No No No No No Yes Yes Yes Yes Outcomes Outcomes research cost- for proposal effectiveness Yes Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes Outcomes Outcomes research for proposal use appropriate Yes Yes Yes Yes No Yes Yes Yes cost-effectiveness analysis were proposed. These plans These plans proposed. were analysis cost-effectiveness a Yes Yes Yes Yes Budget Budget Impact Analysis Yes Yes No Yes No Yes Yes Yes T=0 Dossier b b b b Cost- Cost- effectiveness assessment Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Pharmacotherapeutic Pharmacotherapeutic assessment Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes full cost-effectiveness assessment. Instead, plans for for plans assessment. Instead, full cost-effectiveness a guideline for outcomes research, published by ZIN in 2008. by published research, outcomes for guideline the - Reports available per component of and T=0 reports T=4 published for finalized drugs. MAH did not submit did not MAH As outlined in outlined As a b methyl methyl aminolevulinate pemetrexed omalizumab voriconazol ranibizumab rituximab natalizumab trastuzumab agalsidase alpha agalsidase beta agalsidase eculizumab alglucosidase alphaalglucosidase were subsequently assessed. subsequently were Finalized drug Finalized Table D

Appendices

250 11 Appendices 251 11

4 package4 Impact Analysis Impact Analysis Budget Budget Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Cost- effectiveness assessment Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Committee conducting appraisal of evidence of conducting appraisal Committee Committee Appraisal Committee Appraisal Committee Appraisal Committee Appraisal Committee Assessment Committee Assessment Committee Assessment Committee Appraisal Committee Assessment Committee Appraisal Committee Appraisal Committee Assessment T=4 Dossier appraisalof evidence in relation T=4 at theto Report on outcomes research Report research on outcomes use appropriate for Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes committeethat conducted the Pharmacotherapeutic Pharmacotherapeutic assessment Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes - The a Finalized drug Finalized alglucosidase alpha alpha alglucosidase alpha agalsidase beta agalsidase eculizumab rituximab natalizumab trastuzumab omalizumab voriconazol ranibizumab aminolevulinate methyl pemetrexed criteria for finalized drugs. Table E analysis Information Information Value of of Value No No No No No No No No No No No No Yes Yes Yes Yes Outcomes Outcomes research cost- for proposal effectiveness No Yes Yes Yes Yes Yes Yes Yes Outcomes Outcomes research for proposal use appropriate Yes cost-effectiveness analysis were proposed. These plans These plans proposed. were analysis cost-effectiveness a YesYesYesYes Yes Yes Yes Yes Budget Budget Impact Analysis NoYesYesYes No Yes Yes Yes Yes Yes Yes Yes No Yes Yes T=0 Dossier b b b b Cost- Cost- effectiveness assessment Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Pharmacotherapeutic Pharmacotherapeutic assessment Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes full cost-effectiveness assessment. Instead, plans for for plans assessment. Instead, full cost-effectiveness a guideline for outcomes research, published by ZIN in 2008. by published research, outcomes for guideline the - Reports available per component of and T=0 reports T=4 published for finalized drugs. MAH did not submit did not MAH As outlined in outlined As a b trastuzumab omalizumab voriconazol ranibizumab methyl aminolevulinate pemetrexed alglucosidase alphaalglucosidase alpha agalsidase beta agalsidase eculizumab rituximab natalizumab were subsequently assessed. subsequently were Finalized drug Finalized Table D probability probability the introduction of generic of introduction products. the comparator, there is little risk for incurring high incurring high little risk is for there comparator, the reference ICER value of €80,000/QALY, of €80,000/QALY, ICER value reference calculated ICER of €15,535/QALY are acceptable and well and well acceptable are €15,535/QALY ICER of calculated L-Amb to compared Vfend of cost-effectiveness a Conclusions on cost-effectiveness (CE) in clinical practice on cost-effectiveness Conclusions ICERs presented substantiated by evidence submitted and are and are submitted evidence by substantiated presented ICERs of €80,000/QALY. value threshold above At to 40%. 10% is between cost-effective being Pemetrexed for will be soon, cost-effectiveness of patent it go out Should due ot improved ICERs presented substantiated by evidence submitted and are and are submitted evidence by substantiated presented ICERs of €80,000/QALY. value threshold above and are submitted evidence by substantiated presented ICERs of €80,000/QALY. value threshold above and are submitted evidence by substantiated presented ICERs of €80,000/QALY. value threshold above and submitted evidence by substantiated not presented ICERs €80,000/QALY. of value threshold above are and comparable value added therapeutic of absence Dueto with costs No submitted. evidence by substantiated not presented ICERs CE. regarding reached can be conclusion The substantiated. No submitted. evidence by substantiated not presented ICERs CE. regarding reached can be conclusion The is favourable. (comparator) No submitted. evidence by substantiated not presented ICERs CE. regarding reached can be conclusion ICER´s. Therefore, despite insufficient evidence, no additional additional no evidence, insufficient despite Therefore, ICER´s. be collected. should data Conclusions on quality of evidence for for on quality of evidence Conclusions T=4 at (CE) submitted cost-effectiveness Sufficient Sufficient Sufficient Sufficient Sufficient Not sufficient Not sufficient Not sufficient Sufficient Not sufficient Sufficient Not sufficient Conclusions on appropriate use (AU) (AU) use on appropriate Conclusions in clinical practice AU confirmed AU conclusions no confirmed; not AU on evidence based reached be could submitted confirmed AU conclusions no confirmed; not AU on evidence based reached be could submitted confirmed AU conclusions no confirmed; not AU on evidence based reached be could submitted AU confirmed AU confirmed AU confirmed AU in implemented confirmed; not AU than intended population broader confirmed AU confirmed AU scientific adequacy of evidencesubmitted T=4to answer at questionsT=0, raisedT=4 at appropriate at andcost-effectiveness use(AU) (CE) T=4 at per finalized drug. Conclusions on quality of evidence for for on quality of evidence Conclusions T=4 at submitted (AU) use appropriate Sufficient Not sufficient Sufficient Not sufficient Sufficient Not sufficient Sufficient Sufficient Sufficient Sufficient Sufficient Sufficient : Incremental Cost-Effectiveness Ratio; QALY: Quality-Adjusted Life-Year. Quality-Adjusted QALY: Ratio; Cost-Effectiveness : Incremental - Assessment Committee’s conclusions on the Abbreviations: ICER Abbreviations: Finalized drug Finalized methyl aminolevulinate methyl pemetrexed ranibizumab voriconazol trastuzumab omalizumab natalizumab rituximab agalsidase beta agalsidase eculizumab agalsidase alpha agalsidase alglucosidase alpha alpha alglucosidase Table F

Appendices

252 11 Appendices 253 11 the probability the probability introduction of generic of introduction products. the comparator, there is little risk for incurring high incurring high little risk is for there comparator, the reference ICER value of €80,000/QALY, of €80,000/QALY, ICER value reference calculated ICER of €15,535/QALY are acceptable and well and well acceptable are €15,535/QALY ICER of calculated L-Amb to compared Vfend of cost-effectiveness a At to 40%. 10% is between cost-effective being Pemetrexed for will be soon, cost-effectiveness of patent it go out Should due ot improved The The substantiated. No submitted. evidence by substantiated not presented ICERs CE. regarding reached can be conclusion The is favourable. (comparator) No submitted. evidence by substantiated not presented ICERs CE. regarding reached can be conclusion and are submitted evidence by substantiated presented ICERs of €80,000/QALY. value threshold above ICERs presented substantiated by evidence submitted and are and are submitted evidence by substantiated presented ICERs of €80,000/QALY. value threshold above and are submitted evidence by substantiated presented ICERs of €80,000/QALY. value threshold above and are submitted evidence by substantiated presented ICERs of €80,000/QALY. value threshold above and submitted evidence by not substantiated presented ICERs €80,000/QALY. of value threshold above are and comparable value of added therapeutic absence Dueto with costs additional no evidence, insufficient despite Therefore, ICER´s. be collected. should data No submitted. evidence by not substantiated presented ICERs CE. regarding reached can be conclusion cost-effectiveness (CE) submitted at T=4at (CE) submitted cost-effectiveness practice (CE) in clinical cost-effectiveness on Conclusions Conclusions on quality of evidence for for on quality of evidence Conclusions Sufficient Sufficient Not sufficient Sufficient Not sufficient Sufficient Not sufficient Not sufficient Not sufficient Sufficient Sufficient Sufficient AU confirmed AU confirmed AU conclusions no confirmed; not AU on evidence based reached be could submitted confirmed AU conclusions no confirmed; not AU on evidence based reached be could submitted confirmed AU conclusions no confirmed; not AU on evidence based reached be could submitted AU confirmed AU confirmed AU confirmed AU in implemented confirmed; not AU than intended population broader confirmed AU Conclusions on appropriate use (AU) (AU) use on appropriate Conclusions in clinical practice Sufficient Sufficient Not sufficient Sufficient Not sufficient Sufficient Not sufficient Sufficient Sufficient Sufficient Sufficient Sufficient Conclusions on quality of evidence for for on quality of evidence Conclusions T=4 at submitted (AU) use appropriate : Incremental Cost-Effectiveness Ratio; QALY: Quality-Adjusted Life-Year. Quality-Adjusted QALY: Ratio; Cost-Effectiveness : Incremental - Assessment Committee’s conclusions on the scientific adequacy of evidencesubmitted T=4to answer at questionsT=0, raisedT=4 at appropriate at andcost-effectiveness use(AU) (CE) T=4 at per finalized drug. Abbreviations: ICER Abbreviations: natalizumab trastuzumab omalizumab voriconazol ranibizumab aminolevulinate methyl pemetrexed alglucosidase alpha alpha alglucosidase alpha agalsidase beta agalsidase eculizumab rituximab Finalized drug Finalized Table F case of defaults defaults case of the more transparent system for for system transparent more for system transparent more for system transparent more a a a basic healthcare package. basic healthcare basic healthcare package until results from package from healthcare basic results until the basic healthcare package. healthcare basic package. healthcare basic package. healthcare basic drug will be prescribed. In be prescribed. drug will the the the the the new framework specific to drugs for orphan diseases. to drugs specific new framework for orphan diseases. to drugs specific new framework for orphan diseases. to drugs specific new framework drug from drug from drug from drug from the the the (European) study to investigate predictive factors for factorsfor predictive investigate study to (European) factorsfor predictive investigate study to (European) factorsfor predictive investigate study to (European) marketing authorisation holder (MAH) should sign Pay-for- sign should marketing(MAH) holder authorisation the the the a a a marketing (MAH). holder authorisation marketing (MAH). holder authorisation marketing (MAH). holder authorisation the drug to drug to drug to drug to the the the removal of this drug from from this drug of removal the the the the necessary parties up set to necessary parties up set to necessary parties up set to reimbursement of reimbursement of reimbursement of reimbursement separate financial framework for drugs for orphan diseases. drugs for framework financial separate for orphan diseases. drugs for framework financial separate for orphan diseases. drugs for framework financial separate the the the a a a the the the implementation of start- of implementation stop-criteria. and start- of implementation stop-criteria. and start- of implementation stop-criteria. and Temporarily continue reimbursement of reimbursement continue Temporarily of reimbursement continue Temporarily of reimbursement continue Temporarily Develop Develop Transfer Develop Transfer Develop Transfer Negotiate price negotiations with negotiations price Negotiate with negotiations price Negotiate with negotiations price Negotiate Discuss with clinicians if, and how, costs per QALY can be reduced (e.g. through dose modification). dose through (e.g. can be reduced per QALY costs how, and if, with clinicians Discuss Demand start- develop & stop-criteria effectiveness, clinical develop and modification). dose through (e.g. can be reduced per QALY costs how, and if, with clinicians Discuss Demand start- develop & stop-criteria effectiveness, clinical develop and modification). dose through (e.g. can be reduced per QALY costs how, and if, with clinicians Discuss Demand start- develop & stop-criteria effectiveness, clinical develop and the the the Consider establishing an independent committee to advise clinicians in practice on start- in practice on clinicians advise to stop- and committee an independent establishing Consider drug. with this treatment for decisions start- in practice on advise clinicians to stop- and committee an independent establishing Consider drug. this with treatment for decisions start- in practice on advise clinicians to stop- and committee an independent establishing Consider drug. this with treatment for decisions [separate] Round Table on Multiple Sclerosis are presented. are on Multiple Sclerosis Table Round [separate] Extra conditions N/A N/A drug from of this removal for final decision its ZIN postpones N/A reimbursement, continued guarantee To N/A N/A • • • the Performance (PfP) agreements with all hospitals whereby whereby hospitals with all (PfP) agreements Performance • • • • • • on PfP agreements (e.g. due to lack of cooperation from individual hospitals or no refunds to hospitals based based hospitals to or no refunds hospitals individual from cooperation lack of to due (e.g. agreements on PfP for will advise ZIN on outcomes), • • • • • • • • • • • • Keep drug in basic healthcare package healthcare drug in basic Keep package healthcare drug in basic Keep certain based on conditions. package healthcare drug in basic Keep package. healthcare basic from drug Remove Remove drug from basic healthcare package. healthcare basic from drug Remove package healthcare drug in basic Keep package healthcare drug in basic Keep certain based on conditions. Keep drug in basic healthcare package healthcare drug in basic Keep certain based on conditions. package healthcare drug in basic Keep certain based on conditions. Keep drug in basic healthcare package healthcare in basic drug Keep certainon based conditions. ZIN advice - ZIN advice onreimbursement of finalized drugs based on advice of itsCommittees. ranibizumab voriconazol trastuzumab omalizumab eculizumab rituximab natalizumab agalsidase beta agalsidase agalsidase alpha agalsidase alglucosidase alpha alpha alglucosidase Finalized drug Finalized Table G

Appendices

254 11 Appendices 255 11 case of defaults defaults case of the more transparent system for for system transparent more more transparent system for for system transparent more for system transparent more a a a basic healthcare package. healthcare basic basic healthcare package until results from package from healthcare basic results until the basic healthcare package. healthcare basic basic healthcare package. healthcare basic package. healthcare basic drug will be prescribed. In be prescribed. drug will the the the the the new framework specific to drugs for orphan diseases. to drugs specific new framework new framework specific to drugs for orphan diseases. to drugs specific new framework for orphan diseases. to drugs specific new framework drug from drug from drug from drug from drug from the the the (European) study to investigate predictive factors for factorsfor predictive investigate study to (European) (European) study to investigate predictive factors for factorsfor predictive investigate study to (European) factorsfor predictive investigate study to (European) marketing authorisation holder (MAH) should sign Pay-for- sign should marketing (MAH) holder authorisation the the the a a a marketing (MAH). holder authorisation marketing (MAH). holder authorisation marketing (MAH). holder authorisation the drug to drug to drug to drug to drug to the the the removal of this drug from from this drug of removal the the the the necessary parties up set to necessary parties up set to necessary parties up set to reimbursement of reimbursement reimbursement of reimbursement of reimbursement separate financial framework for drugs for orphan diseases. drugs for framework financial separate separate financial framework for drugs for orphan diseases. drugs for framework financial separate for orphan diseases. drugs for framework financial separate implementation of start- of implementation stop-criteria. and implementation of start- of implementation stop-criteria. and start- of implementation stop-criteria. and Temporarily continue reimbursement of reimbursement continue Temporarily Temporarily continue reimbursement of reimbursement continue Temporarily Temporarily continue reimbursement of reimbursement continue Temporarily Transfer the Transfer Develop Develop a Transfer the Transfer the Transfer Develop Develop a Develop a Negotiate price negotiations with negotiations price Negotiate Negotiate price negotiations with negotiations price Negotiate with negotiations price Negotiate Discuss with clinicians if, and how, costs per QALY can be reduced (e.g. through dose modification). dose through (e.g. can be reduced per QALY costs how, and if, with clinicians Discuss Demand the start- develop & stop-criteria effectiveness, clinical develop and the Discuss with clinicians if, and how, costs per QALY can be reduced (e.g. through dose modification). dose through (e.g. can be reduced per QALY costs how, and if, with clinicians Discuss Demand the start- develop & stop-criteria effectiveness, clinical develop and the modification). dose through (e.g. can be reduced per QALY costs how, and if, with clinicians Discuss Demand the start- develop & stop-criteria effectiveness, clinical develop and the Consider establishing an independent committee to advise clinicians in practice on start- in practice on advise clinicians to stop- and committee an independent establishing Consider drug. with this treatment for decisions Consider establishing an independent committee to advise clinicians in practice on start- in practice on advise clinicians to stop- and committee an independent establishing Consider drug. this with treatment for decisions start- in practice on advise clinicians to stop- and committee an independent establishing Consider drug. this with treatment for decisions [separate] Round Table on Multiple Sclerosis are presented. are on Multiple Sclerosis Table Round [separate] Extra conditions • N/A N/A N/A reimbursement, continued guarantee To • Performance (PfP) agreements with all hospitals whereby whereby all hospitals with (PfP) agreements Performance • the ZIN postpones its final decision for removal of this drug from from drug of this removal for final decision its ZIN postpones N/A N/A • • • • on outcomes), ZIN will advise for for will advise ZIN on outcomes), • • based hospitals to or no refunds hospitals individual from cooperation lack of to due (e.g. agreements on PfP • • • • • • • • • • • • Keep drug in basic healthcare package healthcare in basic drug Keep certain based on conditions. Keep drug in basic healthcare package healthcare drug in basic Keep certain based on conditions. package healthcare drug in basic Keep certain based on conditions. Keep drug in basic healthcare package healthcare drug in basic Keep certain based on conditions. package healthcare drug in basic Keep package. healthcare basic from drug Remove Remove drug from basic healthcare package. healthcare basic from drug Remove package healthcare drug in basic Keep package healthcare drug in basic Keep certain based on conditions. package healthcare drug in basic Keep ZIN advice - ZIN advice onreimbursement of finalized drugs based on advice of itsCommittees. alglucosidase alpha alpha alglucosidase agalsidase alpha agalsidase beta agalsidase omalizumab voriconazol ranibizumab eculizumab rituximab natalizumab trastuzumab Finalized drug Finalized Table G criteria criteria the implementation of such criteria becomes criteriasuch becomes of implementation the clinical guideline, in order to clarify and specify to in order guideline, clinical the clinicians’ societies to update update to societies clinicians’ the Extra conditions ZIN requests ZIN requests N/A for treatment with methyl aminolevulinate thus ensuring that ensuringthus that aminolevulinate methyl with treatment for feasible in practice. in feasible No information regarding the status of status of the regarding No information

these drugs was available in public public in available was drugs these The website. ZIN the on documents to retrieve obliged thus were authors assessors information from relevant the within ZIN. † ZIN ZIN the (n=8) (n=7) are ongoing† pending† are Re-assessments Re-assessments Re-assessments Conditional Financing (CF) scheme. (CF) Financing Conditional the relevant information from assessors within ZIN. within assessors from information relevant (n=2) the added added (n=12) Number of below €2.5 monitoring monitoring therapeutic therapeutic in CF (n=27) no longer of of no longer value (n=22) orphan drugs drugs orphan framework on by by 31.03.2017 Transferred to drugs finalized finalized drugs Budget impact impact Budget million/year or Drugs remaining Keep drug in basic healthcare package healthcare in basic drug Keep certainon based conditions. package healthcare drug in basic Keep ZIN advice status of these drugs was available in public documents on documents in public available of these drugs was status the drugs drugs (n=49) nominated nominated Total number of number of Total

QALY: Quality-Adjusted Life-Year; ZIN: Zorginstituut Nederland. Zorginstituut ZIN: Life-Year; Quality-Adjusted QALY:

authors were thus obliged to retrieve retrieve obliged to thus were authors

Finalized Drugs Finalized scheme (n=27) scheme

Exclusion of Drugs after T=0 Assessments T=0 after Drugs of Exclusion - Overview for drugs of and status of in- and exclusion The The Number of of Number - continued - CF in remaining Drugs A Abbreviations: pemetrexed methyl aminolevulinate methyl Finalized drug Finalized Table G Figure Figure website. † No information regarding regarding † No information

Appendices

256 11

Finalized candidates Finalized

0 Appendices 10 257

Number of Comments Number

Number of comments T=0 & T=4 combined T=4 & T=0 comments of Number

Number of comments on CEA on comments of Number 30 Number of comments on AUA on comments of Number

outcomes outcomes

Number of comments on on ORP of comments Number on AUA of comments Number on CEA of comments Number Number of comments T=0 & T=4 combined Number of comments on ORP on comments of Number the criteria the criteria 40

the 50

Percentage recommendations not implemented Percentage recommendations implemented 60 Assessment Committee regarding regarding Committee Assessment the Assessment Committee on Committee Assessment implementation of such criteria becomes criteriasuch becomes of implementation outcomes research study per finalized drug. study per finalized research outcomes the the the finalized drugs. finalized the clinical guideline, in order to clarify and specify to in order guideline, clinical Finalized candidates Finalized the Finalized candidates 0% number of critical comments provided by by provided of criticalnumber comments 90% 80% 70% 60% 50% 40% 30% 20% 10% 100% - Percentage of recommendations made by made by recommendations of - Percentage The The clinicians’ societies to update update to societies clinicians’ -

0 the Percentage 60 50 40 30 20 10

outcomes research proposal (T=0) proposal research in outcomes implemented Number of Comments of Number research proposal (ORP; T=0), appropriate use assessment (AUA; T=4), cost-effectiveness assessment assessment cost-effectiveness T=4), (AUA; use assessment appropriate T=0), (ORP; proposal research for of comments number total and T=4), (CEA; the Figure C Figure Figure B Figure Extra conditions for treatment with methyl aminolevulinate thus ensuring that ensuringthus that aminolevulinate methyl with treatment for N/A ZIN requests ZIN requests feasible in practice. feasible Keep drug in basic healthcare package healthcare in basic drug Keep certain based on conditions. package healthcare drug in basic Keep ZIN advice QALY: Quality-Adjusted Life-Year; ZIN: Zorginstituut Nederland. Zorginstituut ZIN: Life-Year; Quality-Adjusted QALY: - continued - Abbreviations: methyl aminolevulinate methyl pemetrexed Finalized drug Finalized Table G drug drug a proposed proposed the proposed proposed the CF framework in in CF framework proposed design for for design proposed proposed design for for design proposed the the the Assessment and Appraisal and Appraisal Assessment CF scheme in 2006. CF scheme CF scheme in 2006. CF scheme the the the reimbursement package after T=4 based T=4 based packagereimbursement after the Dutch Healthcare Authority (NZa) established established (NZa) Authority Healthcare Dutch Ministry (VWS) of Health for responsible was proposed design for for design proposed underlying policy framework for CF. underlying policy for framework CF scheme in 2006. CF scheme CF scheme in 2006. CF scheme ultimate decision to keep, or remove, remove, or keep, to decision ultimate CF scheme in practice. CF scheme Some medical specialists provided feedback on on feedback Some provided medical specialists the Provided feedback on feedback Provided Provided feedback on feedback Provided The The the 2006. the the The The the from on ZIN’s advice. on ZIN’s on feedback provided ZonMW for design Was responsible for designing and implementing and implementing designing for responsible Was the Reimbursement of CF drugs from T=0 onwards T=0 onwards CF drugs from of Reimbursement claims. on hospital based Members of on feedback provided Committees for design • N/A • Other • N/A • • • • • • drug, drug, re- . the appropriate appropriate the scientific evidence evidence scientific . the appropriate use and use and appropriate MAH on therapeutic value of of value therapeutic on MAH reimbursement package. reimbursement the the drug, budget impact analysis, budget impact analysis, drug, drug from drug from Provide feedback on preliminary versions of ZIN preliminary on of ZIN versions feedback Provide reports T=4. at preliminary on of ZIN versions feedback Provide reports T=4. at Re-assess and re-appraise for evidence submit and Prepare of value of therapeutic assessment budget impact analysis, impactbudget analysis, analysis cost-effectiveness submitted by by submitted the use and cost-effectiveness analysis use and cost-effectiveness Issue final advice to VWS to keep, or remove, remove, or keep, to VWS to advice Issue final a Role T=4 at N/A • N/A • N/A • • •

limited number number limited a implementation of of implementation the CF scheme. outcomes research study. research outcomes Collaborate with pharmaceutical with pharmaceutical Collaborate medical specialists industry, societyand medical facilitate to studies. research outcomes Implement outcomes research research Implement outcomes data in practice (i.e. studies collection). ZonMW financed financed ZonMW of outcomes research studies studies research of outcomes as part implemented (n=2) of the Collaborate with academic/private with academic/private Collaborate and specialists medical hospitals, implement societies to medical the Finance Finance studies. research outcomes • N/A Role during outcomes research phase research Role during outcomes • N/A • • N/A • drug, drug, MAH drug. drug, drug, scientific scientific the the the the the conditional financing (CF) scheme.(CF) the conditional financing outcomes research research outcomes Netherlands Organization Netherlands Organization establishment of outcomes outcomes of establishment outcomes research proposals proposals research outcomes the Provide feedback on preliminary on feedback Provide T=0. of ZIN reports versions at on therapeutic value of value on therapeutic during input Provide the proposals. research preliminary on feedback Provide T=0. of ZIN reports versions at Prepare and submit evidence on on evidence and submit Prepare of value therapeutic evidence submitted by by submitted evidence The The and Research Health for (ZonMW)Development provided on feedback methodological the drugs. some for appraise and Assess budget impact analysis and impactbudget and analysis preliminary cost-effectiveness analysis. feedback methodological Provide on drugs. for proposals budget impact analysis and impactbudget and analysis preliminary cost-effectiveness analysis. an outcomes and submit Prepare provide to proposal research and use on appropriate evidence of cost-effectiveness • • • N/A • N/A • • Role T=0 at • • CVZ: College voor Zorgverzekeringen; MAH: Marketing Authorization Holder; NZa: Nederlandse Zorgautoriteit; Holder; Marketing Nederlandse NZa: Zorgautoriteit; Authorization MAH: Zorgverzekeringen; CVZ: voor College – Roles of different stakeholders in in stakeholders different of Roles – Abbreviations: ZIN: Zorginsitituut Nederland; ZonMW: De Nederlandse organisatie voor gezondheidsonderzoek en zorginnovatie. gezondheidsonderzoek voor De Nederlandse organisatie Nederland; ZonMW: Zorginsitituut ZIN: Patient Organizations Patient Academic/ Private Hospitals Private Academic/ Medical Societies Specialists Healthcare Insurers Healthcare Pharmaceutical IndustryPharmaceutical Zorginstituut Nederland Nederland Zorginstituut CVZ), previously (ZIN; Assessment including: Appraisal Committee, and Assessors Committee External Public Bodies (i.e. Bodies (i.e. External Public non-ZIN) Stakeholders Table 1 6 - APPENDIX - 6 CHAPTER

Appendices

258 11 Appendices 259 11 a drug a drug the proposed the proposed the proposed the proposed CF framework in in CF framework proposed design for for design proposed proposed design for for design proposed the the the Assessment and Appraisal and Appraisal Assessment CF scheme in 2006. CF scheme CF scheme in 2006. CF scheme the the the reimbursement package after T=4 based T=4 based packagereimbursement after Dutch Healthcare Authority (NZa) established established (NZa) Authority Healthcare Dutch Ministry (VWS) of Health for responsible was underlying policy framework for CF. underlying policy for framework proposed design for for design proposed ultimate decision to keep, or remove, remove, or keep, to decision ultimate CF scheme in 2006. CF scheme CF scheme in 2006. CF scheme CF scheme in practice. CF scheme The The the Provided feedback on feedback Provided Provided feedback on feedback Provided Some medical specialists provided feedback on on feedback Some provided medical specialists the design for for design from the from advice. on ZIN’s on feedback provided ZonMW The The the the the 2006. Reimbursement of CF drugs from T=0 onwards T=0 onwards CF drugs from of Reimbursement claims. on hospital based Was responsible for designing and implementing and implementing designing for responsible Was the Members of Committees provided feedback on on feedback provided Committees for design Other • • • N/A N/A • • • • • • re- . the drug, the drug, appropriate appropriate the scientific evidence evidence scientific . the appropriate use and use and appropriate MAH on therapeutic value of of value therapeutic on MAH reimbursement package. reimbursement the the drug, budget impact analysis, budget impact analysis, drug, drug from drug from Prepare and submit evidence for for evidence and submit Prepare assessment of therapeutic value of value of therapeutic assessment Re-assess and re-appraise impactbudget analysis, cost-effectiveness analysis cost-effectiveness Provide feedback on preliminary versions of ZIN preliminary on of ZIN versions feedback Provide reports T=4. at Provide feedback on preliminary versions of ZIN preliminary on of ZIN versions feedback Provide reports T=4. at submitted by by submitted the analysis use and cost-effectiveness remove, or keep, to VWS to advice Issue final a Role T=4 at N/A N/A • • • N/A • •

limited number number limited a implementation of of implementation CF scheme. outcomes research study. research outcomes Collaborate with pharmaceutical with pharmaceutical Collaborate medical specialists industry, societyand medical facilitate to studies. research outcomes ZonMW financed financed ZonMW of outcomes research studies studies research of outcomes as part implemented (n=2) of the Collaborate with academic/private with academic/private Collaborate and specialists medical hospitals, implement societies to medical the Implement outcomes research research Implement outcomes data in practice (i.e. studies collection). Finance the Finance studies. research outcomes • N/A Role during outcomes research phase research Role during outcomes • • N/A • N/A • MAH the drug, the drug, the the drug. the drug, the drug, the scientific the scientific conditional financing (CF) scheme.(CF) the conditional financing outcomes research research outcomes Netherlands Organization Netherlands Organization outcomes research proposals proposals research outcomes establishment of outcomes outcomes of establishment The The and Research Health for (ZonMW)Development provided on feedback methodological the drugs. some for Prepare and submit an outcomes an outcomes and submit Prepare provide to proposal research and use on appropriate evidence of cost-effectiveness Prepare and submit evidence on on evidence and submit Prepare of value therapeutic impactbudget and analysis preliminary cost-effectiveness analysis. Assess and appraise appraise and Assess by submitted evidence of value on therapeutic impactbudget and analysis preliminary cost-effectiveness analysis. feedback methodological Provide on the drugs. for proposals Provide feedback on preliminary on feedback Provide T=0. of ZIN reports versions at preliminary on feedback Provide T=0. of ZIN reports versions at Provide input during during input Provide the proposals. research • • • • • • N/A • N/A • Role T=0 at CVZ: College voor Zorgverzekeringen; MAH: Marketing Authorization Holder; NZa: Nederlandse Zorgautoriteit; Holder; Marketing Nederlandse NZa: Zorgautoriteit; Authorization MAH: Zorgverzekeringen; CVZ: voor College – Roles of different stakeholders in in stakeholders different of Roles – Abbreviations: ZIN: Zorginsitituut Nederland; ZonMW: De Nederlandse organisatie voor gezondheidsonderzoek en zorginnovatie. gezondheidsonderzoek voor De Nederlandse organisatie Nederland; ZonMW: Zorginsitituut ZIN: External Public Bodies (i.e. Bodies (i.e. External Public non-ZIN) Zorginstituut Nederland Nederland Zorginstituut CVZ), previously (ZIN; Assessment including: Appraisal Committee, and Assessors Committee IndustryPharmaceutical Academic/ Private Hospitals Private Academic/ Organizations Patient Healthcare Insurers Healthcare Medical Societies Specialists Stakeholders Table 1

Master’s program. Master’s the

SA: Followed courses on qualitative research methods within within methods research qualitative on courses Followed SA:

Master’s program. Master’s

HN: Followed courses on qualitative research methods within within methods research qualitative on courses Followed HN:

scientific literature. scientific have? have?

AM: Conducted several qualitative research projects published in in published projects research qualitative several Conducted AM: researcher the did training or experience What training and Experience 5.

SA: Female SA:

HN: Male HN:

4. Gender 4. researcher male or female? female? or male researcher the Was AM: Male AM:

SA: Master’s student Master’s SA:

HN: Master’s student Master’s HN:

3. Occupation 3. What was their occupation at at occupation their was What AM: Policy Advisor and PhD candidate PhD and Advisor Policy AM: study? the of time the

SA: BSc. SA:

HN: BSc. HN: MD MD

2. Credentials 2. researcher’s credentials? E.g. PhD, PhD, E.g. credentials? researcher’s the were What AM: Pharm D, MSc. D, Pharm AM:

AM and SA conducted conducted SA and AM private stakeholders. private the for interviews the group? group?

1. Inter viewer/facilitator Inter 1. Which author/s conducted conducted author/s Which interview or focus focus or interview the AM and HN conducted conducted HN and AM public stakeholders. public the for interviews the

Personal Characteristics Personal

Domain 1: Research team and reﬂexivity and team Research 1: Domain

Item Item No. Guide questions/description Guide Response

International Journal for Quality in Health Care Health in Quality for Journal International groups. . 2007. Volume 19, Number 6: pp. 349 – 357 – 349 pp. 6: Number 19, Volume 2007. .

Developed from: Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): (COREQ): research qualitative reporting for criteria Consolidated J. Craig P, Sainsbury A, Tong from: Developed 32-item checklist for interviews and focus focus and interviews for checklist 32-item a

completed consolidated criteria for reporting qualitative studies (COREQ) 32-item checklist. 32-item (COREQ) studies qualitative reporting for criteria consolidated completed The – 2 Table

Appendices

260 11

Appendices 261

content analysis analysis content

discourse analysis, ethnography, phenomenology, phenomenology, ethnography, analysis, discourse

this study as detailed by Hsieh et al. (2005). al. et Hsieh by detailed as study this study? e.g. grounded theory, theory, grounded e.g. study? the underpin to and Theory and

methodological orientation of of orientation methodological the was analysis content Directed What methodological orientation was stated stated was orientation methodological What orientation Methodological 9.

Theoretical framework Theoretical

Domain 2: study design design study 2: Domain

interviewers. the

interviewees were aware of any biases/ conflicts of interest of of interest of conflicts biases/ any of aware were interviewees research topic topic research the in interests and

research project beforehand, beforehand, project research the of aims and interviewers the viewer/facilitator? e.g. Bias, assumptions, reasons reasons assumptions, Bias, e.g. viewer/facilitator?

affiliations of of affiliations the of interviewees informing explicitly By inter inter the about reported were characteristics What characteristics Interviewer 8.

research aims. research the and

SA: Participants were informed of of informed were Participants SA: interviewer interviewer the of affiliations the

research aims. research the and

HN: Participants were informed of of informed were Participants HN: interviewer interviewer the of affiliations the research research the doing

researcher? e.g. personal goals, reasons for for reasons goals, personal e.g. researcher? the interviewer the research aims. research the and

7. Participant knowledge of of knowledge Participant 7. participants know about about know participants the did What AM: Participants were informed of of informed were Participants AM: interviewer interviewer the of affiliations the

to study commencement. study to

SA: No relationships with any of of any with relationships No SA: interviewed stakeholders prior prior stakeholders interviewed the

to study commencement. study to

HN: No relationships with any of of any with relationships No HN: interviewed stakeholders prior prior stakeholders interviewed the

professional relationships with any of of any with relationships professional remaining stakeholders. remaining the

relationships with some interviewees from ZIN. AM had no no had AM ZIN. from interviewees some with relationships commencement? commencement?

6. Relationship established Relationship 6. relationship established prior to study study to prior established relationship a Was AM: Through his employment at ZIN, AM had professional professional had AM ZIN, at employment his Through AM:

Relationship with participants with Relationship

Item No. Guide questions/description Guide Response

- continued - 2. Table

and 6 (20%) included two or more. or two included (20%) 6 and

30 interviews conducted, 24 (80%) included one interviewee interviewee one included (80%) 24 conducted, interviews 30 the

sample consisted of 17 (49%) females and 18 (51%) males. Of Of males. (51%) 18 and females (49%) 17 of consisted sample The

master of science (MSc) degree or higher. higher. or degree (MSc) science of master a to equivalent

with senior functions. All interviewees had academic degrees degrees academic had interviewees All functions. senior with

sample consisted of interviewees interviewees of consisted sample the Importantly, manuscript. the date data, demographic e.g. sample? the

important characteristics of of characteristics important the are What sample of Description 16. tables accompanying accompanying tables the in described fully is sample The

during interviews. interviews. during researchers? and

present were participants and researchers the only No, participants the besides present else anyone Was non-participants of Presence 15.

interviewers’ workplace. interviewers’ the

clinic, workplace workplace clinic, phone at at phone the through or interviewees the of workplace the at

data was either collected through face-to-face interviews interviews face-to-face through collected either was data The home, e.g. collected? data the was Where collection data of Setting 14.

Setting

dropped out. dropped Reasons? out? dropped

interviewees approached refused to participate or or participate to refused approached interviewees the of None 13. Non-participation 13. How many people refused to participate or or participate to refused people many How

30 stakeholders. 30

12. Sample size Sample 12. How many participants were in in were participants many How In total, there were 35 interviewees involved, representing representing involved, interviewees 35 were there total, In study? the

face, telephone, mail, email email mail, telephone, face, email invitation. email

Participants were approached by email using using email by approached were Participants standardized a 11. Method of approach of Method 11. How were participants approached? e.g. face-to- e.g. approached? participants were How

or other stakeholders (snowballing). stakeholders other or

further interviewees; whether colleagues from from colleagues whether interviewees; further same institution institution same the

interviewees contacted were provided provided were contacted interviewees chance to recommend recommend to chance the

Financing framework (purposive). Additionally, potential potential Additionally, (purposive). framework Financing

list of stakeholders based on their knowledge of of knowledge their on based stakeholders of list a Conditional Conditional the

convenience, consecutive, snowball snowball consecutive, convenience, to select participants. In first instance, instance, first In participants. select to authors established established authors the

combination of purposive sampling and snowballing was used used was snowballing and sampling purposive of combination A 10. Sampling 10. How were participants selected? e.g. purposive, purposive, e.g. selected? participants were How

Participant selection Participant

Item No. Guide questions/description Guide Response

- continued - 2. Table

Appendices

262 11

Appendices 263

data for phase 2 stakeholders. 2 phase for data the coded SA and AM

data for phase 1 stakeholders. 1 phase for data the coded HN and AM data? the coded coders data many How coders data of Number 24.

Data analysis analysis Data

Domain 3: analysis and ﬁndings and analysis 3: Domain

interviewees for comment and correction. and comment for interviewees the to sent were comment and/or correction? correction? and/or comment

Extensive summaries that were made based upon audio recordings recordings audio upon based made were that summaries Extensive Were transcripts returned to participants for for participants to returned transcripts Were returned Transcripts 23.

decision not to conduct more interviews. more conduct to not decision the motivated what

Was data saturation discussed? discussed? saturation data Was saturation Data 22. Data saturation was discussed amongst authors. This discussion is is discussion This authors. amongst discussed was saturation Data

group? group?

Each interview lasted between 60 and 90 minutes. 90 and 60 between lasted interview Each focus or views inter the of duration the was What Duration 21.

interview or focus group? focus or interview the

interviews. the after and during made were notes Field Were ﬁeld notes made during and/or after after and/or during made notes ﬁeld Were notes Field 20.

interview recorded. interview the have to refuse stakeholder the did interview data? data? the collect

19. Audio/visual recording Audio/visual 19. Audio recording was used for 29 (97%) interviews. In only 1 (3%) (3%) 1 only In interviews. (97%) 29 for used was recording Audio to recording visual or audio use research the Did

many? many?

No repeat interviews were conducted. were interviews repeat No 18. Repeat interviews Repeat 18. Were repeat inter views carried out? If yes, how how yes, If out? carried views inter repeat Were

and future perspectives. future and

perceived functioning of CF, impact of CF and conclusions on CF CF on conclusions and CF of impact CF, of functioning perceived

following domains: perceived aims of CF, CF, of aims perceived domains: following the addressed guide The

2 stakeholders. 2

same interview guide was used in 2017 for interviewing phase phase interviewing for 2017 in used was guide interview same The

interviews.

guide was finalized and used for for used and finalized was guide the Subsequently, remaining remaining the

1 stakeholders. stakeholders. 1 guide was pilot tested in in tested pilot was guide The first 3 interviews. interviews. 3 first the authors? Was it pilot tested? tested? pilot it Was authors? the

An interview guide was developed in 2016 for interviewing phase phase interviewing for 2016 in developed was guide interview An 17. Interview guide Interview 17. Were questions, prompts, guides provided by by provided guides prompts, questions, Were

Data collection Data

Item No. Guide questions/description Guide Response

- continued - 2. Table

feedback period. feedback the during

manuscript and by interviewees interviewees by and manuscript the of authors the by reviewed of minor themes? themes? minor of

description of diverse cases or discussion discussion or cases diverse of description a there Is themes minor of Clarity 32. Description of diverse cases and discussion of minor themes was was themes minor of discussion and cases diverse of Description

manuscript. the of authors the ﬁndings? the

major themes was reviewed by by reviewed was themes major the of presentation the of Clarity Were major themes clearly presented in in presented clearly themes major Were themes major of Clarity 31.

manuscript. the of authors the ﬁndings? the and

data and findings was reviewed by by reviewed was findings and data the between Consistency presented data the between consistency there Was consistent ﬁndings and Data 30.

identiﬁed? e.g. participant number participant e.g. identiﬁed?

manuscript. the in themes/ﬁndings? Was each quotation quotation each Was themes/ﬁndings? the

Quotations have been presented to illustrate themes and findings findings and themes illustrate to presented been have Quotations 29. Quotations presented Quotations 29. Were participant quotations presented to illustrate illustrate to presented quotations participant Were

Reporting Reporting

manuscript. the on

28. Participant checking Participant 28. Did participants provide feedback on on feedback provide participants Did Interviewees have been provided provided been have Interviewees ﬁndings? the chance to provide feedback feedback provide to chance the

data? data? the

MaxQDA version 11 (release 11.1.2, build 151125) was used. was 151125) build 11.1.2, (release 11 version MaxQDA 27. Software 27. What software, if applicable, was used to manage manage to used was applicable, if software, What

data? data? the

Themes were derived from from derived were Themes data. the 26. Derivation of themes of Derivation 26. Were themes identiﬁed in advance or derived from from derived or advance in identiﬁed themes Were

coding tree developed. tree coding the for

interview guide. Please see Figure 3 in in 3 Figure see Please guide. interview the in stated Appendix Appendix the

25. Description of of Description 25. coding tree coding the Did authors provide provide authors Did description of of description a coding tree? tree? coding the basic framework is based upon upon based is framework basic The domains and questions questions and domains the

Item No. Guide questions/description Guide Response

- continued - 2. Table

Appendices

264 11

Appendices 265

at T=0. (29%) T=0. at

Lack of implementation of advice on study design design study on advice of implementation of Lack

critical parameters). (29%) parameters). critical

Excessive collection of data (e.g. additional, non- additional, (e.g. data of collection Excessive

research. (36%) research. additional, non-critical parameters) (27%) parameters) non-critical additional,

Low patient recruitment rates for outcomes outcomes for rates recruitment patient Low Excessive collection of data (e.g. (e.g. data of collection Excessive

research. (33%) research.

of outcomes research. (36%) research. outcomes of outcomes of value influence

Difficult to set up patient registries. (25%) registries. patient up set to Difficult Rapid changes in clinical practice influence value value influence practice clinical in changes Rapid Rapid changes in clinical practice practice clinical in changes Rapid

additional, non-critical parameters). (25%) parameters). non-critical additional, research. (57%) research. outcomes research. (33%) research. outcomes

Excessive collection of data (e.g. (e.g. data of collection Excessive Poor methodological quality of outcomes outcomes of quality methodological Poor Poor methodological quality of of quality methodological Poor

research. (31%) research. points). (64%) points). design and cut-off points) (40%) points) cut-off and design aspects

influence value of outcomes outcomes of value influence

outcomes studies (e.g. study design and cut-off cut-off and design study (e.g. studies outcomes for outcomes studies (e.g. study study (e.g. studies outcomes for Methodological Methodological

Rapid changes in clinical practice practice clinical in changes Rapid No clear methodological requirements for for requirements methodological clear No No clear methodological requirements requirements methodological clear No – CF of Functioning

studies (27%) studies

Financing of outcomes research. (36%) research. outcomes of Financing Financing of outcomes research research outcomes of Financing

(36%)

Lack of monitoring mechanisms (27%) mechanisms monitoring of Lack Monitoring of progress throughout 4-year period. period. 4-year throughout progress of Monitoring

across stakeholders. (43%) stakeholders. across conflicting of interests (43%) interests of conflicting

Ambiguous procedure and distribution of roles roles of distribution and procedure Ambiguous Distribution of roles in relation to to relation in roles of Distribution

roles across stakeholders. (56%) stakeholders. across roles procedural ambiguity. (71%) ambiguity. procedural

Ambiguous procedure and distribution of of distribution and procedure Ambiguous Ambiguity regarding roles (50%) roles regarding Ambiguity Disincentives due to conflicts of interest and and interest of conflicts to due Disincentives Procedural aspects Procedural

4-year time period (90%) period time 4-year 4-year time period. (86%) period. time 4-year The 4-year time period. (94%) period. time 4-year The Functioning of CF – – CF of Functioning

Topic All Stakeholders (n=30; %) (n=30; Stakeholders All Phase 1 Stakeholders (n=14; %) (n=14; Stakeholders 1 Phase Phase 2 Stakeholders (n=16; %) (n=16; Stakeholders 2 Phase

phase 1 stakeholders and phase 2 stakeholders. stakeholders. 2 phase and stakeholders 1 phase

– Summary of of Summary – 3 Table The The analysis. content directed of results the themes identified for all topics are cited in order of decreasing mention for all stakeholders, stakeholders, all for mention decreasing of order in cited are topics all for identified themes

practice. (25%) practice.

More appropriate use of drugs in clinical clinical in drugs of use appropriate More

stakeholder collaboration. (25%) collaboration. stakeholder stakeholder collaboration. (29%) collaboration. stakeholder stakeholder collaboration. (27%) collaboration. stakeholder

Increased awareness of need for multi- for need of awareness Increased Increased awareness of need for multi- for need of awareness Increased Increased awareness of need for multi- for need of awareness Increased

public debate. (31%) debate. public agreements. (33%) agreements.

healthcare became important topics of of topics important became healthcare of drugs in routine clinical practice. (36%) practice. clinical routine in drugs of entry managed of implementation the

Cost-effectiveness and displacement of of displacement and Cost-effectiveness use the on generated evidence (real-world) More (Policy) Experiences gained in in gained Experiences (Policy)

agreements. (31%) agreements. agreements. (36%) agreements. practice. (40%) practice.

implementation of managed entry entry managed of implementation the implementation of managed entry entry managed of implementation the use of drugs in routine clinical clinical routine in drugs of use the on

(Policy) Experiences gained in in gained Experiences (Policy) (Policy) Experiences gained in in gained Experiences (Policy) More (real-world) evidence generated generated evidence (real-world) More

practice. (44%) practice. debate. (79%) debate. topics of public debate. (53%) debate. public of topics

use of drugs in routine clinical clinical routine in drugs of use the on healthcare became important topics of public public of topics important became healthcare of healthcare became important important became healthcare of Positive effects Positive

More (real-world) evidence generated generated evidence (real-world) More Cost-effectiveness and displacement of of displacement and Cost-effectiveness Cost-effectiveness and displacement displacement and Cost-effectiveness Impact of CF – – CF of Impact

non-binding criteria on cost-effectiveness. (29%) cost-effectiveness. on criteria non-binding

Influence of legal criteria on effectiveness versus versus effectiveness on criteria legal of Influence

process. (43%) process. the process. (33%) process. the of start the

appropriate use were apparent from from apparent were use appropriate start of of start the or appropriate use were apparent from from apparent were use appropriate or

Conclusions on cost-effectiveness and/or and/or cost-effectiveness on Conclusions Conclusions on cost-effectiveness and/ cost-effectiveness on Conclusions

process. (25%) process. the of start the beyond T=4. (40%) T=4. beyond

or appropriate use were apparent from from apparent were use appropriate or decisions at T=4. (57%) T=4. at decisions national reimbursement package package reimbursement national the

Conclusions on cost-effectiveness and/ cost-effectiveness on Conclusions External (political) factors influenced final final influenced factors (political) External Impossible to remove drugs from from drugs remove to Impossible

T=4. (25%) T=4. T=4. (43%) T=4.

outcomes research to final decisions at at decisions final to research outcomes research to final decisions at T=4. (64%) T=4. at decisions final to research outcomes research to final decisions at at decisions final to research outcomes

Minimal contribution of results from from results of contribution Minimal Minimal contribution of results from outcomes outcomes from results of contribution Minimal Minimal contribution of results from from results of contribution Minimal aspects

decisions at T=4. (44%) T=4. at decisions final decisions at T=4. (50%) T=4. at decisions final reimbursement package beyond T=4. (64%) T=4. beyond package reimbursement Decision-making Decision-making

Impossible to remove drugs from from drugs remove to Impossible External (political) factors influenced final final influenced factors (political) External national the External (political) factors influenced influenced factors (political) External Functioning of CF – – CF of Functioning

Topic All Stakeholders (n=30; %) (n=30; Stakeholders All Phase 1 Stakeholders (n=14; %) (n=14; Stakeholders 1 Phase Phase 2 Stakeholders (n=16; %) (n=16; Stakeholders 2 Phase

- continued - 3 Table

Appendices

266 11 Appendices

267

Abbreviations: CF: Conditional Financing. Conditional CF: Abbreviations:

diverging SH interests. (25%) interests. SH diverging

in policy framework in relation to to relation in framework policy in effectiveness at T=0. (30%) T=0. at effectiveness

Re-consideration of underlying incentives incentives underlying of Re-consideration Make (definitive) conclusions on cost- on conclusions (definitive) Make

process. (30%) process.

effectiveness at T=0. (29%) T=0. at effectiveness of progress per drug throughout CF CF throughout drug per progress of

Abandon rigid 4-year timeframe. (25%) timeframe. 4-year rigid Abandon Make (definitive) conclusions on cost- on conclusions (definitive) Make Need for extra steps for monitoring monitoring for steps extra for Need

collaboration. (25%) collaboration.

policy framework and better SH SH better and framework policy

per drug throughout CF process. (43%) process. CF throughout drug per CF procedure. (30%) procedure. CF the throughout roles

Agreement amongst SH’s on aims of of aims on SH’s amongst Agreement Need for extra steps for monitoring of progress progress of monitoring for steps extra for Need Better governance and distribution of of distribution and governance Better

(33%)

interests. (43%) interests. relation to diverging SH interests. interests. SH diverging to relation

effectiveness at T=0. (31%) T=0. at effectiveness policy framework in relation to diverging SH SH diverging to relation in framework policy incentives in policy framework in in framework policy in incentives

Make (definitive) conclusions on cost- on conclusions (definitive) Make Re-consideration of underlying incentives in in incentives underlying of Re-consideration Re-consideration of underlying underlying of Re-consideration

collaboration. (37%) collaboration. Improvement of CF of Improvement

CF procedure. (38%) procedure. CF the throughout roles framework and better SH collaboration. (50%) collaboration. SH better and framework of policy framework and better SH SH better and framework policy of future perspectives - - perspectives future

Better governance and distribution of of distribution and governance Better Agreement amongst SH’s on aims of policy policy of aims on SH’s amongst Agreement Agreement amongst SH’s on aims aims on SH’s amongst Agreement Conclusions and and Conclusions

postpone difficult decisions. (36%) decisions. difficult postpone

Negative effects Negative reimbursement package and an excuse to to excuse an and package reimbursement

No major or minor themes identified. themes minor or major No No major or minor themes identified. themes minor or major No CF perceived as as perceived CF “back-door” into into “back-door” a national national the Impact of CF – – CF of Impact

Topic All Stakeholders (n=30; %) (n=30; Stakeholders All Phase 1 Stakeholders (n=14; %) (n=14; Stakeholders 1 Phase Phase 2 Stakeholders (n=16; %) (n=16; Stakeholders 2 Phase

- continued - 3 Table parliamentary the CF process. It becomes CF process. uncertainty thus was the The The start of the clinicians and medical specialists, data medical specialists, clinicians and the monitoring and auditing of research progress progress auditing of research and monitoring appropriate use of drugs in clinical practice.” – PI1 – practice.” clinical use of drugs in appropriate registry within 4 years. However, for overall survival period overall for However, registry 4 years. within the support of a long time before that happens. For example, extra overall extra overall example, For happens. that long time before 4-year registry would not be informative.” – MS3 not be informative.” registry would 4-year the a a the tug-of-war between ZIN and pharmaceutical companies. That That tug-of-war companies. pharmaceutical and ZIN between a process.” – MS2 process.” responsibilities of other parties, such as healthcare providers and of other parties, providers responsibilities such as healthcare the the quality of data collected was affected.” – PI3 affected.” qualitywas collected of data 4-year period is impractical. It may have been motivated by by been motivated It period is impractical. have may 4-year context of drugs, this choice is illogical because for some drugs one knows some drugs because for is illogical this choice of drugs, context the end of the the the responsibility of clinicians and medical specialists. However, they were often less willing they were However, and medical specialists. of clinicians responsibility the whole procedure.” – PO1 whole procedure.” the way.” – HI2 way.” the public perception of CF was that it was it was that was of CF public perception The The Quotes from phase 2 interviews from Quotes “We have repeatedly stated that that stated repeatedly have “We “ collection “Data was at implemented and been designed should have mechanisms “Monitoring period of 4 years. However, for for However, period 4 years. of quickly take others it could if they work, for while meant that little attention was paid to to paid was attention little that meant for responsible also who were medical specialists, to cooperate with industry on doing so. Without Without industry with cooperate so. on doing to useless to inspect results at T=4, otherwise. By then, it is difficult to correct for any errors that have happened have that any errors for otherwise.T=4, correct to By then, it is difficult inspect at useless to results along aspect not function did was of CF that well “Another survival of 7 months can be adequately captured in survival captured adequately can be of 7 months of up to 9 to 10 years in metastatic breast cancer, cancer, breast in metastatic 10 years 9 to of up to “Although it was often clear at T=0 what should have been done, there was much confusion regarding T=4. InT=4. regarding much confusion was there done, been have should what T=0 often clear at was it “Although and why. were responsibilities stakeholder’s each what was unclear it year, final that towards obvious increasingly throughout throughout collection cost excessive time and excessive collection cost decisions drugs the negative negative a hands of the the authority end to framework? What framework? the the studies. If ZIN one wants studies. the too rigid and short and rigid needs One too timeframe. a data from outcomes research to ZIN (i.e. to research outcomes from data the drug would no longer be reimbursed, causing longer be reimbursed, no drug would appraisal committee (ACP), ZonMW and VWS in and ZonMW (ACP), committee appraisal studies may have delivered better results; for for results; better delivered have studies may the more direct involvement by ZIN in evaluating ZIN in by directinvolvement more the a the case that better evidence was to be expected if to was evidence better case that initial decision for reimbursement would be reversed. This be reversed. would reimbursement for initial decision the roles of roles the specific stakeholder accountable.” - ZA1 accountable.” stakeholder specific the actual implementation of data collectionactual in of data was implementation a mid-term evaluation of progress of of progress evaluation mid-term outcomes research studies would have proceeded longer. On longer. proceeded have would studies research outcomes other hand, progress of progress pharmaceutical company or healthcare providers) making it difficult providers) to or healthcare company pharmaceutical decision-making process in this framework?” – ZA3 this framework?” in decision-makingprocess a The The “This [4-year period, ed.] was was “This ed.] period, [4-year to monitor this, then ZIN should also be given be given also then ZIN should this, monitor to the hold made eventually which stakeholder me clear to “It never was “ of monitoringno mention was there “Throughout involvement, my mention even wasn’t There ed.] ZIN, studies [by research of outcomes of of inadequate are or that falter to seem that studies research outcomes – ZS4 quality.” “It wasn’t the the tailored approaches to be able to determine how much time would be time would much how determine to be able to approaches tailored cost- use or on appropriate evidence sufficient generate needed to ZW1 – disease indications.” for drugs in different effectiveness directlywas which stakeholder us clear to not was it worse, “Even delivering for responsible governs who example: framework.in this For were Quotes from phase 1 interviews from Quotes being not cost-effective (which was to be expected for many of these for many expectedto be was (which cost-effective being not drugs), that mean would advice on reimbursement be issued by ZIN at T=4 due to to due T=4 ZIN at by be issued on reimbursement advice pharmaceutical industry pharmaceutical little incentive who had specialists and medical Should on cost-effectiveness. research conduct robust to the the problems for both stakeholders.” – ZW2 both stakeholders.” for problems example through yearly progress meetings.” – FG1 meetings.” progress yearly through example eriod Distribution of Distribution of relation in roles conflicting of to interests of Lack monitoring mechanisms 4-year time time 4-year p Ambiguity roles regarding Theme – Illustrative – quotes for themes identified interviews. in Functioning of CF – – of CF Functioning aspects Procedural Functioning of CF – – CF of Functioning aspects Procedural Functioning of CF – – CF of Functioning aspects Procedural Functioning of CF – – of CF Functioning aspects Procedural Topic Table 4

Appendices

268 11 Appendices 269 11 the parliamentarythe CF process. It becomes CF process. uncertainty thus was the The The start of the clinicians and medical specialists, data medical specialists, clinicians and the monitoring and auditing of research progress progress auditing of research and monitoring appropriate use of drugs in clinical practice.” – PI1 – practice.” clinical use of drugs in appropriate registry within 4 years. However, for overall survival period overall for However, registry 4 years. within the supportof a long time before that happens. For example, extra overall extra overall example, For happens. that long time before 4-year registry would not be informative.” – MS3 not be informative.” registry would 4-year the a a the tug-of-war between ZIN and pharmaceutical companies. That That tug-of-war companies. pharmaceutical and ZIN between a process.” – MS2 process.” responsibilities of other parties, such as healthcare providers and of other parties, providers responsibilities such as healthcare the the quality of data collected was affected.” – PI3 affected.” qualitywas collected of data 4-year period is impractical. It may have been motivated by by been motivated It period is impractical. have may 4-year context of drugs, this choice is illogical because for some drugs one knows some drugs because for is illogical this choice of drugs, context the end of the the the responsibility of clinicians and medical specialists. However, they were often less willing they were However, and medical specialists. of clinicians responsibility the whole procedure.” – PO1 whole procedure.” way.” – HI2 way.” public perception of CF was that it was it was that was of CF public perception The The Quotes from phase 2 interviews from Quotes “Monitoring mechanisms should have been designed and implemented at at implemented and been designed should have mechanisms “Monitoring period of 4 years. However, for for However, period 4 years. of “Data collection “Data was quickly take others it could they work, if for while “ “We have repeatedly stated that that stated repeatedly have “We meant that little attention was paid to to paid was attention little that meant medical specialists, who were also responsible for for responsible also who were medical specialists, along the along useless to inspect results at T=4, otherwise. By then, it is difficult to correct for any errors that have happened have that any errors for otherwise.T=4, correct to By difficult then, it is inspect at useless to results aspect not function did was of CF that well “Another survival of 7 months can be adequately captured in survival captured adequately can be of 7 months cancer, breast in metastatic 10 years 9 to of up to to cooperate with industry on doing so. Without Without industry with cooperate so. on doing to that final year, it was unclear what each stakeholder’s responsibilities were and why. and why. were responsibilities stakeholder’s each what was unclear it year, final that “Although it was often clear at T=0 what should have been done, there was much confusion regarding T=4. InT=4. regarding much confusion was there done, been should have what T=0 often clear at was it “Although towards obvious increasingly throughout the the throughout collection cost excessive time and excessive collection cost the decisions the a negative negative a hands of the drugs the the authority end to framework? What framework? the the studies. If ZIN one wants studies. the too rigid and short and rigid needs One too timeframe. a data from outcomes research to ZIN (i.e. to research outcomes from data the drug would no longer be reimbursed, causing longer be reimbursed, no drug would appraisal committee (ACP), ZonMW and VWS in and ZonMW (ACP), committee appraisal studies may have delivered better results; for for results; better delivered have studies may the more direct involvement by ZIN in evaluating ZIN in by directinvolvement more the a the case that better evidence was to be expected if to was evidence better case that initial decision for reimbursement would be reversed. This be reversed. would reimbursement for initial decision the roles of roles specific stakeholder accountable.” - ZA1 accountable.” stakeholder specific the actual implementation of data collectionactual in of data was implementation mid-term evaluation of progress of of progress evaluation mid-term outcomes research studies would have proceeded longer. On longer. proceeded have would studies research outcomes other hand, progress of progress pharmaceutical company or healthcare providers) making it difficult providers) to or healthcare company pharmaceutical decision-making process in this framework?” – ZA3 this framework?” in decision-makingprocess The The “This [4-year period, ed.] was was “This ed.] period, [4-year “ of monitoringno mention was there “Throughout involvement, my mention even wasn’t There ed.] ZIN, studies [by research of outcomes a of be given also then ZIN should this, monitor to of inadequate are or that falter to seem that studies research outcomes – ZS4 quality.” “It wasn’t the the tailored approaches to be able to determine how much time would be time would much how determine to be able to approaches tailored cost- use or on appropriate evidence sufficient generate needed to ZW1 – disease indications.” for drugs in different effectiveness directlywas which stakeholder us clear to not was it worse, “Even delivering for responsible the a hold made eventually which stakeholder me clear to “It never was governs who example: framework.in this For were Quotes from phase 1 interviews from Quotes the pharmaceutical industry pharmaceutical little incentive who had specialists and medical Should on cost-effectiveness. research conduct robust to to due T=4 ZIN at by be issued on reimbursement advice of these for many expectedto be was (which cost-effective being not the drugs), that mean would the example through yearly progress meetings.” – FG1 meetings.” progress yearly through example problems for both stakeholders.” – ZW2 both stakeholders.” for problems eriod Distribution of Distribution of relation in roles conflicting of to interests of Lack monitoring mechanisms 4-year time time 4-year p Ambiguity roles regarding Theme – Illustrative – quotes for themes identified interviews. in Functioning of CF – – CF of Functioning aspects Procedural – of CF Functioning aspects Procedural Functioning of CF – – of CF Functioning aspects Procedural – CF of Functioning aspects Procedural Topic Table 4 protocol. From From protocol. the huge set of variables for for of variables huge set a lack of knowledge on what you exactlylack of knowledge want you on what lack of knowledge on what you exactlylack of knowledge want you on what a a fundamental design error.” – MS3 error.” design fundamental fundamental design error. In some cases, extra In variables cases, some error. design fundamental a a Therefore, it is Therefore, studies were running. These were subsequently added to added to subsequently were These running. studies were the case that data collection did not happen but rather collection data rather but happen did not case that case that data collection did not happen but rather collection data rather but happen did not case that large number of stakeholders with different interests together, you get together, interests different of stakeholders with number large a the the governance and financing of patient registries differed significantly per registry.” – PI3 registry.” per significantly differed registries financing of patient and governance methodological viewpoint, this cannot be allowed in prospective research.” – MS3 research.” in prospective be allowed this cannot viewpoint, methodological The The Quotes from phase 2 interviews from Quotes to collect and in which patient groups. groups. collect patient and in which to “It not was “ bring“If you a which information is desired. In contrast, we should be looking to core datasets that really matter within smaller within matter really that datasets should be looking core we to In contrast, is desired. which information perspectives in outcomes of patient be looking also should incorporation better to We populations. patient – PI1 studies.” research “It not was it is Therefore, groups. collect patient and in which to ZIN while by requested were design control control time.” – ZS3 time.” the the the period 4 years of quality of the the research.” – FG1 research.” latter case, patients case, latter feeling that this that feeling studies. This resulted in resulted This studies. hands of public bodies.of public hands the the the the the intervention and control and control intervention new drug became new drug lack of independence (i.e. of independence lack financing structurefinancing of outcomes the pharmaceutical industrypharmaceutical also infrastructure required to collect to required infrastructure the the the evidence.” – ZS2 evidence.” The The outcomes of outcomes The The the the predominantly clinical perspective, rather than rather perspective, clinical predominantly a relevance of of relevance aims and expectations of CF framework, there was no consensus on no consensus was CF framework, there control group or that that or group control a the the the first round of CF, several outcomes research studies were financedwere studies research outcomes several round of CF, first positive outcome. In these cases there was not financing through financing not was In cases there these outcome. positive outcomes research studies. Some expected stakeholders RCT’s studies. research outcomes quality of information submitted at T=4 and T=4 at submitted quality of information a change in standard clinical practice throughout clinical practice throughout change in standard absence of absence data was also not readily available.” – ZW2 available.” readily not also was data remaining studies being financed by pharmaceutical industry. Within industry. pharmaceutical by financed being studies remaining mandatory made and were questions research some protocols, study the main problem encountered with outcomes research studies was research with outcomes encountered main problem the cost-effectiveness perspective. perspective. cost-effectiveness The The A outcomes research studies at T=4 was quite poor. Elements such as low such Elements poor. quite T=4 was at studies research outcomes collection data practice and in clinical fragmented recruitment, patient the “ “Throughout “In Quotes from phase 1 interviews from Quotes who did not wish to be treated with be treated wish to who did not a affected really “ the In comparisons. not suitable for were groups huge databases with huge databases of no was there Furthermore, did not. while others be conducted, to on consensus by ZonMW. However, some stakeholders had stakeholders some However, ZonMW. by rounds of upon them. subsequent In forced was mechanism financing with studies, outcomes specific used for only were funds ZonMW CF, the the Many specialists. medical by raised non-mandatoryother were questions and questions research numerous with structuresfinancing emerged, PE3 – address.” to attempted teams research that resources numerous how see “It to disappointing quite is the “There reflect outcomes on which time to not enough was ZS3: relevant. ZIN deemed parameters raised critical questions: they wanted to know to product when their would criticalthey wanted questions: raised on such outcome based reimbursement for “adequate” deemed be also ZIN since questions such to be provided could No answer parameters. at areas drugs/ indication these with experience had little research studies removed all authority all from studies removed research research industryIt let pharmaceutical to not wise was outcomes finance – ZA3 products.” own on their studies was emerged that problem “Another group. This undoubtedly led to selection making comparisons led to bias, undoubtedly any This group. – FG1 of little value.” independent research, independent financing of research). Two outcomes Two research). of financing independent research, independent assessed were ed.] body, [public performed studies ZonMW by research with industry on influence no thus Low quality quality Low of outcomes research No clear No clear methodological requirements outcomes for studies Financing of of Financing outcomes studies research Theme

- continued - Methodological Methodological aspects Functioning of CF – – of CF Functioning Methodological Methodological aspects Functioning of CF – – of CF Functioning Functioning of CF – – of CF Functioning aspects Procedural Topic Table 4

Appendices

270 11 Appendices 271 11 protocol. From From protocol. the huge set of variables for for of variables huge set a lack of knowledge on what you exactlylack of knowledge want you on what lack of knowledge on what you exactlylack of knowledge want you on what a a fundamental design error.” – MS3 error.” design fundamental fundamental design error. In some cases, extra In variablescases, some error. design fundamental a a studies were running. These were subsequently added to added to subsequently were These running. studies were the case that data collection did not happen but rather collection data rather but happen did not case that case that data collection did not happen but rather collection data rather but happen did not case that large number of stakeholders with different interests together, you get together, interests different of stakeholders with number large a the the governance and financing of patient registries differed significantly per registry.” – PI3 registry.” per significantly differed registries financing of patient and governance methodological viewpoint, this cannot be allowed in prospective research.” – MS3 research.” in prospective be allowed this cannot viewpoint, methodological The The Quotes from phase 2 interviews from Quotes to collect and in which patient groups. Therefore, it is Therefore, groups. collect patient and in which to “It not was “If you bring“If you “ a “It not was research studies.” – PI1 studies.” research patient populations. We should also be looking to better incorporation of patient perspectives in outcomes perspectives in outcomes of patient be looking also should incorporation better to We populations. patient which information is desired. In contrast, we should be looking to core datasets that really matter within smaller within matter really that datasets should be looking core we to In contrast, is desired. which information it is Therefore, groups. collect patient and in which to ZIN while by requested were time.” – ZS3 time.” the design the the control control the the period 4 years of quality of the the research.” – FG1 research.” latter case, patients case, latter feeling that this that feeling studies. This resulted in resulted This studies. hands of public bodies.of public hands the the the the the intervention and control and control intervention new drug became new drug lack of independence (i.e. independence lack of financing structurefinancing of outcomes the pharmaceutical industrypharmaceutical also infrastructure required to collect to required infrastructure the the the evidence.” – ZS2 evidence.” The The outcomes of outcomes The The the the predominantly clinical perspective, rather than rather perspective, clinical predominantly a relevance of of relevance aims and expectations of CF framework, there was no consensus on no consensus was CF framework, there control group or that that or group control a the the first round of CF, several outcomes research studies were financedwere studies research outcomes several round of CF, first positive outcome. In these cases there was not financing through financing not was In cases there these outcome. positive outcomes research studies. Some expected stakeholders RCT’s studies. research outcomes quality of information submitted at T=4 and T=4 at submitted quality of information change in standard clinical practice throughout clinical practice throughout change in standard data was also not readily available.” – ZW2 available.” not readily also was data absence of absence remaining studies being financed by pharmaceutical industry. Within industry. pharmaceutical by financed being studies remaining mandatory made and were questions research some protocols, study main problem encountered with outcomes research studies was research with outcomes encountered main problem cost-effectiveness perspective. perspective. cost-effectiveness The The A a “Throughoutthe the of no was there Furthermore, did not. while others be conducted, to on consensus with huge databases “ as low such Elements poor. quite was T=4 at studies research outcomes collection data practice and in clinical fragmented recruitment, patient the “In the “In had stakeholders some However, ZonMW. by rounds of upon them. In subsequent forced was mechanism financing with studies, outcomes specific used for only were funds ZonMW CF, the the Many specialists. medical by raised non-mandatoryother were questions and questions research numerous with structuresfinancing emerged, PE3 – address.” to attempted teams research that resources numerous how see “It to disappointing quite is Quotes from phase 1 interviews from Quotes the “There reflect outcomes on which time to not enough was ZS3: relevant. ZIN deemed parameters know to product when their would criticalthey wanted questions: raised on such outcome based reimbursement for “adequate” deemed be also ZIN since questions such to be provided could No answer parameters. at areas drugs/ indication these with experience had little affected really “ In comparisons. suitable for not were groups with be treated wish to who did not the research studies removed all authority all from removed studies research research industryIt let pharmaceutical to not wise was outcomes finance – ZA3 products.” own on their studies was emerged that problem “Another group. This undoubtedly led to selection making comparisons led to bias, undoubtedly any This group. – FG1 of little value.” industry on influence no thus independent research, independent financing of research). Two outcomes Two research). of financing independent research, independent assessed were ed.] body, [public performed studies ZonMW by research a with No clear No clear methodological requirements outcomes for studies quality Low of outcomes research Financing of of Financing outcomes studies research Theme

- continued - Functioning of CF – – of CF Functioning Methodological aspects – of CF Functioning Methodological aspects Functioning of CF – – of CF Functioning aspects Procedural Topic Table 4 public outcry The The conclusion would would conclusion reimbursement reimbursement items onitems drugs we choose to choose drugs we the the the the remaining drugs in CF.” – PO2 in CF.” drugs remaining product from product from a the reimbursement package.” – HO2 package.” reimbursement drug but no new conclusions.” – PI2 no new conclusions.” drug but the case in hemato-oncology, metastatic melanoma metastatic case in hemato-oncology, the national reimbursement package, becomes it reimbursement national the healthcare environment is very and where dynamic environment healthcare bigger picture.” – MS1 bigger picture.” the the market because of which treatments can move through through move can market treatments of which because the examples of Pompe’s and Fabry’s diseases. diseases. and Fabry’s Pompe’s of examples drug out of drug out the the the data at generated through outcomes research studies by T=4 was studies by research outcomes through generated at data the drug while it is in drug while drug cost €200,000 per patient and that we theoretically cannot generate generate cannot theoretically we that and per patient €200,000 drug cost a data on initial use patterns are no longer relevant.” – PO2 relevant.” no longer are use patterns on initial data the the relevant ones by T=4. This is This T=4. by ones relevant result, one lost sight of lost sight one result, There were many other unsolved, interesting questions though.” – HO2 questions though.” interesting unsolved, other many were There lasting impact on how ZIN proceeded with lasting impact ZIN proceeded how on package. Your negotiation power and argumentation to do so is immediately is immediately so do to argumentation and power negotiation Your package. a conclusion that that conclusion a the market. Research designed now can therefore not mean much 2 years later.” – HI3 later.” much 2 years not mean can therefore now designed market. Research the patient with with patient the a the field of oncology, many drugs rapidly reach rapidly drugs many oncology, field of re-assessment of drugs by ZIN was quite technocratic; ZIN only wanted to tick off all tick off to wanted ZIN only technocratic; re-assessmentquite ZIN was by drugs of submissions checklist. As the The The Quotes from phase 2 interviews from Quotes “Once you have treated treated have you “Once on new insights on omalizumab provided research “Outcomes “One disadvantage of CF is that it takes too long, while long, it takes too of CF is that disadvantage “One remove to difficult extremely it becomes “Withtypes these of frameworks, “ many new drugs reach reach new drugs many package. This is usually due to external political pressure by stakeholders. In my opinion, ZIN should dare to say to should dare opinion, ZIN In my stakeholders. by external political pressure is usually due to This package. – HI1 often.” more ed.]. reimbursement, [to “no” “Externalgiven factors certainly had an influence, difficult to remove it from it remove to difficult be: that we already knew. We knew We knew. already we be: that per patient. QALY than 1 more “We were excited to being with our outcomes research. Not because we didn’t know what didn’t because we Not research. outcomes being with our to excited were “We “In around these two drugs left two these around diminished.” – HI3 diminished.” as also take our responsibility should we already, years 10 using this drug for been treated already “If have patients cut to and not suddenly decide decision makers the different treatment lines or be administered in different combinations. Because of that, Because of that, combinations. in different or be administered lines treatment different no longer T=0 are assess at Therefore, cancer… and prostate “ZIN repeatedly came to came to “ZIN repeatedly of little use because of patient populations being too small to generate significant evidence.” – MS3 evidence.” significant generate small to being too populations because of patient of little use legal The The advice ZINadvice beginning pharmaceutical outcomes outcomes the conclusions at conclusions the the CVZ/ZIN dailywas the the the use of evidence set number of drugsset number answers before, or before, answers a patient.” – ZS1 – patient.” the cost-effectiveness of drugscost-effectiveness ACP could have stayed its stayed have could ACP the the the The The question: what can we do can we what question: Appraisal Committee (ACP) had (ACP) Committee Appraisal rapid pace, particularly pace, rapid in This just doesn’t work.” – ZA3 work.” just doesn’t This a collection of new information oncollection of new information the the decisions being made.” – ZS1 made.” decisions being question whether question the summer of 2012, summer of 2012, analysis of observational of analysis decreasing data, the the second-line drug would be included drug would second-line the the a standard of care changed several times several changed of care standard first T=4 dossiers. the initial decision to remove remove to initial decision decision were judged to be too large.” – PE1 large.” be too judged to decision were information available at T=0. Even after conduct ofT=0. Even at available information the drug would never be cost-effective. These warningsThese be cost-effective. never drug would non-robust evidence generated by by generated evidence non-robust evening news in association with drugs for Pompe’s Pompe’s for drugs with in association news evening outcomes research studies in pharmacoeconomic in pharmacoeconomic studies research outcomes permanent solution by by solution permanent case that case that CF procedure, complicating complicating CF procedure, uncertainties relating to to uncertainties relating the a the the result, result, the the the the a the year later.” – PE3 later.” year the the studies?” – ZS4 – studies?” outcomes at T=4. People said too easily at at easily said too People T=4. at outcomes a national reimbursement package had to be reversed. package be reversed. reimbursement had to national the the registry, only to find out that it has been registered for first-linefor registered it has been out that find to only registry, the a studies, studies, citizen for for citizen fundamental problem resides in resides problem fundamental A “Unfortunately, “Unfortunately, that warned we cases “In many already to T=0 at evidence had enough We useless. was these drugs that conclude research studies actually delivered more useful information at T=4 inT=4 at useful information more actually studies delivered research to comparison “This when very became apparent for issue advice to it still remains opinion, “In my “It often was Quotes from phase 1 interviews from Quotes models.” – ZS2 models.” generated by by generated throughout throughout ground on negative advice on reimbursement but could also not punishalso could but on reimbursement advice on negative ground the the from that say could you So in some cases, ignored. were oncology. As As oncology. into into treatment at is happening “Drug development industry.” – ZA1 – industry.” “External political factors certainly on influence had an T=4. In issued at eventually during, implications of implications – ZW2 T=0.” at obvious already T=4 were outcomes these need really do we been wondering: should have “One know not already Did we studies? research mentioned in in mentioned and Fabry’s disease. Patients also deliver pressure by (rightly) saying that saying (rightly) by pressure deliver also Patients disease. and Fabry’s accusations and Emotional responses not be denied hope. they should impact definitely front on that remained large at T=4.” – PE2 T=4.” at large remained “ but temporarily provided is that something “It remove very is to difficult as experienced with observational, real-world data? There are many methodological many are There data? with observational, real-world with associated challenges trust in that RCTs would not be conducted within CF and that we will rely on will rely we that within CF and not be conducted would RCTs that our decisions. observational for data Impossible Impossible remove to from drugs reimbursement package T=4 at Conclusions T=4 at already predictable at T=0 Effect of external external of Effect factors) (political advice on ZIN’s Outcomes research contributed decision to little making T=4 Rapid change in Rapid change in practiceclinical Theme

- continued - Decision-making aspects – of CF Functioning Decision-making aspects Functioning of CF – – of CF Functioning Decision-making aspects Functioning of CF – – of CF Functioning Functioning of CF – – CF of Functioning Decision-making aspects Functioning of CF – – of CF Functioning Methodological Methodological aspects Topic Table 4

Appendices

272 11 Appendices 273 11 public outcry The The conclusion would would conclusion items onitems drugs we choose to choose drugs we the the reimbursement reimbursement the the the remaining drugs in CF.” – PO2 in CF.” drugs remaining product from product from a the reimbursement package.” – HO2 package.” reimbursement drug but no new conclusions.” – PI2 no new conclusions.” drug but the case in hemato-oncology, metastatic melanoma metastatic case in hemato-oncology, the national reimbursement package, becomes it reimbursement national the healthcare environment is very and where dynamic environment healthcare bigger picture.” – MS1 bigger picture.” the the market because of which treatments can move through through move can market treatments of which because the examples of Pompe’s and Fabry’s diseases. diseases. and Fabry’s Pompe’s of examples drug out of drug out the the the data at generated through outcomes research studies by T=4 was studies by research outcomes through generated at data the drug while it is in drug while drug cost €200,000 per patient and that we theoretically cannot generate generate cannot theoretically we that and per patient €200,000 drug cost a data on initial use patterns are no longer relevant.” – PO2 relevant.” no longer are use patterns on initial data the the relevant ones by T=4. This is This T=4. by ones relevant result, one lost sight of lost sight one result, lasting impact on how ZIN proceeded with lasting impact ZIN proceeded how on package. Your negotiation power and argumentation to do so is immediately is immediately so do to argumentation and power negotiation Your package. a conclusion that that conclusion a the market. Research designed now can therefore not mean much 2 years later.” – HI3 later.” much 2 years not mean can therefore now designed market. Research the patient with with patient the a the field of oncology, many drugs rapidly reach rapidly drugs many oncology, field of re-assessment of drugs by ZIN was quite technocratic; ZIN only wanted to tick off all tick off to wanted ZIN only technocratic; re-assessmentquite ZIN was by drugs of submissions checklist. As The The Quotes from phase 2 interviews from Quotes difficult to remove it from it remove to difficult “Outcomes research on omalizumab provided new insights on new insights on omalizumab provided research “Outcomes “Once you have treated treated have you “Once “External factors certainly had an influence, given “Externalgiven factors certainly had an influence, “no” [to reimbursement, ed.]. more often.” – HI1 often.” more ed.]. reimbursement, [to “no” package. This is usually due to external political pressure by stakeholders. In my opinion, ZIN should dare to say to should dare opinion, ZIN In my stakeholders. by external pressure political is usually due to This package. “ “With these types of frameworks, it becomes extremely difficult to remove remove to difficult extremely it becomes “With types these of frameworks, many new drugs reach reach new drugs many “One disadvantage of CF is that it takes too long, while long, it takes too of CF is that disadvantage “One be: that we already knew. We knew We knew. already we be: that – HO2 questions though.” interesting unsolved, other many were There per patient. QALY than 1 more decision makers and not suddenly decide to cut to and not suddenly decide decision makers “If patients have already been treated using this drug for 10 years already, we should also take our responsibility as should also take our responsibility we already, years 10 using this drug for been treated already “If have patients diminished.” – HI3 diminished.” the know what didn’t because we Not research. our outcomes being with to excited were “We different treatment lines or be administered in different combinations. Because of that, that, Because of combinations. in different or be administered lines treatment different “In the “In assess at T=0 are no longer T=0 are assess at drugs left two these around and prostate cancer… Therefore, Therefore, cancer… and prostate “ZIN repeatedly came to came to “ZIN repeatedly of little use because of patient populations being too small to generate significant evidence.” – MS3 evidence.” significant generate small to being too populations because of patient of little use The legal The advice ZINadvice the conclusions at conclusions the beginning the the pharmaceutical the CVZ/ZIN dailywas the outcomes outcomes the the the use of evidence set number of drugsset number answers before, or before, answers a patient.” – ZS1 – patient.” the cost-effectiveness of drugscost-effectiveness ACP could have stayed its stayed have could ACP the the the The The question: what can we do can we what question: Appraisal Committee (ACP) had (ACP) Committee Appraisal rapid pace, particularly pace, rapid in a collection of new information oncollection of new information the the decisions being made.” – ZS1 made.” decisions being question whether question the summer of 2012, summer of 2012, analysis of observational of analysis decreasing data, the the second-line drug would be included drug would second-line the the a standard of care changed several times several changed of care standard first T=4 dossiers. initial decision to remove remove to initial decision decision were judged to be too large.” – PE1 large.” be too judged to decision were information available at T=0. Even after conduct ofT=0. Even at available information the drug would never be cost-effective. These warningsThese be cost-effective. never drug would non-robust evidence generated by by generated evidence non-robust evening news in association with drugs for Pompe’s Pompe’s for drugs with in association news evening outcomes research studies in pharmacoeconomic in pharmacoeconomic studies research outcomes permanent solution by by solution permanent case that case that CF procedure, complicating complicating CF procedure, uncertainties relating to to uncertainties relating a the the the the the the a result, the result, a the year later.” – PE3 later.” year studies?” – ZS4 – studies?” outcomes at T=4. People said too easily at at easily said too People T=4. at outcomes national reimbursement package had to be reversed. package be reversed. reimbursement had to national the registry, only to find out that it has been registered for first-linefor registered it has been out that find to only registry, citizen for for citizen fundamental problem resides in resides problem fundamental A “In my opinion, it still remains it still remains opinion, “In my inT=4 at useful information more actually studies delivered research to comparison the “Unfortunately, the from that warned we cases “In many already to T=0 at evidence had enough We useless. was these drugs that conclude “It often was “This when very became apparent for issue advice to Quotes from phase 1 interviews from Quotes the studies, the studies, the of implications were ignored. So in some cases, you could say that that say could you So in some cases, ignored. were into a into a treatment at is happening “Drug development As oncology. the throughout by generated – ZS2 models.” not punishalso could but on reimbursement advice on negative ground the remained large at T=4.” – PE2 T=4.” at large remained “ but temporarily provided is that something “It remove very is to difficult as experienced T=4 were already obvious at T=0.” – ZW2 T=0.” at obvious already T=4 were outcomes these need really do we been wondering: have should “One know not already Did we studies? research the during, industry.” – ZA1 – industry.” “External political factors certainly on influence had an T=4. In issued at eventually with observational, real-world data? There are many methodological many are There data? with observational, real-world with associated challenges trust in mentioned in in mentioned that saying (rightly) by pressure deliver also Patients disease. and Fabry’s accusations and Emotional responses not be denied hope. they should impact definitely front on that that RCTs would not be conducted within CF and that we will rely on will rely we that within CF and not be conducted would RCTs that – ZA3 work.” just doesn’t This our decisions. observational for data Outcomes Outcomes research contributed decision to little making T=4 Impossible remove to from drugs reimbursement package T=4 at Conclusions T=4 at already predictable at T=0 Rapid change in Rapid change in practiceclinical external of Effect factors) (political advice on ZIN’s Theme

- continued - Functioning of CF – – of CF Functioning Decision-making aspects – of CF Functioning Decision-making aspects – of CF Functioning Decision-making aspects Functioning of CF – – of CF Functioning Methodological aspects – CF of Functioning Decision-making aspects Topic Table 4 treatments they treatments the discussions around CF. That That CF. discussions around procedure. We could see see could We procedure. table. We should also have should also have We table. the the the knowledge to design future policy future knowledgedesign to result of result a the establishment of start- and stop criteria for of start-establishment criteria for and stop costs associated with with associated costs the evidence generated on drugs through CF may have have CF may on drugs through generated evidence the the facts, the CF experience is to have have is to CF experience the right research questions and parameters list, valuable information could have have could information valuable list, and parameters questions research right process.” – PI2 process.” the the treatment in practice. This should be done for all drugs, not only those in CF.” – PI1 those in CF.” only not all drugs, done for should be This practice. in treatment Conditional Coverage scheme [other managed entry agreement currently implemented implemented currently entry [other managed scheme agreement Coverage Conditional increased collaboration is remarkable.” – MS3 remarkable.” is collaboration increased healthcare arena started collaborating more as started more arena collaborating healthcare the work floor began to think more about to think more began floor work the the the the Netherlands*, ed.].” – HI1 Netherlands*, ed.].” scope, organization and financing of outcomes research studies and registries would best be conductedon best be would registries and studies research of outcomes and financing organization scope, advantage of having gone through gone through having of advantage lesson learned is to do this together, rather than sitting on opposite sides of on opposite than sitting rather do this together, lesson learned is to the European level. With regards to financing both, specifically, industry return, an important In role. play should both, specifically, financing to regards With level. European The The The The A Quotes from phase 2 interviews from Quotes been generated about been generated “ “Even doctors on “Even with seen that have to good “It’s “ in stakeholders “All “ a – HO2 these products.” improve to enable them on their products to information valuable receive they would instruments, such as such instruments, is an important gain, even for ZIN. If you look at at look ZIN. Ifis an important you for gain, even dialogues at an earlier of stage dialogues at of all steps into incorporated being organizations patient see to like would “I in been mediocre but been mediocre prescribe and appropriate care, due to their heightened awareness of these topics.” – MS2 these topics.” of awareness their heightened to due care, appropriate and prescribe with CF that important aspects were unknown to us, such as such as important CF that with unknown us, to aspects were – PO3 treatment.”

factors sanctions the the correct setscorrect new line national subsidiary, subsidiary, a discussions the a lessons learnt the the the payers, they’re best they’re payers, ones to impose sanctions impose ones to current incentives of incentives current the more collaborative collaborative more a early stages of HTA (e.g. (e.g. early of HTA stages the the system for sanctions; withoutfor system scheme.” – ZS3 scheme.” the a debate on costs and cost- on costs debate operationalisation of cost- operationalisation CF drugs out of out CF drugs the the the future. Because of CF, Because of CF, future. the the modern T=0’s, where modern T=0’s, effort industry) (including stakeholders CF scheme (e.g. with eculizumab and with eculizumab CF scheme (e.g. better alternative would be would alternative better coming of these new drugs displaced other new drugs displaced of these coming the drug. Otherwise,drug. the A demands of demands the agreement on growth in health costs not in health costs growth on agreement the the the the societal awareness of this displacement and its of this displacement societal awareness indication field in question and you arrive at you arrive in question and field indication later stage.” – FG1 stage.” later the The The a package which could be more cost-effective and deliver deliver and package cost-effective be more could which positive effects of CF is of effects positive re-assessment of drugs for Pompe’s and Fabry’s disease, disease, Fabry’s and Pompe’s re-assessment for drugs of the the the involved medical societies to develop start- develop to societies and stop- medical criteria for involved future, medical societies, ZIN and healthcare insurers should discuss should insurers ZIN and healthcare societies, medical future, HTA process has changed. It has changed. become has process HTA best strategy with all stakeholders.” – FG2 with all stakeholders.” best strategy the of thought has been set regarding been set regarding has of thought As conditions. based on these needed) (where in responsibilities take up our (new) should all We do so. to positioned – ZS2 schemes.” future “Partly due to learnings from CF, ZIN’s working methods have shifted working methods have ZIN’s CF, learnings from to due “Partly on with other stakeholders collaboration more to one-sidedfrom HTA Looking to quality and of healthcare. care appropriate issues such as the In my together. schemes reimbursement conditional for conditions also be should insurers healthcare opinion, without such as CF schemes implement not should “One “If anything, we may have probably learned how we shouldn’t do shouldn’t we learned how probably have may we “If anything, in managed entry agreements “Since “Evidence generated through outcomes research studies allowed some studies allowed research outcomes through generated “Evidence of – FG1 administration.” treatment “One of “One effectiveness. Due to Due effectiveness. Quotes from phase 1 interviews from Quotes on drugs that came after on drugs that pertuzumab). These were effectiveness in decision making. Examples relate to relate in decision making. Examples effectiveness of checks-and-balances. There should be should be There of checks-and-balances. that, schemes such as these will not yield benefits. Moreover, Moreover, yield benefits. will not as these schemes such that, the be proportionalshould to exceeding 1% to 1.5%, 1.5%, to 1% exceeding drugs in from CF have been applied. ” – ZS3 ” been applied. CF have from process, including collaboration even in even collaboration including process, should invest to meet to invest should more to society. society. to more – ZA2 CF.” by boosted was consequences “Most importantly, do not finance do not finance “Most importantly, through scoping meetings). In that way, one knows more of of one knows more meetings). In way, scoping that through reimbursement package. reimbursement temporary funds structure. That way, stakeholders including medical stakeholders way, That temporary structure. funds re-assessment that could be aware also would and patients specialists at access affect the really impactingreally “Consider making patient inclusion in outcomes research is obligatoryresearch making in outcomes inclusion patient “Consider to access for in return stakeholders can once again lead to underpowered, low-quality underpowered, to again lead can once stakeholders studies.” – FG2 studies.” Consensus on on Consensus aims scheme and importance of collaboration Underlying and incentives accountability Policy Policy experience gained Increased for awareness collaboration Societal debate Societal debate on cost- effectiveness and of displacement healthcare real- Valuable evidence world generated Theme

- continued -

Conclusions & Future & Future Conclusions - Perspectives of CF Improvement Perspectives - - Perspectives of CF Improvement Conclusions & & Conclusions Future Impact of CF scheme – – CF scheme effects Positive – CF scheme effects Positive Impact of Impact of – CF scheme effects Positive Impact of – CF scheme effects Positive Topic Table 4

Appendices

274 11 Appendices 275 11 treatments they treatments the discussions around CF. That That CF. discussions around procedure. We could see see could We procedure. table. We should also have should also have We table. the the the knowledge to design future policy future knowledgedesign to result of result a the establishment of start- and stop criteria for of start-establishment criteria for and stop costs associated with with associated costs the evidence generated on drugs through CF may have have CF may on drugs through generated evidence the the facts, the facts, the CF experience is to have have is to CF experience the right research questions and parameters list, valuable information could have have could information valuable list, and parameters questions research right process.” – PI2 process.” the the treatment in practice. This should be done for all drugs, not only those in CF.” – PI1 those in CF.” only not all drugs, done for should be This practice. in treatment Conditional Coverage scheme [other managed entry agreement currently implemented implemented currently entry [other managed scheme agreement Coverage Conditional increased collaboration is remarkable.” – MS3 remarkable.” is collaboration increased healthcare arena started collaborating more as more started arena collaborating healthcare the work floor began to think more about to think more began floor work the the the the Netherlands*, ed.].” – HI1 Netherlands*, ed.].” scope, organization and financing of outcomes research studies and registries would best be conducted on best be would registries and studies research of outcomes and financing organization scope, advantage of having gone through gone through having of advantage lesson learned is to do this together, rather than sitting on opposite sides of on opposite than sitting rather do this together, lesson learned is to European level. With regards to financing both, specifically, industry return, an important In role. play should both, specifically, financing to regards With level. European The The The The A Quotes from phase 2 interviews from Quotes they would receive valuable information on their products to enable them to improve these products.” – HO2 these products.” improve to enable them on their products to information valuable receive they would “ a “ “All stakeholders in stakeholders “All “ “It’s good to have seen that with seen that have to good “It’s about been generated as such instruments, “Even doctors on “Even is an important gain, even for ZIN. If you look at at look ZIN. If is an important you for gain, even but been mediocre dialogues at an earlier of stage dialogues at of all steps into incorporated being organizations patient see to like would “I in the in with CF that important aspects were unknown to us, such as such as important CF that with unknown us, to aspects were – PO3 treatment.” prescribe and appropriate care, due to their heightened awareness of these topics.” – MS2 topics.” of these awareness their heightened to due care, appropriate and prescribe

the factorsthe the sanctionsthe correct setscorrect new line a the a subsidiary, subsidiary, a lessons learnt the national the the discussions the the payers, they’re best they’re payers, ones to impose sanctions impose ones to current incentives of incentives current the more collaborative collaborative more a early stages of HTA (e.g. (e.g. early of HTA stages the the system for sanctions; withoutfor system scheme.” – ZS3 scheme.” the a debate on costs and cost- on costs debate operationalisation of cost- operationalisation CF drugs out of out CF drugs the the the future. Because of CF, Because of CF, future. the the modern T=0’s, where modern T=0’s, effort industry) (including stakeholders CF scheme (e.g. with eculizumab and with eculizumab CF scheme (e.g. better alternative would be would alternative better coming of these new drugs displaced other new drugs displaced of these coming the drug. Otherwise,drug. the A demands of demands the agreement on growth in health costs not in health costs growth on agreement the the the the societal awareness of this displacement and its of this displacement societal awareness indication field in question and you arrive at you arrive in question and field indication later stage.” – FG1 stage.” later the The The a package which could be more cost-effective and deliver deliver and package cost-effective be more could which positive effects of CF is of effects positive re-assessment of drugs for Pompe’s and Fabry’s disease, disease, Fabry’s and Pompe’s re-assessment for drugs of the the involved medical societies to develop start- develop to societies and stop- medical criteria for involved HTA process has changed. It has changed. become has process HTA discuss should insurers ZIN and healthcare societies, medical future, best strategy with all stakeholders.” – FG2 with all stakeholders.” best strategy “Evidence generated through outcomes research studies allowed some studies allowed research outcomes through generated “Evidence the of – FG1 administration.” treatment “Since the “Since the shifted working methods have ZIN’s CF, learnings from to due “Partly on with other stakeholders collaboration more to one-sidedfrom HTA Looking to quality and of healthcare. care appropriate issues such as the In my together. schemes reimbursement conditional for conditions also be should insurers healthcare opinion, As conditions. based on these needed) (where in take up our (new) responsibilities should all We do so. to positioned – ZS2 schemes.” future without such as CF schemes implement not should “One “One of “One to Due effectiveness. do shouldn’t we learned how probably have may we “If anything, in managed entry agreements been set regarding has of thought to relate in decision making. Examples effectiveness came after on drugs that pertuzumab). These were Quotes from phase 1 interviews from Quotes process, including collaboration even in even collaboration including process, should be There of checks-and-balances. Moreover, yield benefits. will not as these schemes such that, be proportionalshould to including medical stakeholders way, That temporary structure. funds re-assessment that could be aware also would and patients specialists at access affect exceeding 1% to 1.5%, 1.5%, to 1% exceeding drugs in – ZS3 ” been applied. CF have from “Consider making patient inclusion in outcomes research is obligatory research making in outcomes inclusion patient “Consider to access for in return through scoping meetings). In that way, one knows more of of one knows more meetings). In way, scoping that through impactingreally the meet to invest should do not finance “Most importantly, package. reimbursement more to society. society. to more – ZA2 CF.” by boosted was consequences stakeholders can once again lead to underpowered, low-quality underpowered, to again lead can once stakeholders – FG2 studies.” Valuable real- Valuable evidence world generated Increased Increased for awareness collaboration on Consensus aims scheme and importance of collaboration Underlying and incentives accountability Societal debate Societal debate on cost- effectiveness and of displacement healthcare Policy experience gained Theme

- continued -

Impact of – CF scheme effects Positive Impact of – CF scheme effects Positive & Conclusions Future - Perspectives of CF Improvement & Future Conclusions - Perspectives of CF Improvement Impact of – CF scheme effects Positive Impact of – CF scheme effects Positive Topic Table 4 tap at the rate of drainage, whereas we could better shut better could we whereas of drainage, rate the NHS, we should define clear threshold values and say no if new drugs exceed these exceed no if new drugs values and say define clear threshold should NHS, we the United Kingdom United by prices for these new drugs are too high. We should be quite strict regarding cost-effectiveness at T=0. at cost-effectiveness strict quite be should regarding We high. too new drugs are these for prices beginning.” – MS2 beginning.” the The The Quotes from phase 2 interviews from Quotes “There should be harder rules with regards to acceptable and non-acceptable cost-effectiveness. As usually doneAs cost-effectiveness. and non-acceptable acceptable to “There regards rules with be harder should in – MS1 is established.” after system this later, come use) Other appropriate values. aspects (e.g. “ trying lessen is with CF do now to we’re What the outcomes outcomes the effectiveness and effectiveness the good basis for future future for good basis a outcomes research studies.” – FG1 studies.” research outcomes new drug. Based upon that, new drug. the the mandate to do so, then it should do so, to mandate studies whereby monitoring results results monitoring studies whereby the the question is: how and who should do that? and who should how is: question usefulness of re-assessmentusefulness of at of cost-effectiveness progress made in progress pre-defined when ZINpoints, time or set of time point, the The The a the current form of CF served of CF form as current cost-effectiveness of cost-effectiveness authority to stop authority stop to the the the Additionally, if ZIN would have have ZIN would if Additionally, also have – ZS3 progress.” little indicate “There should be monitor would “In my opinion, there should be continuous monitoring of monitoring continuous be should there opinion, “In my studies. research Quotes from phase 1 interviews from Quotes “We can say that at initial assessment (T=0), initial assessment at that can say “We known already much is regarding agreements can be made on other aspects such as: appropriate use, use, aspects on other be made can appropriate such as: agreements We restrictionreimbursement. or of agreements financial-based otherwise doubt T=4.” – PE2 T=4.” “Although efforts, we believe that we can say much more on much more say we can that believe we efforts, cost-effectiveness of drugs at T=0.” – PE3 T=0.” at of drugs cost-effectiveness Monitoring and procedures reviews mid-term Theme Definitive onconclusions cost-effectiveness T=0 at - continued - Abbreviations: CF: Conditional Financing; HTA: Health Technology Assessment; VWS: Ministerie voor Volksgezondheid, Welzijn en Sport; ZIN: Zorginstituut Nederland. en Sport; Zorginstituut ZIN: Welzijn Volksgezondheid, VWS: Ministerie voor Assessment; Technology Health HTA: Financing; Conditional CF: Abbreviations: Conclusions & Conclusions - Perspectives Future CF of Improvement Topic Conclusions & Conclusions - Perspectives Future CF of Improvement Table 4

Appendices

276 11 Appendices 277 11 tap at the rate of drainage, whereas we could better shut better could we whereas of drainage, rate the NHS, we should define clear threshold values and say no if new drugs exceed these exceed no if new drugs values and say define clear threshold should NHS, we the United Kingdom United by prices for these new drugs are too high. We should be quite strict regarding cost-effectiveness at T=0. at cost-effectiveness strict quite be should regarding We high. too new drugs are these for prices beginning.” – MS2 beginning.” The The Quotes from phase 2 interviews from Quotes in the in “There should be harder rules with regards to acceptable and non-acceptable cost-effectiveness. As usually doneAs cost-effectiveness. and non-acceptable acceptable to “There regards rules with be harder should “ trying lessen is with CF do now to we’re What values. Other aspects (e.g. appropriate use) come later, after this system is established.” – MS1 is established.” after system this later, come use) Other appropriate values. aspects (e.g. the the outcomes outcomes the effectiveness and effectiveness the good basis for future future for good basis a outcomes research studies.” – FG1 studies.” research outcomes new drug. Based upon that, new drug. the the mandate to do so, then it should do so, to mandate studies whereby monitoring results results monitoring studies whereby the the question is: how and who should do that? and who should how is: question usefulness of re-assessmentusefulness of at of cost-effectiveness progress made in progress pre-defined ZIN when points, time or set of time point, the The The a the current form of CF served of CF form as current cost-effectiveness of cost-effectiveness authority to stop authority stop to the “In my opinion, there should be continuous monitoring of monitoring continuous be should there opinion, “In my studies. research have ZIN would if Additionally, also have – ZS3 progress.” little indicate “There should be monitor would Quotes from phase 1 interviews from Quotes “We can say that at initial assessment (T=0), initial assessment at that can say “We known already much is the regarding agreements can be made on other aspects such as: appropriate use, use, aspects on other be made can appropriate such as: agreements We restrictionreimbursement. or of agreements financial-based otherwise doubt T=4.” – PE2 T=4.” the “Although efforts, we believe that we can say much more on much more say we can that believe we efforts, cost-effectiveness of drugs at T=0.” – PE3 T=0.” at of drugs cost-effectiveness Monitoring and procedures reviews mid-term Theme Definitive onconclusions cost-effectiveness T=0 at - continued - Abbreviations: CF: Conditional Financing; HTA: Health Technology Assessment; VWS: Ministerie voor Volksgezondheid, Welzijn en Sport; ZIN: Zorginstituut Nederland. en Sport; Zorginstituut ZIN: Welzijn Volksgezondheid, VWS: Ministerie voor Assessment; Technology Health HTA: Financing; Conditional CF: Abbreviations: Conclusions & Conclusions - Perspectives Future CF of Improvement Topic Conclusions & Conclusions - Perspectives Future CF of Improvement Table 4 health health The The . FG2 “back-door” to to “back-door” cost-effectiveness cost-effectiveness a scientifically scientifically reimbursement reimbursement a treatment of severe of severe treatment the the the drugs would not be not be would drugs the ; an important; an example scheme as scheme conduct of outcomes conduct of outcomes the ZS2,ZS4 absence of such legislation, it it of such legislation, absence the . the legal perspective. legal impact of ZIN advices at T=4. T=4. impact at of ZIN advices PO2 a the case of omalizumab for for omalizumab case of EuroQol 5-dimensional EuroQol (EQ-5D) scale the the . In MS2 final studies implemented final studies case of treatments for metastatic melanoma, CF led to to CF led melanoma, metastatic for treatments case of . the ZW2 words of one stakeholder “Once they [drugs] were in, there was was there in, were they [drugs] “Once one stakeholder of words the reimbursement package. In reimbursement reimbursement package[17]. However, there are no equivalent equivalent no are there package[17].reimbursement However, the pay-for-performance scheme enticed adherence to strictto start- adherence and enticed scheme pay-for-performance the the the Netherlands may have diminished diminished have Netherlands may reimbursement package and an excuse to postpone difficult decisions. In their their In decisions. postpone difficult to packagereimbursement excuse and an outcome research study proposals by public bodies at T=0 was often not often not was T=0 bodies at public by proposals study research outcome relates to recommendations to include Health-Related Quality of Life (HRQoL) (HRQoL) Quality Health-Related include Life of to recommendations to relates collection using outcomes 4/14 (29%) of phase 1 stakeholders emphasized how national legislation in in legislation national how emphasized stakeholders 1 phase (29%) of 4/14 the removed. In removed. no turning back” the uncertainties higher on relatively and costs higher relatively drugs with opinion, conditions T=0 under at admitted were and cost-effectiveness use appropriate were decisions difficult Inso, doing drugs. conventional for than scrutiny of less that apparent it became time, ByT=4. that shifted to 5/14 (36%) of phase 1 stakeholders experienced experienced stakeholders phase 1 (36%) of 5/14 asthma, asthma, criteria practice in clinical stop-treatment Theme (%) Theme on provided feedback that iterated stakeholders phase 1 of (29%) 4/14 the into incorporated insurance law (“Zorgverzekeringswet”) states that drugs with drugs that states (“Zorgverzekeringswet”) law insurance proven added therapeutic value (“stand van de wetenschap en praktijk”) en should de wetenschap van (“stand value added therapeutic proven be part of clauses in current legislation on requirements pertaining to on requirements legislation in current clauses of drugs in of drugs research studies in itself stimulated appropriate use of drugs in clinical practice. practice. in clinical of drugs use appropriate itself stimulated studies in research in example, For awareness deliveryexpertise increased and of healthcare to centers centralization use in practice on drug package after re-assessment at T=4 from T=4 from package afterre-assessment at 4/16 (25%) of phase 2 stakeholders stated that that stated stakeholders phase 2 (25%) of 4/16 became impossible to remove cost-ineffective drugs from from drugs cost-ineffective remove to impossible became – Unique themes identified throughcomparative analysis of phase 1 and phase 2 studies. Abbreviations: CF: Conditional Financing; ZIN: Zorginstituut Nederland. ZIN: Zorginstituut Financing; Conditional CF: Abbreviations: Decision-making aspects Negative effects effects Negative of CF Topic Methodological Methodological aspects Positive effects effects Positive of CF Table 5

Appendices

278 11 Appendices 279 11

– Standardized e-mail invitation for stakeholder representatives. stakeholder e-mailfor – Standardized invitation en weesgeneesmiddelen de totstandkoming van van perceptie Uw van financiering voorwaardelijke de also onderdeel de beleidsregels ervaringenUw van de beleidsregels financiering voorwaardelijke met onderdeel als financiering voorwaardelijke voor scenario’s toekomst Enkele Uw rol in voorwaardelijke financiering als onderdeel van de beleidsregel dure dure beleidsregel van de financiering in voorwaardelijke onderdeel als rol Uw • • • • ij, Amr Makady (farmacoeconomische beoordelaar) en Hugo Neijmeijer (masterstuden Radboud Makady (masterstuden Neijmeijer Hugo en Amr MC beoordelaar) ij, (farmacoeconomische Gezien alle recente nationale en internationale discussies over flexibele vergoedingssystemen is vanuit vanuit is vergoedingssystemen flexibele over discussies en internationale nationale Gezien recente alle In hoofdlijnen willen we de volgende zaken bespreken: de volgende we willen In hoofdlijnen Nijmegen)/Sandrine van Acker (masterstudent VU Amsterdam), werken momenteel aan deze evaluatie evaluatie aan deze momenteel werken Amsterdam), VU (masterstudent Nijmegen)/Sandrine Acker van manier betrokken u op een mail omdat bent Goettsch. deze U ontvangt Wim van onder begeleiding stellen vragen wat wij u graag willen reden Om deze beleidsregels. deze de uitvoering bij van geweest ervaringen uw bijlage de over financiering voorwaardelijke met in het beleidsregels. In van deze kader voorwaardelijke van procedure de volledige uiteindelijk die met medicijnen u een tabel vinden kunt financiering zijn ondergaan. W

Geachte [naam interviewee], Geachte Zorginstituut Nederland het plan opgevat om het gebruik van voorwaardelijke financiering, dat een een dat financiering, voorwaardelijke van gebruik om het Nederland het plan opgevat Zorginstituut de is Het evalueren. te (2006-2012), en weesgeneesmiddelen dure de beleidsregels van was onderdeel toekomstige van aan de inrichting bijdragen zullen evaluatie deze van uitkomsten de dat bedoeling dit onderzoek ook van zullen resultaten De vergoeding/financiering. voorwaardelijke van vormen een voor gebruikt worden Utrecht de Universiteit aan lopend promotietraject reeds het omtrent de effectiviteit van beoordeling de (RWD)) data voor observationele van (real-world gebruik gegevens geneesmiddelen. van Metvriendelijke groet, 1 Figure Uw bijdrage aan dit interview is van grote toegevoegde waarde. Wij willen spoedig een datum en en een datum willen spoedig Wij waarde. toegevoegde dit interview aan bijdrage Uw grote is van deelnemen. wilt of u hieraan graag interview het voor en vernemen daarom vaststellen plaats [naam onderzoeker] Wat is /was uw rol in het kader uw rol financiering? voorwaardelijke is /was van Wat het kader in financiering? voorwaardelijke uw organisatie van van de rol is/was Wat kader in het financiering? voorwaardelijke uw organisatie u of van van belang is het Wat Wat was, vanuit uw perspectief of het perspectief van uw organisatie, het oorspronkelijke doel doel het oorspronkelijke uw perspectief vanuit of het perspectief uw organisatie, was, van Wat financiering?voorwaardelijke van als dit kader verwezenlijken doel te om het zelfde gedaan pogingen zijn er eerdere In hoeverre financiering?voorwaardelijke van van komen stand bij het tot betrokken geweest u/ is uw organisatie bent mate In welke financiering?voorwaardelijke financiering voorwaardelijke moest waaraan criteria gesteld concrete zijn er In hoeverre kader het kunnen later Dit om evalueren. te voldoen? Hoe denkt u over het functioneren van voorwaardelijke financiering? (evt. toelichten dat maar maar dat toelichten financiering? voorwaardelijke (evt. het functioneren van u over Hoe denkt vragen) daarover en mening doorgekomen zijn procedure volledig 47 de 12 van Hoe denkt u over de afgesproken tijdsperiode van 4 jaar voor kandidaten en het overschrijden kandidaten het overschrijden en voor 4 jaar tijdsperiode van de afgesproken u over denkt Hoe periode?die van In welke mate is er op T=0 sprake van goed geplande uitkomstenonderzoeken met duidelijke met duidelijke uitkomstenonderzoeken geplande goed van T=0 sprake is er op mate In welke geweest? en kosteneffectiviteit gebruik gepast voor doelstellingen aan goed die uitkomstenonderzoeken uitgevoerde van T=4 sprake is er op mate In welke aansluiten/aansloten? en kosteneffectiviteit gebruik gepast voor doelstellingen studies? de uitgevoerde kwaliteit de van over kunnen mening u uw geven Zou die middels uitkomstenonderzoek data opgeleverd van T=4 sprake is er op mate In welke doelstelling? op primaire antwoorden relevante en betrouwbaarheid data opgeleverde kwaliteit de van over kunnen uw mening u geven Zou resultaten? de van • • • • • • • • • • • • • • Rol en belang: Het is de bedoeling dat de uitkomsten van deze evaluatie zullen bijdragen aan de inrichting van van inrichting aan de bijdragen zullen evaluatie deze van uitkomsten de dat Het is de bedoeling onderzoek van dit resultaten De vergoeding/financiering. voorwaardelijke van vormen toekomstige Utrecht omtrent Universiteit aan de promotietraject lopend een reeds voor gebruikt ook worden zullen de van de beoordeling (RWD)) data voor observationele (real-world van gebruik het gegevens geneesmiddelen. van effectiviteit Inleiding: vanuit is vergoedingssystemen flexibele over discussies internationale en nationale Gezien recente alle een dat financiering, voorwaardelijke van gebruik om het Nederland het plan opgevat Zorginstituut kader Dit evalueren. te (2006-2012), weesgeneesmiddelen en dure beleidsregels de van was onderdeel kan dynamiek.t=0,t=4 de zijn als bekend misschien beter Tot stand komen van voorwaardelijke financiering (aim): voorwaardelijke van stand komen Tot De huidige situatie van voorwaardelijke financiering (functioning): voorwaardelijke van De huidige situatie Procedurele aspecten: Procedurele Methodologische aspecten:Methodologische

Appendices

280 11 Appendices 281 11 2 centra, duidelijke duidelijke 2 centra, a conditional approval based based approval conditional a product, following following product, restricted patient population then expanding to to then expanding population patient restricted a a benefit-risk balance of balance benefit-risk the on early data (using surrogate endpoints) considered predictive of importantof predictive clinical considered endpoints) surrogate (using earlyon data outcomes; confirming confirming approval in stages, beginning with beginning in stages, approval wider patient populations; patient wider medicine’s development. medicine’s – Interview guide. Ű Ű Ű Ű 1. iterative development, which either means: which either development, 1. iterative gathering evidence through real-life use to supplement clinical trial clinical data; supplement to use real-life through evidence gathering on bodies in discussions health-technology-assessment and of patients early involvement a In hoeverre denkt u/uw organisatie dat resultaten uit uitkomstenonderzoeken invloed hebben hebben invloed uit uitkomstenonderzoeken resultaten dat organisatie denkt u/uw In hoeverre toelichten? u uw antwoord T=4? Kunt op advies het uiteindelijke op gehad T=4) invloed op evidentie externevan (los dat factoren denkt u/uw organisatie In hoeverre toelichten? uw antwoord u Kunt T=4? advies op het uiteindelijke op gehad hebben In welke mate denkt u/uw organisatie dat voorwaardelijke financiering zijn gewenste doel doel financiering voorwaardelijke zijn gewenste dat organisatie u/uw denkt mate In welke toelichten? antwoord u uw Kunt heeft bereikt? voorwaardelijke van effecten of positieve enkele voordelen u/uw organisatie volgens zijn Wat toelichten? u uw antwoord Kunt financiering? functionerende aspecten matig of nadelen de van u/uw organisatie zijn volgens Wat toelichten? antwoord u uw Kunt financiering?voorwaardelijke Wat zijn volgens u de belangrijkste punten voor verbetering van voorwaardelijke financiering? voorwaardelijke van verbetering voor punten u de belangrijkste zijn volgens Wat beleid? toekomstig voor worden moeten die meegenomen punten u concrete zijn volgens Wat Het VF kaderVF stoppen. Het is. het nu zoals kaderVF continueren, Het leerpunten. kaderVF geïdentificeerde van aanleiding naar aanpassen Het (Buiten “adaptivepathways”[1]. Nieuw van kader met uitgangspunten oprichten 1 participatie patiënten, plicht fonds, afgebakend vergoedingssysteem, gebruik. effectiviteit/kosteneffectiviteit/gepast mbt doelstelling u uw antwoord Kunt toekomst? financiering voorwaardelijke in de voor rol u een duidelijke Ziet toelichten? • • • • • • • • • • • • • • • [1]Bron:http://www.ema.europa.eu/ema/index.jsp?curl=pages/regulation/general/general_ content_000601.jsp principles: three on based is pathways Adaptive Figure 2 Figure Het beoordelen van data en besluitvorming op T=4: besluitvorming en op data van beoordelen Het Uw opvatting over het kader and disadvantages): (advantages over opvatting Uw Toekomst voorwaardelijke financiering (future perspectives): financiering (future voorwaardelijke Toekomst Aantal scenario’s voor VF in de toekomst VF in de voor scenario’s Aantal

full coding tree developed for content analysis (part analysis 1 of 4) content for developed tree coding full The The –

Figure Figure 3a – The full coding tree developed for content analysis (part 1 of 4) Figure 3a Figure

Appendices

282 11 Appendices 283 11

full coding tree developed for content analysis (part analysis 2 of 4) content for developed tree coding full The The –

Figure Figure 3b – The full coding tree developed for content analysis (part 2 of 4) Figure 3b Figure

full coding tree developed for content analysis (part analysis 3 of 4) content for developed tree coding full The The –

Figure Figure 3c – The full coding tree developed for content analysis (part 3 of 4) Figure 3c Figure

Appendices

284 11 Appendices 285 11

full coding tree developed for content analysis (part analysis 4 of 4) content for developed tree coding full The The –

Figure Figure 3d – The full coding tree developed for content analysis (part 4 of 4) Figure 3d Figure (n=16)

(n=16) 2

0,00% 6,25% 12,50% 50,00% 31,25% 0,00% 43,75% 25,00% 31,25% 12,50% Phase aims.

CF. its

of (n=14) Phase

0,00% 4,55% (n=14) 47,62% 23,81% 22,73%

future achieved 0,00% 0,00% 5,56% 61,11% 33,33%

Phase CF

the

on (n=30) Phase whether 0,00% 5,41% 32,43% 35,14% 27,03% All views on

(n=30)

8,82% 0,00% 52,94% 14,71% 29,41% All 0% views 90% 80% 70% 60% 50% 40% 30% 20% 10%

(other) (AP) 100%

CF form

0% policy policy 90% 80% 70% 60% 50% 40% 30% 20% 10% 100% new new

generation)

current

Stakeholders in access)

AP: Adaptive Pathways; CF: Conditional Financing; MEA: Managed MEA: Entry Agreement. Financing; Conditional CF: Pathways; Adaptive AP: with with reintroduce

CF MEAs Stakeholder &

CF CF

(other) (evidence (early

CF/ – Comparative analysis of stakeholder views on future perspectives. on future views of stakeholder analysis – Comparative – Comparative analysis of stakeholder views on achievement of conditional financing (CF) aims. (CF) financing conditional of achievement on views of stakeholder analysis – Comparative

Stop Replace Replace Improve Continue

Partially Partially Partially No Yes Percentage Percentage Figure 4 Figure Figure 5 Figure Abbreviations:

Appendices

286 11 Appendices 287 11 Does not me apply to 41 30 19 18 26 47 30 25 29 26 24 30 6 11 16 18 20 20 11 21 29 30 12 11 21 29 30 19 11 26 33 30 19 21 28 Very relevant - - 6 11 21 6 - - 7 21 18 - 25 32 26 35 - 7 26 26 12 - 31 18 32 24 - 38 18 21 17 10 12 7 22 Relevant 6 10 31 14 11 - - 25 4 11 18 10 31 25 21 18 10 53 33 32 12 - 25 29 11 6 - 6 29 21 - - 19 14 17 Relevance Neutral 12 - 6 11 11 12 - - 4 - 6 - 12 7 26 12 10 - 7 11 6 20 - 18 21 6 - 12 18 21 - - 6 14 - Not Not relevant 24 10 12 21 21 18 - 6 7 16 18 - 6 7 - 12 10 7 7 5 24 10 12 7 5 24 30 6 7 5 22 20 19 25 17 Not relevant all at 18 50 25 25 11 18 70 44 50 26 18 60 19 18 11 6 50 13 15 5 18 40 19 18 11 12 40 19 18 5 28 40 25 18 17 Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=15) III Stage (n=27) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=18) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers 9 - APPENDIX - 9 day toilet chair chair short the a a the house long walk long Question in Question in QLQ-C30: EORTC bed or bed during Need help with help with Need dressing, eating, washing or yourself using Limitations in in Limitations either doing or work your daily other activities Limitations in in Limitations your pursuing or hobbies leisure other time activities Trouble doing doing Trouble strenuous activities Trouble taking Trouble a Trouble Trouble taking outside walk the Need to stay in in stay to Need Appendix 1 - Relevance of questions from EORTC QLQ-C30 questionnaire in our study population CHAPTER CHAPTER 29 21 Does not me apply to 29 30 - 11 21 31 30 6 11 16 35 30 19 22 26 47 30 19 35 40 19 18 33 38 30 6 18 11 25 30 7 7 16 Very relevant 18 10 12 36 32 - - 12 22 37 - - 12 4 21 - - 25 11 16 12 - 19 7 22 6 - 19 11 50 25 10 20 32 21 Relevant 18 30 38 25 32 19 10 44 15 26 - - 19 19 11 6 10 6 11 11 6 - 12 29 - 19 10 38 21 11 12 20 53 29 26 Relevance Neutral 24 - 25 7 11 25 10 19 7 5 18 20 6 15 11 6 - 6 7 5 - 10 12 - 11 12 - 6 11 11 12 10 7 7 5 Not Not relevant 6 10 12 - 5 19 10 6 7 5 29 - 19 7 5 18 - 19 11 26 29 10 6 4 11 12 10 6 4 - 19 10 7 4 16 Not relevant all at 21 - 6 40 12 37 11 18 50 25 33 26 24 60 25 32 21 18 40 31 43 22 12 50 25 36 17 6 20 7 21 16 6 20 12 Stage I (n=16) I Stage (n=10) II Stage (n=16) III Stage (n=27) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=27) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=16) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=16) I Stage (n=10) II Stage (n=15) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Question in Question in QLQ-C30: EORTC Feeling weak Feeling Lack of appetite Lack Nausea/ Feeling Feeling Nausea/ sick Short of breath Pain Needed more more Needed rest time to Trouble Trouble sleeping Appendixcontinued 1 -

Appendices

288 11 Appendices 289 11 12 20 12 4 11 Does not me apply to 19 20 12 - 16 35 30 19 18 32 29 22 13 7 26 53 30 25 25 37 50 30 19 29 42 50 30 19 14 26 47 Very relevant 12 10 25 54 42 6 - 12 21 21 12 22 20 32 11 29 30 25 36 - - 6 7 - - - 6 11 - - - 6 14 5 Relevant 50 50 38 21 32 29 20 25 7 16 18 22 33 11 26 35 30 25 29 26 - - 19 7 5 6 - 25 18 16 - - 25 18 21 Relevance Neutral - - 12 7 - - - 12 11 11 29 11 13 18 5 12 - 25 18 5 - - - 11 11 6 - 6 11 26 12 - 6 11 5 Not Not relevant 6 - - - - 12 20 6 7 11 6 - - 7 26 6 10 - 4 11 18 - - 7 21 19 10 6 4 5 19 - 6 11 26 Not relevant all at 18 11 18 30 25 36 11 18 22 33 11 5 6 10 12 11 - 29 70 50 43 26 19 60 38 29 11 19 70 38 32 16 12 20 12 Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=9) II Stage (n=15) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=16) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=16) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=16) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Question in Question in QLQ-C30: EORTC Did pain Did pain with interfere daily your activities? Difficulty in Difficulty concentrating on things Feeling tense Feeling Have you you Have vomited? Were you you Were constipated? Diarrhea Tiredness Appendixcontinued 1 - Does not me apply to 35 20 19 14 21 27 30 12 15 21 38 20 12 7 21 25 30 12 14 21 12 20 19 7 16 12 30 7 4 16 29 20 6 18 11 Very relevant 18 20 19 18 42 7 30 12 22 16 25 20 25 32 32 31 20 31 36 32 41 40 44 32 47 25 20 14 18 21 18 30 38 21 42 Relevant 24 20 19 25 11 33 10 19 19 21 19 20 25 32 32 25 30 31 29 32 35 30 31 32 16 38 10 29 50 37 35 20 19 18 12 Relevance Neutral 10 12 25 - 27 - 19 19 11 6 20 25 21 5 6 - 12 18 5 - 12 - - 11 - 19 20 29 11 11 12 20 25 21 5 Not Not relevant - 20 19 7 21 6 - - - 5 6 - - - 5 12 - 12 7 16 - 10 6 11 16 6 10 21 7 16 6 - 12 18 16 Not relevant all at 19 19 11 6 20 12 7 5 6 20 12 4 5 12 30 19 11 11 - - - 7 5 - 10 - 11 - - 10 - 4 5 7 10 Stage I (n=16) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=16) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=18) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=16) I Stage (n=10) II Stage (n=14) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=17) I Stage (n=10) II Stage (n=16) III Stage (n=28) IV Stage (n=19) Carers Stage I (n=15) I Stage (n=10) II Stage (n=16) III Stage (n=27) IV Stage (n=19) Carers Question in Question in QLQ-C30: EORTC Physical Physical condition or medical treatment with interfered life family your Physical Physical condition or medical treatment with interfered social your activities Physical Physical condition or medical treatment you caused financial difficulties Worrying Feeling irritable Feeling Feeling Feeling depressed Difficulty Difficulty remembering things Appendixcontinued 1 -

Appendices

290 11 Appendices 291 11 time to time to Quality of the the Melanoma stage, Melanoma stage, the influence of influence the Dutch National Healthcare Institute. So So Institute. Healthcare National Dutch the aim of this study is therefore to find out what truly matters to to matters truly what out find to therefore aim of this study is country you live in and Melanoma on therapies country live you The The GetReal project and is conducted as collaboration between MPNE, MPNE, between GetReal project as collaboration is conducted and the the approval and reimbursement of new therapies – so please take – so please take of new therapies and reimbursement approval the time of diagnosis, diagnosis, time of Melanoma Patient Network Europe channels! Network Europe Melanoma Patient Melanoma Patient Network Europe, and ZIN, and ZIN, Network Europe, Melanoma Patient Melanoma patients reached through our network. Quality of Life data is also increasingly network. our increasingly also is through data Quality reached Melanoma of Life patients This study is partThis of What is Quality of Life in Melanoma for YOU? YOU? Melanoma in Quality is for What of Life Appendix 2. The Melanoma Survey: QualityThe of Life web-based survey a 25-item 2. Appendix carer, or Dear Melanoma patient Melanoma Patient Network Europe and National Healthcare Institute Healthcare and National Network Europe Melanoma Patient MPNE and ZIN MPNE Thank you for you time and effort. effort. and time you for you Thank used for used for we and confidential are answers Your survey complete. to This should take 20 minutes via Insights reports and shared be obviously will results. anonymous publish will only the the important consider themselves for Melanoma what patients looked at have studies few far, Qualitytheir own of Life. the understand like to would We thoughts! your share the collect such used to be media could see if social to also want We of Melanoma patients. Life can be collaboration about this information More perspectives. on patient information our website. on found

Very important O O 7 – Excellent 7 – O Important O O 6 O Neutral O O 5 O

4 O Not Not important O O Melanoma patient’s Quality of Life right now? right Quality of Life Melanoma patient’s patient’s Quality of Life today. Quality of Life patient’s the the Melanoma patient’s Quality of Life good Quality of Life Melanoma patient’s good Quality of Life Melanoma patient’s 3 O

the the Not Not important all at O O 2 O

O 1 – poor 1 – scale from 1 to 7, please rate your/ 7, please rate 1 to scale from 3 things that today make your/ your/ make today 3 things that your/ make today 3 things that your/ improve would single thing that a The The The The The On 7. How important are for you you important7. How for are 1. [Open Field] 1. [Open Field] 2. [Open Field] 3. [Open Field] 1. [Open Field] 2. [Open Field] 3. [Open energy (e.g. well-being Physical pain) nauseau, level, well-being Social/ Family support and family from (e.g. life) sex friends, 6. 5. 4. 3. 2. What is Quality of Life in Melanoma for you? is Quality in Melanoma for of Life What 2. Field] [Open Quality of Life 1. Quality of Life in Melanoma – which aspects come to your mind? 1. Quality in Melanoma – which aspects your of Life to come Field] 1. [Open Field] 2. [Open Field] 3. [Open Field] 4. [Open Field] 5. [Open Field] 6. [Open Field] 7. [Open Field] 8. [Open Field] 9. [Open Field] [Open 10. [Open Field] [Open We value your opinion – 2 opinion your value We We value your opinion your value We

Appendices

292 11 Appendices 293 11

O O O O O O O O O O O O O Does Does not apply me to O O O O O O O O O O Very relevant O O O O O O O O O O O O O O O O O O O O O O O Relevant O O O O O O O O O O O O O O O O O O O O O O O Neutral O O O O O O O O O O O O O O O O O O O O O O O Not Not relevant O O O O O O O O O O O O O O O O O O O O O O O Field] [Open

Not Not all at relevant O O O O O O O O O O O O O O O O O O O O following aspects for you aspectsfor following chair chair a the long walk long short walk a a house day the toilet the the How relevant are are relevant How heavy shopping bag or heavybag or shopping suitcase newspaper or watching or watching newspaper 9. taking Trouble taking Trouble bed or in stay to Need during eating, help with Need or yourself washing dressing, using either in doing Limitations daily other or work your activities Short of breath Pain rest to time more Needed sleeping Trouble weak Feeling of appetite Lack sick Feeling Nausea/ vomited? you Have 8. Please comment on question 7 comment 8. Please Field] [Open of outside Trouble doing strenuous doing strenuous Trouble activities, carrying like a a Were you constipated? constipated? you Were Diarrhoea Tiredness with your interfere Did pain activities?daily Difficulty concentrating in reading like on things, a television tense Feeling Emotional well-being (e.g. feeling feeling (e.g. well-being Emotional to related nervous,sad or worries Melanoma or treatments) Functional well-being (e.g. ability ability (e.g. well-being Functional life) work,and enjoy to sleep Other below) (please specify Other specify) (please O O O O O O O N/A O O O O O O O O O Stage IV Stage O O O O O O O O O Stage III Stage O O O O O O O O O Stage II Stage O O O O O O O O O Stage I Stage O O [Open Field] [Open O O O O O O Field] [Open O Melanoma patient Melanoma patient a highest level of education you have completed? completed? have you education of level highest Female Male the carer or or carer Melanoma patient a the What is What 16. Your relationship to Melanoma to relationship Your 16. 15. Where did you find this survey? find this survey? did you Where 15. 14. 14. 13. What is your age? age? is your What 13. 12. What is your Country of Residence? Country of Residence? is your What 12. 11. I am O O 10. Please comment on question 8 comment 10. Please Field] [Open I am I am Feeling irritable Feeling depressed Feeling remembering Difficulty things medical or condition Physical with interfered treatment life family your medical or condition Physical with interfered treatment activities social your medical or condition Physcial caused you treatment difficulties financial Other specify) (please Worrying whose disease is in disease is whose Other(please specify) Tell us about yourself about us Tell

Appendices

294 11 Appendices 295 11 [Open Field] [Open patient have? have? patient the patient’s Melanoma have? Melanomahave? patient’s the No Yes No Yes BRAF mutant BRAF wild-type NRASmutant c-kit mutant GNA11 GNAQ/ know I don’t Other(please specify) Field] [Open Melanoma diagnosis was Melanoma diagnosis Which mutations does your/ does your/ mutations Which The The or type of MelanomaWhat do you [Open Field] [Open If yes, what type of chemotherapy? type what of chemotherapy? If yes, 22. Did/ do you have chemotherapy for your Melanoma? Melanoma? your for chemotherapy have 22. Did/ do you No Yes [Open Field] [Open If yes, what type of radiotherapy? type what of radiotherapy? If yes, [Open Field] [Open Melanoma? your for radiotherapy have 21. Did/ do you O O If yes, what type what of surgery? If yes, 20. Did you have surgery for your Melanoma? surgery your for have 20. Did you O O 19. 19. 18. 18. 17. 17. O O O O O O O Other specify) (please Melanoma therapies and treatments Melanoma therapies MPNE MPNE the MPNE network MPNE network the No ZELBORAF® - Roche Vemurafenib/ - Ex-GSK, Novartis now TAFINLAR® Dabrafenib/ - Ex-GSK, MEKINIST® Novartis now Trametinib/ - Roche COTELLIC® Cobimetinib/ Array now – Ex-Novartis, LGX8181 Encorafenib/ Array now Ex-Novartis, 162 – MEK Binimetinib Other(please specify) No - BMS YERVOY® Ipilimumab/ KEYTRUDA®- MSD Pembrolizumab/ - BMS OPDIVO® Nivolumab/ - Amgen IMLYGIC® Laherparepvec/ Talimogene T-Veck/ (CT011 – Curetech anti-PD1) Pidilizumab (anti-PD-L1) – BMS Atezolizumab BMS – (anti-PDL1) BMS936559 – academic Vaccine Dendritic Cell Lymphocytes)academic – (T-infiltrating TILs like Therapies Cell Adaptive Other specify) (please general public. To make sure you don’t miss updates, please sign up to up to sign please miss updates, don’t you sure make To public. general the results of this survey will be shared in any anonymous form with form anonymous of this survey in any results will be shared 25. Anything else you would like to let us know? let us know? to like would else you 25. Anything Field] [Open 24. Did/ do you have targeted therapies for your Melanoma? (please tick all that apply) Melanoma? (please tick all that your for therapies targeted have 24. Did/ you do O O O O O O O O Field] [Open 23. Did/ do you have immune therapies for your Melanoma? (please tick all that apply) Melanoma? tick all that (please your for immune therapies have 23. Did/ do you O O O O O O O O O O O Field] [Open The The Thank you for helping us understand what Quality of Life means to Melanoma patients. Melanoma patients. to means Quality what us understand helping of Life for you Thank Thank you Thank MPNE and ZIN MPNE and newsletter.

Appendices

296 11

CHAPTER

Summary Samenvatting in het Nederlands 12 Acknowledgements List of publications About the author

Summary 301 12 entire entire results results general general the respective respective The The the other hand, other hand, the constitution of of constitution the the good starting point potential alternative alternative potential a A value of drugs for for drugs of value study exploring the a assessment of scientific evidence evidence scientific of assessment human right in human right the a appraisal of evidence to reach decisions. In this In this decisions. reach to of evidence appraisal results indicate that policies depend on two factors: factors: two depend on policies that indicate results the remainder of this thesis, RWD was defined as health RWD as health defined thesis, of this was remainder The The concept. Discrepancies in definitions of definitions in Discrepancies amongst RWD concept. the analysis and/or synthesis of RWD. of and/or synthesis analysis “real-world” and thus qualify as RWD. On as RWD. qualify and thus “real-world” the moment, little is known with regards to RWD use in HTA of of RWD to in HTA use is known little moment, with regards the the context of RCTs. Meanwhile, real-world evidence (RWE) refers to to (RWE) evidence refers real-world Meanwhile, of RCTs. context the use of RWE. Chapter 3 presents RWE.use of 3 presents Chapter the the highest degree available; thus conventionally randomized controlled controlled randomized conventionally thus available; degree highest question at hand and hand at question the the different understanding amongst different stakeholders of what RWD precisely whatprecisely of stakeholders RWD different amongst understanding different study of definitions available in literature and stakeholder interviews. interviews. and stakeholder literature in available study of definitions a a value of RWD for HTA; stakeholders could disagree on whether particular disagree data could stakeholders of RWD HTA; value for definitions identified could be categorized into 4 categories, each imposing different each imposing different categories, 4 into categorized be could identified definitions World Health Organization (WHO). non- and Organization Health governmental In World countries, most greatest health gains for their citizens. citizens. their for health gains greatest the Consensus on definitions notwithstanding, it is important to explore available policies policies available is important it notwithstanding, on definitions explore to Consensus One framework facilitating transparent and accountable decision making on this front making decision front this on accountable and transparent facilitating framework One To begin with, it is important to define what real-world data (RWD) is. Chapter 2 Chapter (RWD) data is. real-world it is important with, begin what define to To When conducting health technology assessments, HTA agencies often resort to scientific often agencies resort scientific to HTA assessments, conductingWhen technology health The The SUMMARY policies of 6 European HTA agencies. agencies. HTA of 6 European policies Achieving good health has been recognized as been recognized has health good Achieving sources really do represent do represent really sources indicated indicated refer could stakeholders not all Moreover, as RWD. qualify or does not, does, criteria on what of definition an established to discussions confusions throughout to lead undoubtedly could stakeholders different on form which may been developed for definitions RWD have several For consensus-seeking. future for is. is. of HTA agencies for agencies for of HTA governmental parties participate in healthcare systems which aim to provide parties provide aim to which governmental participate systems healthcare in from derived is that evidence pertains to health technology assessment (HTA). In general, HTA is conducted by (public (public by is conducted HTA In general, pertains (HTA). assessment health technology to stages; two and comprises agencies HTA or private) the not infinite. are budgets healthcare However, healthcare. good to with access population decision Consequently, rise worldwide. to continue expenditures healthcare Meanwhile, achieve to resources allocate to on how questions with challenging faced makers are the for available clinical practice. routine of conditions under population patient so-called is regard this in RCTs complement may which evidence scientific for source data (RWD). data At real-world outside generated data presents presents pharmaceutical drugs and subsequent decision making. This thesis aims to address this gap gap this address to aims thesis This making. decision subsequent and drugs pharmaceutical in knowledge. evidence of of evidence thesis, we limit our scope to HTA of pharmaceutical drugs, as opposed to other health other health to as opposed drugs, of pharmaceutical HTA to our scope limit we thesis, and interventions. technologies highly selected patient (e.g. pertaininglimitations Several RCTs to trialsclinical (RCTs). all answering for them less suitable make of patients) follow-up and controlled populations assess to aim which agencies, HTA to relevance of questions RWE future. future. inability inability the learnings learnings the conduct of conduct of envisioned envisioned other hand, other hand, the Netherlands. Netherlands. the the result, result, the the the a following step would would step following valuable insights RWE insights valuable scheme. For example, example, For scheme. the the the scientific robustness of robustness RWE scientific Dutch healthcare setting and and setting healthcare Dutch same aims of CF in CF in same aims of the the the sole evidence source for HTA decision decision HTA for source evidence sole a implementation of CF in implementation the different contexts identified above. Chapter 4 Chapter above. identified contexts different belief that CF only partially met its aims, there partiallyonly CF that belief there aims, its met the the pre-defined period of 4 years for nearly all drugs. pre-definedfor nearly drugs. all years period4 of reports indicated that reports that indicated the difficulty of collecting RWE in practice, which required difficulty collecting required of which practice, RWE in HTA questions posed did not provide not provide posed did questions HTA relevance of RWE to HTA; one example being one example RWE of HTA; to relevance the appropriate use of drugs in practice. However, stakeholders stakeholders However, use of drugs in practice. appropriate positive impactpositive of CF on the the the parameters for which RWE is used. Three contexts emerged emerged contexts Three RWE which used. for is parameters the use of RWE is HTA and decision making has been implemented and decision making been implemented has use of RWE HTA is the perceived aim of CF. Conversely, there was some agreement agreement some was there Conversely, aim of CF. perceived the need for new policy new address need for to use of RWE in REA and PEA (i.e. cost-effectiveness assessments; CEA) CEA) assessments; use of RWE cost-effectiveness and PEA (i.e. in REA the the conditional financing (CF) of hospital drugs. Chapters 5 and 6 present present 5 and 6 Chapters (CF) of hospital drugs. financing conditional the the the actual use of RWE within context of CF was often of inadequate scientific quality. As quality. scientific often of inadequate of CF was context the the study on Netherlands, Netherlands, a the stakeholders on stakeholders CRS, namely namely CRS, context of use and and use of context CF process extended beyond extended beyond CF process Currently, few examples exist in literature whereby RWE is used for HTA purposes. In In purposes. RWE HTA for is used whereby in literature exist examples few Currently, In Having created an overview of HTA agencies’ policies on RWE, policies agencies’ an overview of HTA created Having a submitted for two-thirds of for submitted amongst differences demonstrated 6 interviews Stakeholder chapter in conducted answers. the amongst stakeholders on stakeholders amongst collected in collected to referred stakeholders example, For CF. for points improvement extensive inter-stakeholder collaboration, substantial funding and raised issues related to to issues related and raised funding substantial collaboration, inter-stakeholder extensive Despite governance. and ownership data However, stakeholders diverged on whether CF should be improved based on should be improved CF on whether diverged stakeholders However, principle, increased adoption of RWE use for HTA could thus be facilitated by by thus be facilitated could of RWE adoption HTA increased use for principle, can provide with regards to to regards with can provide as RWE that be regarded not emphasized also could policy with new replaced schemes. or practice, re-introducedinto and identified Focusing on RWE-related issues, issues, on RWE-related Focusing RWE practice implementing clinical when in changes rapid protocols. study for account to to also referred Stakeholders on agreement was making on effectiveness and cost-effectiveness. In this regard, stakeholders referred to referred stakeholders regard, In this cost-effectiveness. making and on effectiveness factors impacted that several the in in-depth with gained of experiences analyses shortcomings related indicated 5 CF drugs in chapter for published dossiers of HTA Analysis and decision-making methodological aspects of procedural, to of drugs for metastatic melanoma by 5 European HTA agencies. In general, RWE In inclusion general, agencies. HTA 5 European melanoma by metastatic for of drugs in prevalence melanoma It estimate mostly used to was REAs. than CEAs higher in was emerged differences several Moreover, CEAs. in effectiveness predict long-term or to REAs RWE use of assessments. their in agencies’ between presents presents the whereby RWE is used: relative effectiveness assessment (REA), pharmacoeconomic analysis analysis pharmacoeconomic (REA), assessment effectiveness RWE is used: relative whereby RWE (relative) In use for general, (CRS). schemes reimbursement conditional and (PEA) On by agencies. encouraged not was contexts in all estimates effectiveness in Differences and CRSs. to PEAs specific parameters other for directly requested it was RWE disease to use for regards with instance for agencies, between emerged also policies orphan diseases). (e.g. is challenging conducting RCTs where areas be to assess assess be to

Summary

302 12 Summary 303 12 use of use of literature literature summary the context of of context a a results imply imply results potential use use potential the the summaryIMI- of The The a third of all requests for for all requests of third a survey. survey. study whereby study whereby a a HTA community move towards towards community move HTA the topic of Chapter 10. Firstly, 10. Firstly, of Chapter topic full potential RWE can bear, thus better better thus RWE potential full bear, can the recommendations will help HTA agencies agencies will help HTA recommendations the the results demonstrated that social media may social media may that demonstrated results The The advantages and limitations of its use could be brought brought be could its use of limitations and advantages quick and time-efficient valuable assemble to manner the a conventional sources of RWE, such as registries and observational and RWE,of registries as such sources conventional study whereby social media was used to gather melanoma patient patient melanoma gather used to was social media study whereby the a points discussed above, how should should how above, discussed points lack of methodological guidance and standard practices to do so. practices to and standard guidance methodological lack of a IMI-GetReal initiative attempted to conduct such research in conduct such research to IMI-GetReal attempted initiative the establishment and conduct of patient/citizen panels at various stages of HTA HTA of stages various at panels conductand patient/citizen of establishment potential source of RWE of particularly REA, for source on aspectspotential events such as adverse the The The a Provided Provided Looking beyond Lookingbeyond perspectives on health-related quality of life (HRQoL) through (HRQoL) quality through of life perspectiveshealth-related on of drugs, symptom occurrence, quality of life, and drug adherence behaviour. Meanwhile, Meanwhile, behaviour. adherence drug and quality of life, occurrence, symptom of drugs, 9 presents chapter numerous case studies between 2015 and 2017. Chapter 7 presents presents 7 Chapter 2017. 2015 and between case studies numerous RWE. and using only In in accessing general, experiences GetReal’s presents 8 Chapter REAs. for evidence gather media to of social provide social media may be less resource-intensive and more efficient than similar data collection data than similar efficient and more be less resource-intensive may social media through decision making. However, many limitations are also associated with social media use for use for with social media associated also are limitations many making.decision However, including HTA, robust scientific research whereby whereby research scientific robust to light. light. to topic. on this conducted was review provide may social media that collect would data such to social media perspectives on HRQoL. Employing patient on data trials or point-of-care multi-centre than Similarly, studies. less resources require studies, could RWE be generated from novel sources that have not yet been exploited for for exploited been not yet have that sources novel from RWE could be generated studies, exploring by this question address 8 and 9 aim to purposes? Chapters HTA access to individual patient-level data (IPD) from RWE repositories submitted across all case all case across RWE submitted from repositories (IPD) data patient-level individual to access not being datasets to mostly related inaccessibility Reasons for successful. were studies an alternative As data. share to project within or unwillingness timelines research-ready from (AD) data using aggregate for options explored study teams case IPD, accessing to AD can although that indicated findings and observational However, registries studies. itself to lending mostly usefulness, it is often of limited literature, from obtained be easily different across effects treatment of analysis to than rather analyses statistical descriptive and populations. settings is provided of chapters 2-9, followed by an overview of new and ongoing RWE initiatives overview an and ongoing RWE of new by initiatives followed 2-9, of chapters is provided are recommendations eight Finally, thesis. in this questions addressed with overlap that In our opinion, both policy research. methodological and addressing measures provided this in developments future for important of focus stipulate areas recommendations these that field. rapidly-evolving It is our aspiration realizing to closer ever inch makers decision and more optimal use of RWE in decision making? This is This RWE of optimal use more in decision making? healthcare systems for all citizens. all citizens. for systems healthcare

Health Technology Assessment (HTA) biedt een transparant en verantwoord verantwoord en transparant een biedt (HTA) Assessment Technology Health Bij het uitvoeren van HTA maken HTA instanties het liefst gebruik van wetenschappelijke wetenschappelijke van gebruik het liefst instanties maken HTA HTA van Bij het uitvoeren studie een 2 bevat Hoofdstuk worden. RWD het begrip moet gedefinieerd Allereerst SAMENVATTING IN HET NEDERLANDS IN SAMENVATTING In de constitutie van de Wereldgezondheidsorganisatie (WHO) gezondheid goede dat staat Wereldgezondheidsorganisatie de In van de constitutie Overheidsinstantiesprivépartijen kaderen het in samen vaak is. werken mensenrecht een en realiseren te burgers voor gezondheid doel goede met als gezondheidssystemen van enerzijds de beschikbare zijn budgetten Echter, maken. te toegankelijk gezondheidszorg aan uitgaven de jaren afgelopen de stijgen beperkt anderzijds, en gezondheidszorg voor hiervan beslissingen moeilijke gevolg er vaak Als moeten opmerkelijk snel. gezondheidszorg te gezondheidswinst mogelijk veel zo middelen beperkte met de om worden genomen burgers. alle voor behalen In ondersteunen. het algemeen te kader besluitvorming om in de gezondheidszorg fasen: twee kent en instanties HTA of privé) (publieke door uitgevoerd HTA wordt evidentie van de hoogste kwaliteit en betrouwbaarheid. In de meeste gevallen betekent betekent In gevallen meeste de kwaliteit en betrouwbaarheid. de hoogste van evidentie RCTs van eigenschappen aantal Een klinische (RCTs). dit gerandomiseerde studies protocollen gecontroleerde en patiëntpopulaties homogene geselecteerde, (bijvoorbeeld het zijn voor geschikt altijd niet ze dat er toe leiden patiënten) van het monitoren voor van de waarde op richten die zich vaak vragen HTA-gerelateerde alle van beantwoorden klinische de dagelijkse van in de context de hele patiëntpopulatie voor geneesmiddelen informatie aanvullende die evidentie wetenschappelijke voor bron praktijk. alternatieve Een (RWD). dit moment Op data” “real-world is RCTs van opzichte ten kunnen opleveren zou en geneesmiddelen voor RWD van gebruik het in HTA omtrent bekend is er nog weinig genereren. te kennis heeft thesis Deze hierover als doel besluitvorming. resultaten De en interviews stakeholders. met beschikbarevan uit de literatuur definities De definities voor RWD. hanteren definities verschillende stakeholders dat zien laten wat omschrijvingen voor met verschillende in 4 categorieën, kunnen worden opgesplitst definitie bestaande een niet in staat waren stakeholders aantal RWD of géén Een wél is. in RWDbij het tot misverstanden definities kunnen leiden Verschillen RWDvoor citeren. kunnen stakeholders het oneens zijn of RWD van HTA; voor bediscussiëren en bespreken en dus weerspiegelen zijn Anderzijds, zijn. RWD “real-world” de databronnen specifieke voor definities RWD ontwikkeldmeerdere een goed startpunt die vormen zouden kunnen is RWD thesis als deze gedefinieerd Voor dit begrip. over consensus toekomstige voor “real-world is Verder verzameld. wordt RCTs van context de buiten die gezondheidsdata van RWD analyseren het is op die gebaseerd (RWE) als evidentie gedefinieerd evidence” bronnen). andere met óf in combinatie (alleen de wetenschappelijke beoordeling van de beschikbare evidentie (oftewel “assessment”) “assessment”) de beschikbare (oftewel van evidentie beoordeling de wetenschappelijke (bijvoorbeeld: overige van in de context en vervolgens bespreking de evidentie de van “appraisal”). komen (oftewel te besluit een om tot overwegingen maatschappelijke) en hulpmiddelen van niet geneesmiddelen, van name op HTA zich met richt thesis Deze gezondheidsinterventies. overige

Samenvatting in het Nederlands

304 12 Samenvatting in het Nederlands 305 12

Nadat een overzicht van RWE beleid gemaakt is, is de volgende stap om het om het stap RWE is de volgende van is, gemaakt beleid overzicht een Nadat In RWE Nederland van gebruik is het kader in het in besluitvorming toegepast van Naast consensus over definities, is het belangrijk het beleid van HTA instanties omtrent omtrent instanties HTA van beleid het belangrijk het is definities, over consensus Naast daadwerkelijk gebruik van RWE in de praktijk te analyseren. Hoofdstuk 4 bevat een studie een studie 4 bevat Hoofdstuk RWE analyseren. van gebruik te daadwerkelijk praktijk in de assessments”; “cost-effectiveness (oftewel en PEAs RWE het gebruik van REAs waarbij voor geneesmiddelen van beoordeling de voor wordt onderzocht instanties Europese bij 5 CEA) RWE in gebruikt vaker melanomen. Over wordt algemeen het van de behandeling voor voor incidentie en prevalentie de om gebruikt grotendeels RWE wordt dan REAs. CEAs te schatten termijn in op de lange middelen van effectiviteit en om de schatten in te REAs RWE instanties in meenemen de manier waarop in zien wij verschillen Verder CEAs. voor financiering voorwaardelijke namelijk vergoeding, de voorwaardelijke van dure (VF), rapporten voor HTA van een analyse 5 bevat Hoofdstuk geneesmiddelen. intramurale er tekortkomingen omtrent dat zien laten waren DeVF geneesmiddelen. resultaten De procedure VF kader. en besluitvormingsaspecten het van methodologische procedurele, periode de afgesproken dan langer geneesmiddelen alle nagenoeg voor VF duurde voor de rapporten laten zien punten, op RWE-gerelateerde men focust Wanneer 4 jaar. van RWE de verzamelde van kwaliteiten betrouwbaarheid matig de wetenschappelijke dat twee-derde voor heeft Derhalve alle kunnen RWE opleveren van was. geen antwoorden interviews stakeholder studie waarin een uitgevoerd zijn 6 bevat Hoofdstuk beleidsvragen. hoe stakeholders zien in verschillen laten De resultaten VF. hun ervaringen omtrent met met betrekking tot overeenkomsten er waren Anderzijds, VF voorstellen. zich het doel van RWE dat aan bijvoorbeeld belangrijke gaven Stakeholders VF kader. het van de voordelen gaven praktijk. de in middelen van het gepast gebruik Echter, gaf omtrent inzichten (kosten) als bewijs voor kon worden RWE beschouwd ook aan dat ze bron enige als niet van verminderde, relevantie RWE de die factoren ook noemden Stakeholders effectiviteit. en de impact in de praktijk daarvan veranderingen de snelle bijvoorbeeld vooraf op RWE van de verzameling aan dat stakeholders in gaven Verder studieprotocollen. gestelde stakeholders, tussen eiste samenwerking intensieve het omdat was moeizaam de praktijk bij weesziekten. bij hun beoordelingen. het gebruik van RWE van het gebruik RWE het kaart in waarbij een studie bevat 3 Hoofdstuk brengen. te dat aan geven De resultaten onderzocht. wordt instanties HTA Europese 6 van beleid RWE van gebruik perspectiefhet RWE het factoren: twee voor van is afhankelijk beleid kan Onderscheid waarvoorRWE parameters de en worden gemaakt wordt. gebruikt assessment”; effectiveness (“relative effectiviteit drietussen perspectieven: relatieve voorwaardelijke en PEA) analysis”; (“pharmacoeconomic kosteneffectiviteit REA), CRS). In wordt algemeen het schemes”; reimbursement (“conditional vergoedingskaders HTA door afgeraden effectiviteit (relatieve) van RWE inschattingen van gebruik het voor voor parameters RWE naar andere vraag directe is er een voor Daarentegen instanties. in bijvoorbeeld, instanties, tussen beleidsverschillen bestaan er Verder en CRSs. PEAs zoals zijn, voeren uit te moeilijk RWE van het gebruik RCTs waarbij bij ziekten voor beleid

Op dit moment zijn er weinig voorbeelden waarbij RWE gebruikt wordt voor HTA. Het Het HTA. voor RWE waarbij voorbeelden gebruikt wordt zijn er weinig Op dit moment RWE gegenereerd in hoeverre thesis is ook onderzocht de van onderdeel Als hoe de HTA suggesties aantal een van 10 is het beschrijven hoofdstuk van De focus uitvoeren van een wetenschappelijk robuust onderzoek waarin de voor- en nadelen van van en nadelen de voor- onderzoek waarin robuust wetenschappelijk een van uitvoeren kan Het RWERWE het opnemen van worden, in besluitvorming faciliteren. goed toelicht in het kader van voeren onderzoek uit te IMI-GetReal hierover project heeft geprobeerd van samenvatting een bevat 7 Hoofdstuk en 2017. 2015 tussen case studies meerdere ervaringen IMI-GetReal van krijgen met het RWE tot toegang het gebruik hiervan. en van krijgen te om toegang pogingen de van Inderde men maar in een slaagde algemeen het er geen dat redenen, uit RWE De (IPD) belangrijkste data bronnen. patiënt individuele tot onderzoek er óf niet bruikbaar voor datasets dat waren waren was, de data tot toegang IPD gebruik, voor probeerden alternatief Als delen. te data om de was geen bereidheid en observationele te studies uit registraties (AD) data onderzoekers geaggregeerde AD dat aan AD makkelijker de resultaten dat Ondanks gebruiken. verkrijgen geven te is, in de geneesmiddelen van effectiviteit de onderzoek omtrent niet bruikbaar robuust is voor klinische praktijk. en patiëntregistraties zoals bronnen bekende van in plaats bronnen, uit nieuwe wordt bevat 8 Hoofdstuk vraag. deze op zich observationeleHoofdstukken richten 9 en 8 studies. van bron potentiele een media of sociale onderzoeksvraag met als review een literatuur gebruikt wordt media sociale zien dat laten kan De REA. resultaten gegevens zijn voor bijwerkingen frequentie geneesmiddelen, van omtrent gegevensverzameling voor een studie bevat 9 Hoofdstuk en therapietrouw. leven kwaliteit van symptomen, van omtrent verzamelen perspectieven te om patiënt werden media gebruikt sociale waarbij een effectievere sociale media dat suggereren (HRQoL). De resultaten leven kwaliteit van verzamelen ten te perspectieven zouden om deze bieden kunnen manier en efficiëntere de in klinische(internationale) registraties of ad-hoc studies burgerpanels, van opzichte klinische praktijk. Anderzijds beperkingen zijn er belangrijke betrekking met gebruik het tot van afwezigheid de waaronder onderzoek vragen, die verder HTA media voor sociale van methodes. gestandaardiseerde en richtlijnen methodologische RWE van optimaal gebruik naar kangemeenschap besluitvorming in de op basis streven hoofdstukken samengevat. 2-9 worden eerste Ten resultaten. beschreven eerder de van die RWE lopende en toekomstige van initiatieven gegeven een overzicht wordt tweede Ten worden slotte, Ten thesis. huidige de van vraagstelling met de zien laten overeenkomsten verder als beleidsmaatregelen zowel voor gebracht naar voren aanbevelingen concrete acht belangrijke aanbevelingen deze onderzoek. mening wijzen methodologisch onze Naar ontwikkelingen hoop dat toekomstige is onze Het aan voor op dit gebied. aandachtspunten meeste het om helpen zullen verder en besluitnemers instanties HTA aanbevelingen deze gezondheidssystemen betere van uit RWE de realisatie tot zal meehelpen en halen te aanzienlijke financiering vereiste en ingewikkelde vragen omtrent de eigendom en beheer beheer en eigendom de ingewikkelde en omtrent vragen vereiste financieringaanzienlijke elkaar het met stakeholders de meeste nieuw dat eens waren Echter opriep. de data van verwezenlijken. VF te doelen van om de in de toekomst is nodig beleid voor iedereen. voor

Samenvatting in het Nederlands

306 12 Acknowledgements 307 12

midst midst the niceties in in niceties lot harder! lot harder! a the word of thanks. Your Your of thanks. word a animated notes, subtle subtle notes, animated beginning of 4 years of of of 4 years beginning tired mind. Furthermore, Furthermore, mind. tired full-time job. If it weren’t If weren’t it job. full-time a a milestones we reach in our in our reach we milestones the The The wonderful chapter. However, I I However, wonderful chapter. a the remedy to to remedy hobby reminds one of reminds hobby chapters written and ensured their their and ensured written the chapters end of challenges of exploring new territory. I I territory. new of exploring challenges a a joy of jazz! joy the the the muscle aches… Who knew they worked so well knewWho so well they worked muscle aches… the pleasure and privilege of working with such renowned and privilege of workingpleasure with such renowned people we become and become we people gentle touch of touch gentle the the the grit of research and and of research grit “Worry Sector”. With that in mind, to my colleagues at ZIN, thank thank ZIN, at colleagues my to mind, in that With “WorrySector”. the lot from you and enjoy our many discussions together. As for my my for As together. discussions our many and enjoy you lot from vast network of colleagues, friends and family; network of colleagues, vast has had each of whom the a a beginning of our continued collaboration, rather than farewell. than rather collaboration, continued of our beginning gym, thank you for for you gym, thank student have have student a outcome of more than merely our personal qualities and efforts. I am quite I am quite and efforts. qualities our personal merely than more of outcome the the the humble man’s attempt to thank all of you here. here. all of you thank to attempt man’s humble saying goes: “Big brains are important, but big biceps are importanter.” To my my To importanter.” important, are are but big biceps brains “Big goes: saying welcoming (and inspiring) group that you are, this would have all been all been have would this are, you that (and inspiring) group welcoming a contributions of many co-authors. To them, I wish to extend I wish to them, To co-authors. of many contributions “Health Sector” or Sector” “Health the the valuable and lasting impact on my modest and often tumultuous journey through life. life. modest and often impact and lasting journey tumultuous through valuable my on ACKNOWLEDGEMENTS also feel indebted to to indebted also feel It is an irrevocable truth that that truth Itirrevocable is an humbled and thankful to be writing this piece at at be writing piece this to thankful and humbled lives are are lives Dear Ton, Hans, Olaf and Wim… These words often marked words These Wim… and Olaf Hans, Ton, Dear Here’s Here’s a e-mails. Gentlemen, thank you for your patience and guidance throughout this period. this period. throughout and guidance patience your e-mails. for Gentlemen, thank you Seldom does scientists. Better still, those whom, despite their considerable achievements and positions, and positions, achievements considerable their whom, despite those still, Better scientists. this wish that been. I sincerely have as you and accepting encouraging as accessible, remain marksthesis Any claim on quality and impact this thesis may be perceived to make is also due to due to also is make to on qualityclaim impactand perceived be Any may thesis this the When literally translated, our department at Zorginstituut Nederland could either mean mean either Nederland could our department Zorginstituut at translated, literally When the many affiliations and diverse interests have enriched have interests and diverse affiliations many with all your success you and I wish as much as I have together our time enjoyed you hope endeavours. future relevance to many stakeholders, instead of one. To Rachel Kalf, Anne Willemsen, Renske ten Renske ten Willemsen, Anne Rachel Kalf, To one. of instead stakeholders, many to relevance Nijmeijer: and Hugo thank you Veelen Kim van Meijboom, Ard Ham, SandrineAcker, van me through helping for you for never making with never PhD me worry for my combining you about for learn to I continue as headaches? against buddies at at buddies Academic journeys are riddled with obstacles, frustrations and set-backs. In and set-backs. frustrations riddled obstacles, with are journeys Academic colleagues at Utrecht University, though we only recently met, thank you for making for me thank you met, recently only we though University, Utrecht at colleagues this past year. have you as involved and welcome as feel of these grim circumstances, circumstances, of these grim for you thank Band, Big Weathertown life. often were are you that group lovely and dynamics

promise to to promise pitch. Today, Today, pitch. a the passion for knowledge knowledge for passion the lifetime of friendship and lifetime a world of joy and compassion. joy of world a saying: “ibn balad”. You don’t have to look far to find its true true find its to far look to have don’t You balad”. “ibn saying: a wonderful childhood, for instilling in us in instilling wonderfulfor childhood, a very different arena. Here’s to Here’s arena. very different a favour. giftof the the family we share it with. Throughout these years, they have been my rock and and rock my been they have years, these Throughout it with. share family we lives of your children have been my biggest lesson and an honour to experience. experience. to an honour lesson and biggest my been have children of your lives Faithfully, the my dearest Femke, we did it! As we close this chapter at each other’s side, we start we our side, other’s each at chapter this close we As it! did we Femke, dearest my the we are defenders in defenders are we Gianni, in Egypt we have have we in Egypt Gianni, Thank you for for you Thank To my dearest Mother and Father, your unconditional generosity, devotion and commitment commitment and devotion generosity, unconditional your Mother Father, and dearest my To to always return return always strength. my meaning. We once stood alongside each other as defensive mid-fielders on mid-fielders as defensive other each alongside stood once We meaning. Finally, one’s greatest wealth resides in that one elusive, yet eternal place we call home call home we place eternal yet elusive, one in that resides wealth greatest one’s Finally, and to our eyes up opening and for Yours Yours Amr To come together, Always it. to I look forward how Oh, new one has no end. that next: except, may! what

Acknowledgements

308 12 List of publications 309 12

future. future. review review A the Assessment Assessment the status quo and quo status the world data studies of treatment treatment of studies data world ‐ comparative study of six HTA agencies. agencies. HTA study of six comparative a Dutch reality check. Health Policy. 2018 2018 check. reality Dutch Health Policy. The The Netherlands: Past, Present and Future. Results from Stakeholder Stakeholder Results from and Future. Present Netherlands: Past, the A, Stegenga H, Ciaglia A, Debray TP, Lees M, Happich M, Ryll B, Abrams K, Thwaites R, R, Thwaites K, Ryll Abrams M, M, Happich B, Lees TP, Debray A, H, Ciaglia A, Stegenga OF PUBLICATIONS OF Tool for Assessing Patient Perspectives on Quality of Life in Metastatic Melanoma: in Metastatic Melanoma: on Quality of Life Perspectives Patient Assessing for Tool a Comparative Study of Five HTA Agencies. PharmacoEconomics. 2017 Dec 2017 6:1-0. PharmacoEconomics. Agencies. HTA Five of Study Comparative feasibility study. Health and Quality of Life Outcomes. 2018 Jan *(under review). Jan *(under 2018 Outcomes. Quality Health and of Life study. feasibility LIST LIST Kalf R, Makady A, ten Ham R, Meijboom K, Goettsch W. Use of Social Media Use in Kalf R, MakadyW. Ham R, Meijboom K, ten A, Goettsch Garner S, Jonsson P. Practical implications of using real-world evidence (RWE) in comparative (RWE) evidence comparative in using real-world of implications Practical Garner S, Jonsson P. Effectiveness IMI-GetReal.Comparative learningsfrom Journal of research: effectiveness 6:6. Aug 2017 Research. Makady Interviews. Value in Health. 2018 Jan *(article prepared for submission). Jan *(article 2018 Health. in for prepared Value Interviews. Makady A, ten Ham R, de Boer A, Hillege H, Klungel O, Goettsch W. Policies for use of real- of use for Policies W. Makady Goettsch de Boer Ham R, A, Hillege A, ten H, Klungel O, (HTA): assessment in health technology data world of definitions based on literature and stakeholder interviews. Value in Health. 2017 2017 Value in Health. interviews. stakeholder and based on literature of definitions Jul 1;20(7):858-65. Makady A, de Boer A, Hillege H, Klungel O, Goettsch W. What is real-world data? data? real-world is What W. Makady Goettsch de Boer A, A, Hillege Klungel H, O, Berger ML, Sox H, Willke RJ, Brixner DL, Eichler HG, Goettsch W, Madigan D, Makady A, Madigan D, W, Brixner HG, Goettsch DL, Eichler RJ, Willke H, ML, Sox Berger real Good practicesfor SV. Wang R, Tarricone S, Schneeweiss Journal of Comparative Effectiveness Research. 2018 Jan *(article prepared for submission). Jan *(article 2018 Research. Effectiveness prepared Journal of Comparative a Health for Evidence Real-World W. Makady Goettsch A, de Boer A, Hillege H, Klungel O, on Perspectives of Pharmaceuticals: Assessment Technology Makady A, van Acker S, Nijmeijer H, de Boer A, Hillege H, Klungel O, Goettsch W. Conditional Conditional W. Goettsch Makady Nijmeijer S, de Boer H, Hillege A, Acker van A, KlungelH, O, in of Drugs Financing Social Media W. Makady A, Kalf Goettsch Ryll Spurrier R, G, de Boer Hillege A, B, Klungel H, O, as Jan *(under review). Jan *(under or Hopeful oncology. examples from review with explorative An Effectiveness: of Relative publication). for Journalhopeless? 2018 Mar of Medical Internet*(Accepted Research. Makady A, van Veelen A, Jonsson P, Moseley O, D’Andon A, de Boer A, de A, Hillege Klungel H, D’Andon MoseleyO, P, A, Jonsson Veelen Makady A, van Practice: (HTA) Assessment Technology in Health Data Real-World Using W. Goettsch O, A Implementing W. Goettsch Boer A, de HillegeA, H, Klungel O, Veelen Makady A, van in Practice: Managed Entry Agreements Value in Health. 2017 Apr 1;20(4):520-32. 2017 Apr Health. in Value

http://www. http://www. ISPE Special ISPE Special ‐ joint ISPOR joint https://www.imi-getreal.eu/ the

world evidence in health care decision making. Value in Health. 2017 2017 in Health. Value decision making. in health care evidence world ‐ W, Makady A. WP1: Deliverable D1.3. Glossary D1.3. terms Deliverable WP1: common of definitions of Makady A. W, A, Goettsch W. WP1: Deliverable D1.2 Review of current policies/perspectives policies/perspectives D1.2 Review of current WP1: Deliverable W. A, Goettsch P, Thwaites R, Makady A, Happich M, Granados A, Goettsch W, Garner S, Cerrata F, F, Garner S, Cerrata W, Makady R, A, Goettsch Granados M, A, Happich Thwaites P, A, Kalf R, Lees M, Goettsch W. WP1: Deliverable D1.5 ReportWP1: Deliverable – Metastatic Melanoma W. A, Kalf Goettsch M, R, Lees D1.6 ReportWP1: Deliverable – Metastatic Melanoma W. A, Kalf Goettsch M, R, Lees Barefoot B. Advancing Evidence Generation for New Drugs: IMI-GetReal’s Recommendations Recommendations New Drugs: IMI-GetReal’s for Generation Evidence Advancing B. Barefoot from: (UK): Manchester GetReal; Dec 2017 Available 7]. Evidence. [cited 2017. on Real-World http://www.imi-getreal.eu/Portals/1/Documents/01%20deliverables/2017-03-29%20-%20 WP1%20-%20Advancing%20Evidence%20Generation%20for%20New%20Drugs.pdf Jonsson Goettsch GetReal; (NL): [Internet]. Utrecht rounds) consultation from & replies comments (including from: Available 18]. Jan 2018 Report2017. [cited 115546. No.: [Internet]. Utrecht (NL): GetReal; 2014. Report No.: 115546. [cited 2017 Dec 7]. Available from: from: GetReal; (NL): Dec 2017 Available 7]. Report 2014. [cited 115546. [Internet]. Utrecht No.: https://www.imi-getreal.eu/Portals/1/Documents/01%20deliverables/GetReal%20D1.2%20 Current%20Policies%20and%20Perspectives%20FINAL_webversion.pdf Makady Makady from: GetReal; 2017 Dec (NL): Available 7]. [cited 2016. [Internet]. Utrecht from: Dec GetReal; 2017 (NL): Available 7]. [cited 2016. [Internet]. Utrecht Task Force on real on Force Task Sep1;20(8):1003-8. Portals/1/Documents/01%20deliverables/D1.3%20-%20GetReal%20Glossary%20of%20 Definitions%20of%20Common%20Terms%20%28Including%20Comments%20%26%20 Replies%20from%20Consultation%20Rounds%29.pdf imi-getreal.eu/Portals/1/Documents/01%20deliverables/Deliverable%201.5%20Report% 20-%20metastatic%20melanoma%20-%20webversion.pdf Makady imi-getreal.eu/Portals/1/Documents/01%20deliverables/Deliverable%20D1.6%20-%20 Metastatic%20Melanomav2_website%20version.pdf and/or comparative effectiveness: Recommendations from from Recommendations effectiveness: and/or comparative

List of publications

310 12 About the author 311 12 project project a Netherlands where Netherlands where the pharmacoeconomic assessor assessor pharmacoeconomic a German in Cairo. University the Dutch National Healthcare Institute as (ZIN) Healthcare National Dutch the ISPOR HTA Training Program. Training ISPOR HTA Huygen’s scholarship, Amr moved to Utrecht, Utrecht, to moved Amr scholarship, Huygen’s the a AUTHOR IMI-GetReal project. Currently, he works as as he works IMI-GetReal Currently, project. the New English School of Kuwait, he then moved to Cairo, Egypt. There, Amr completed his his completed Amr There, Egypt. Cairo, to he then moved SchoolNew English of Kuwait, Having secured secured Having In 2014, Amr began his career at at In began his career Amr 2014, ABOUT THE THE ABOUT Bachelor’s studies for Pharmacy at for studies and Biotechnology Bachelor’s Amr Makady was born in Adan, Kuwait. After completing his secondary education at his secondary completing After at Makady education Amr Kuwait. born in Adan, was The and policy advisor at ZIN. Within this function, Amr participates in numerous international this function, participates international Amr Within numerous in and policy ZIN. advisor at including: use in decision-making, and generation evidence real-world to projects related participates he teaching in numerous Moreover, REPEAT. JA3 and EUnetHTA IMI-ROADMAP, activities, including he completed his Master’s studies for Drug Innovation at Utrecht University. Throughout Throughout University. Utrecht at Drug Innovation studies for his Master’s he completed counterfeit including: fields, in various research conducted has Amr studies, his Master’s of generic modelling drug quality and risk-assessment in healthcare, corruption in drugs, methods. modelling of pharmacoeconomic standardization for manager