BMJ Open Is Committed to Open Peer Review. As Part of This Commitment We Make the Peer Review History of Every Article We Publish Publicly Available

Home , ATC code B05, Hydroxyethyl starch

BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

BMJ Open is committed to open peer review. As part of this commitment we make the peer review history of every article we publish publicly available.

When an article is published we post the peer reviewers’ comments and the authors’ responses online. We also post the versions of the paper that were used during peer review. These are the versions that the peer review comments apply to.

The versions of the paper that follow are the versions that were submitted during the peer review process. They are not the versions of record or the final published versions. They should not be cited or distributed as the published version of this manuscript.

BMJ Open is an open access journal and the full, final, typeset and author-corrected version of record of the manuscript is available on our site with no access controls, subscription charges or pay-per-view fees (http://bmjopen.bmj.com).

If you have any questions on BMJ Open’s open peer review process please email [email protected] http://bmjopen.bmj.com/ on September 25, 2021 by guest. Protected copyright. BMJ Open BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

Impact of real world evidence on medicines regulation in the European Union: a systematic assessment of European Medicines Agency referrals 2013-2017 ForJournal: peerBMJ Open review only Manuscript ID bmjopen-2018-028133

Article Type: Research

Date Submitted by the 23-Nov-2018 Author:

Complete List of Authors: Brown, Jeremy; London School of Hygiene and Tropical Medicine, Wing, Kevin; London School of Hygiene and Tropical Medicine, Evans, Stephen; LSHTM, Medical Statistics Unit Bhaskaran, Krishnan; LSHTM, NCDE Smeeth, Liam; London School of Hygiene and Tropical Medicine, Epidemiology and Population Health Douglas, Ian; London School of Hygiene and Tropical Medicine, Epidemiology and Population Health

Keywords: real world evidence, non-interventional studies, medicines regulation

http://bmjopen.bmj.com/ on September 25, 2021 by guest. Protected copyright.

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 31 BMJ Open

1 2

3 Impact of real world evidence on medicines regulation in the European Union: a systematic BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 assessment of European Medicines Agency referrals 2013-2017 7 8 9 10 Jeremy P Brown1, Kevin Wing1, Stephen J Evans1, Krishnan Bhaskaran1, Liam Smeeth1, Ian J Douglas1 11 12 13 1. Electronic Health Records Group, London School of Hygiene and Tropical Medicine, Keppel Street, 14 15 London, United Kingdom WC1E 7HT 16 17 18 For peer review only 19 Correspondence to: Jeremy Brown ([email protected]), Electronic Health Records Group, 20 21 London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT. Tel: 44 (0)20 22 23 7927 2259. 24 25 26 Word count (excluding title page, abstract, references, figures and tables): 3,378 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 2 of 31

1 2

3 Abstract BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 Objectives: To assess the use, and evaluate the usefulness, of non-interventional studies and 7 8 routinely collected healthcare data in post-marketing assessments conducted by the European 9 10 Medicines Agency (EMA). 11 12 13 14 15 Design: We reviewed and systematically assessed all referrals to the EMA made due to safety or 16 17 efficacy concerns that were evaluated between 1st January 2013 and 30th June 2017. We extracted 18 For peer review only 19 information from the assessment report and the referral notification. Two reviewers independently 20 21 22 assessed the contribution of non-interventional evidence to decision-making. 23 24 25 26 Results: The preliminary evidence leading to the assessment in 52 eligible referrals was mostly from 27 28 spontaneous reports (cited in 26 of 52 referrals) and randomised trials (22/52). In contrast, many 29 30 31 evidence types were used for the full assessment. Non-interventional studies were frequently used 32 33 in the full assessment for the evaluation of product safety (31/52) and product efficacy (18/52). In 34 35 particular, non-interventional studies were relied upon for the evaluation of safety and efficacy in 36 37 subgroups, the evaluation of safety relating to a rare adverse event, understanding product usage http://bmjopen.bmj.com/ 38 39 40 and misuse, and for evaluation of the effectiveness of risk minimisation measures. The most 41 42 common recommendations were changes to product information (43/52) and marketing 43 44 authorisation withdrawal or suspension (12/52). In the majority of referrals non-interventional

45 on September 25, 2021 by guest. Protected copyright. 46 evidence was judged to contribute to the decision made (30/52) and in 3 referrals it was the primary 47 48 49 source of evidence. 50 51 52 53 Conclusions: European regulatory decision-making relies on multiple evidence types, particularly 54 55 randomised trials, spontaneous reports and non-interventional studies. Non-interventional studies 56 57 had an important role particularly for the characterisation and quantification of adverse events, the 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 3 of 31 BMJ Open

1 2

3 evaluation of product usage, and for evaluating the effectiveness of regulatory action to minimise BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 risk. 7 8 9 10 Keywords: real world evidence, non-interventional studies, medicines regulation 11 12 13 14 15 16 17 18 For peer review only 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 4 of 31

1 2

3 Strengths and limitations of this study BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7  We assessed all safety and efficacy post-marketing authorisation referrals completed 8 9 through the European Medicines Agency (EMA) between January 2013 and June 2017. 10 11 Previous studies focused on marketing authorisation withdrawal only, but we included 12 13 referrals regardless of referral outcome. 14 15 16 17  While previous studies investigated which different evidence types are used in regulatory 18 For peer review only 19 decision-making, these did not look in depth at the role of these different evidence types, 20 21 and in particular at the role of non-interventional evidence, which we examined in detail. 22 23 24  Though the majority of studies cited in the referral assessment reports could be identified, 25 26 27 occasionally referencing was incomplete and there was insufficient detail to determine basic 28 29 study information. 30 31 32  Judgement on the role of non-interventional evidence in each assessment was to some 33 34 extent subjective and is dependent of what is recorded in the assessment report. However, 35 36

37 close agreement between two independent reviewers was observed. http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 5 of 31 BMJ Open

1 2

3 Introduction BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 There is an ongoing public debate about the use of routinely collected healthcare data in research, 8 1 2 9 particularly regarding concerns over patient confidentiality. Conducting research that meets strict 10 11 confidentiality requirements is of paramount importance, but for public trust to be established and 12 13 maintained there is also a need for evidence that research using patient records provides clear 14 15 benefits for the wider public. One potentially important and generally agreed benefit is in evaluating 16 17 18 the safety of drugs inFor real world peer use, though review surprisingly, there onlyis no comprehensive and systematic 19 20 evidence of how data from patient records is currently used in this context, with previous summaries 21 22 focussing largely on safety assessments resulting in marketing authorisation withdrawal or 23 24 suspension.3-11 25 26 27 st 28 Real world evidence has been defined in a number of ways. The US 21 Century Cures Act defines it 29 30 as “data regarding the usage, or the potential benefits or risks, of a drug derived from sources other 31 32 than traditional trials”. 12 An alternative definition of real world evidence, is evidence derived from 33 34 information collected for purposes other than research (i.e. routinely collected healthcare data such 35 36 13 37 as electronic healthcare records and insurance claims data). Whilst this evidence can be generated http://bmjopen.bmj.com/ 38 39 by (pragmatic) randomised clinical trials, currently non-interventional studies are the predominant 40 41 source of real-world evidence, and these are the focus of our study.13 14 42 43 44 Regulatory authorities increasingly require non-interventional evidence of drug effects. As a result of 45 on September 25, 2021 by guest. Protected copyright. 46 47 the US 21st Century Cures Act, the US Food and Drug Administration (FDA) is developing a 48 49 framework for the use of non-randomised “real world evidence” in the approval of new indications 50 51 and in post-authorisation medicinal product assessment.12 Similarly the European Medicines 52 53 Agency’s (EMA) adaptive pathway approach forms a new route of approval for medicines, allowing 54 55 56 conditional approval in areas of unmet need, subject to further evidence collection, particularly of 57 15 58 non-randomised real world evidence. EU legislation also now mandates the assessment of 59 60 medication effectiveness in routine clinical care where warranted.16 The focus on using non-

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 6 of 31

1 2

3 interventional data to evaluate the expected effectiveness of medicines is relatively new; agreed BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 methodologies and experience are limited. 7 8 9 The aim of this study was to systematically assess the type of evidence used in post-authorisation 10 11 drug regulation by the European Medicines Agency (EMA) to give a better understanding of the 12 13 contribution of non-interventional evidence and routinely collected data in this setting. 14 15 16 17 18 For peer review only 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 7 of 31 BMJ Open

1 2

3 Methods BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 We identified and reviewed all EMA post-marketing authorisation referrals made for safety and/or 8 st 9 efficacy concerns which were evaluated by an assessment committee between 1 January 2013 and 10 11 30th June 2017. The EMA is the European Union (EU) agency responsible for the scientific evaluation, 12 13 supervision, and safety monitoring of medicines used in the EU. Its work includes the evaluation of 14 15 applications for marketing authorisation and the monitoring of approved medicines. We evaluated 16 17 18 referrals which concludedFor after peer 2012 since EUreview legislation on pharmacovigilance only was strengthened in 19 20 that year. The evaluated referrals were made in accordance with the directives of European 21 22 Parliament: Article 107(i) of Directive 2001/83/EC, Article 31 of Directive 2001/83/EC, and Article 20 23 24 of Regulation No 726/2004 (supplementary appendix A – table 1). 25 26 27 28 When an EU member state or the European Commission has a significant concern regarding the 29 30 safety or efficacy of an approved medicine, a referral process is initiated. The EMA initially publishes 31 32 a notification which details the reasons for the referral. The safety and/or efficacy of the medicine is 33 34 then assessed in depth by designated member states and subsequently evaluated by one or more of 35 36

37 the EMA committees which include the Pharmacovigilance Risk Assessment Committee (PRAC), the http://bmjopen.bmj.com/ 38 39 Committee for Medicinal Products for Human Use (CHMP), and the Co-ordination Group for Mutual 40 41 Recognition and Decentralised Procedures – Human (CMDh). Finally, an assessment report is 42 43 published by the EMA for each referral, providing information on the recommendations made by the 44

45 on September 25, 2021 by guest. Protected copyright. 46 assessment committee and the reasons for these recommendations. 47 48 49 Eligible referrals were identified from the EMA website. One reviewer (JPB) evaluated the 50 51 notification and assessment report of each referral using a form (supplementary appendix B). 52 53 Information was extracted about the notification, the referral, the medicinal product, the adverse 54 55 56 events under study, and the types of evidence assessed (pre-clinical, non-randomised trials, 57 58 randomised trials, non-interventional studies, spontaneous reports and systematic reviews). In 59 60 addition, the reviewer assessed how different study types were used within the referral process and

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 8 of 31

1 2

3 categorised usage into: mechanism of action, pharmacokinetics/pharmacodynamics, efficacy, risk, BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 product usage, and the effectiveness of risk minimisation measures. 7 8 9 For each referral the adverse events under study were recorded and categorised into their 10 11 respective Medical Dictionary for Regulatory Activities (MedDRA) system organ class.17 Drugs were 12 13 categorised by Anatomical Therapeutic Chemical (ATC) classification system code.18 14 15 16 Two reviewers (JPB and IJD) independently assessed the recommendations made in the assessment 17 18 For peer review only 19 report, and the contribution of non-interventional studies to the recommendation made, with 20 21 disagreements resolved through discussion. We aimed to determine whether evidence from non- 22 23 interventional studies, and in particular, non-interventional studies using routinely collected data, 24 25 had an important or pivotal role in the assessment, in order to determine the contribution of this 26 27 28 type of evidence in this context. 29 30 31 Patient involvement 32 33 34 No patients were involved in the development of the research question, definition of study 35 36 outcomes or study design. We will disseminate our study findings to patients through social media 37 http://bmjopen.bmj.com/ 38 39 and using patient groups with an interest in data. 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 9 of 31 BMJ Open

1 2

3 Results BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 Referrals 8 9 st 10 Sixty potentially eligible referrals were identified with a committee opinion date between 1 January 11 12 2013 and the 31st June 2017. Of these 60 referrals, 8 were excluded, either because they related to 13 14 bioequivalence (n=4) or manufacturing concerns (n=3) rather than safety/efficacy concerns, or 15 16 because an assessment report was not yet available as of the 31st October 2017 (n=1). 17 18 For peer review only 19 20 The most frequent initiators of referrals were the European Commission (n=13), France (n=12), the 21 22 UK (n=8), Germany (n=4) and Italy (n=4). According to the referral notification and assessment 23 24 report, 21 of 52 referrals (40%) were made due to a combination of safety and efficacy concerns, 29 25 26 (56%) due to safety concerns only, and 2 (4%) due to efficacy concerns only. 27 28 29 30 Drug groups and adverse events 31 32 33 The most common drug groups defined according to ATC code were sex hormones and modulators 34 35 of the genital system, and analgesics (6 referrals each), followed by drugs used in diabetes, cough 36 37 and cold preparations, anti-inflammatory and anti-rheumatic products, and cardiac therapies (3 http://bmjopen.bmj.com/ 38 39 40 referrals each) (supplementary appendix A - table 2). The most common body systems on which 41 42 referred products acted were, based on ATC code, the nervous system (n=13), the cardiovascular 43 44 system (n=9), the alimentary tract and metabolism (n=8), and the genitourinary system and sex

45 on September 25, 2021 by guest. Protected copyright. 46 hormones (n=8) (supplementary appendix A - table 3). 47 48 49 50 The most commonly investigated adverse events included arterial thromboembolism (n=5), venous 51 52 thromboembolism (n=4), hypersensitivity (n=4) and renal impairment (n=3). The most frequent 53 54 category of adverse events according to MedDRA system organ class were cardiac and vascular 55 56 disorders (n=16); nervous system disorders (n=15); respiratory, thoracic and mediastinal disorders 57 58 59 (n=7); and skin and subcutaneous tissue disorders (n=7) (supplementary appendix A - table 4). 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 10 of 31

1 2

3 Evidence usage BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 Evidence cited by the initial notification and the referral assessment report was categorised by type 8 9 (table 1). Where no notification was available (in 12 of 52 referrals) information on the evidence 10 11 leading to the referral was extracted from the EMA website and the assessment report. The 12 13 evidence leading to referral was most commonly spontaneous reports (50%, 26/52) and randomised 14 15 trials (42%, n=22). Assessment reports also frequently cited spontaneous reports (73%, n=38) and 16 17 18 randomised trials (92%,For n=48), peerbut frequently review cited non-interventional only studies (79%, n=41) too. 19 20 Among the 52 referrals, in the assessment report, 31 (60%) cited non-interventional studies using 21 22 pre-existing routinely collected data (e.g. electronic medical records) and 33 (63%) cited non- 23 24 interventional studies using data collected specifically for research. Evidence was also frequently 25 26 27 cited from non-randomised trials (63%, 33/52), preclinical studies (56%, n=29) and systematic 28 29 reviews of randomised trials (52%, n=27). The quality of study description and referencing varied 30 31 considerably by assessment report. It was not always possible to find a corresponding study 32 33 publication or to ascertain the design for every study mentioned in the assessment; 63% (33/52) of 34 35 36 assessment reports referred to at least one study of unclear design.

37 http://bmjopen.bmj.com/ 38 39 Table 2 summarises how each type of evidence contributed to different aspects of the assessments. 40 41 The efficacy of medications was largely determined through evidence from randomised trials (cited 42 43 with regard to efficacy in 77% (40/52) of referrals), with non-interventional studies contributing 44

45 on September 25, 2021 by guest. Protected copyright. 46 information on efficacy in 25% (13/52) of assessments. Non-interventional studies contributed to 47 48 the assessment of efficacy, to a limited degree, and mostly when clinical trial data was limited, such 49 50 as in a subgroup (e.g. hydroxyethyl starch in trauma patients - EMEA/H/A-107i/1376; intravenous 51 52 nicardipine in children and pregnant women - EMEA/H/A-31/1339), for a product developed prior to 53 54 55 current regulatory requirements (e.g. polymyxin - EMEA/H/A-31/1383), or where a clinical trial 56 57 would be difficult to run due to sporadic and unpredictable need for therapy (e.g. adrenaline auto- 58 59 injectors - EMEA/H/A-31/1398; methysergide for cluster headache - EMEA/H/A-31/1335). 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 11 of 31 BMJ Open

1 2

3 For overall risks, both randomised trials (69%, 36/52) and non-interventional studies (60%, n=31) BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 were commonly assessed, alongside evidence from spontaneous reports (71%, n=37). Product 7 8 usage, where assessed, was almost entirely assessed based on non-interventional evidence (27%, 9 10 n=14). Mechanistic evidence was largely obtained from pre-clinical sources (31%, n=16), whilst 11 12 pharmacokinetics and pharmacodynamics were addressed through non-randomised trials (19%, 13 14 n=10), randomised trials (19%, n=10) and pre-clinical studies (12%, n=6). 15 16 17 18 Investigation of productFor usage peerand misuse wasreview almost entirely basedonly on non-interventional data 19 20 (table 2). Non-interventional evidence was also cited for estimating background incidence rates of 21 22 the adverse event in the population, and for characterising the prevalence of additional risk factors 23 24 and effect modifiers for the outcome under study. 25 26 27 28 Role of non-interventional evidence 29 30 31 Over half of the assessments relied at least in part on evidence from non-interventional studies to be 32 33 able to make recommendations for regulatory action (e.g. MA suspension or change in product 34 35 information) (table 3). Only in 11 of 52 assessments (21%) were no non-interventional studies cited. 36

37 http://bmjopen.bmj.com/ 38 In a further 11 referrals non-interventional studies were cited, but the reports did not indicate that 39 40 they contributed significantly to the decision made, either because only a few pertinent non- 41 42 interventional studies were cited (n=9), or due to limitations of the non-interventional studies (n=2). 43 44 45 In three referrals (combined hormonal contraceptives and thromboembolism; valproate, birth on September 25, 2021 by guest. Protected copyright. 46 47 48 defects and developmental disorders (EMEA/H/A-31/1387); and Kogenate Bayer/Helixate NexGen 49 50 and factor VIII inhibition (EMEA/H/C/275/A20/150/ EMEA/H/C/276/A20/143) non-interventional 51 52 studies alone were the primary source of evidence. When stratified by the outcome of the 53 54 assessment, it appears that non-interventional evidence more often contributed to decision-making 55 56 57 in referrals leading to prescribing changes (64%, 27/42) than those leading to suspension (33%, 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 12 of 31

1 2

3 4/12), though only 12 assessments led to suspension or withdrawal of marketing authorisation BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 (table 3). 7 8 9 Non-interventional studies were of particular use for the evaluation of safety in a subpopulation who 10 11 were largely or completely excluded from clinical trials, such as pregnant women. They were also 12 13 vital for estimating the risk of rare adverse outcomes, such as venous thromboembolism with oral 14 15 contraceptives, for which clinical trials were underpowered. Relative to spontaneous reports, non- 16 17 18 interventional studiesFor were particularly peer useful review when reporting wasonly strongly influenced by the media, 19 20 such as with human papillomavirus (HPV) vaccines (EMEA/H/A-20/1421), and when the outcome 21 22 was unlikely to be picked up by case reports, such as exposure-outcome associations with a long 23 24 latency period (e.g. Caustinerf arsenical and cancer (EMEA/H/A-31/1382)). Non-interventional 25 26 27 studies using routinely collected data were mostly used in a similar way to studies using data 28 29 collected for research (table 2). Studies using routinely collected data were particularly useful when 30 31 the outcome was rare, whereas studies using data collected for research purposes were most useful 32 33 where the outcome was poorly recorded in clinical records (e.g. Numeta G13%E/G16%E and 34 35 36 hypermagnesemia - EMEA/H/A-107i/1373).

37 http://bmjopen.bmj.com/ 38 39 Referral outcomes 40 41 42 The majority (98%, 51/52) of referrals led to regulatory action, with the assessment committee 43 44 recommending changes to the product information (83%, n=43) and particularly changes to the 45 on September 25, 2021 by guest. Protected copyright. 46 47 warnings, posology, undesirable effects and indication sections of the Summary of Product 48 49 Characteristics (table 4). In 12 of 52 (23%) referrals suspension or withdrawal of marketing 50 51 authorisation was recommended. Only for one referral into the safety of HPV vaccines was no 52 53 change recommended. 54 55 56 57 For many referrals (42%, n=22) the assessment committee required further specific studies to be 58 59 conducted, generally to elucidate safety, product usage and the effectiveness of risk minimisation 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 13 of 31 BMJ Open

1 2

3 measures. From a review of the assessment reports and the EU register of post-authorisation studies BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 (EU PAS register) most of these were non-interventional studies using routinely collected data or 7 8 data collected for research purposes (required in 19 referrals). 9 10 11 12 13 14 15 16 17 18 For peer review only 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 14 of 31

1 2

3 Discussion BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 In this comprehensive evaluation, we have shown that a wide range of evidence sources are used to 8 9 aid decision making during EU drug regulatory referrals. The three types cited in the majority of 10 11 assessments were randomised trials, spontaneous reports and non-interventional studies. Although 12 13 non-interventional evidence is rarely cited in notifications leading to a referral, it is cited 14 15 substantially during the detailed assessment of most issues, and in a few referrals was the primary 16 17 18 evidence type used inFor decision-making. peer Notably, review at the end of anonly assessment when 19 20 recommendations were made for evidence gaps to be filled, further non-interventional evidence 21 22 was required more often than any other type. 23 24 25 Each type of evidence appears to contribute to different aspects of a drug safety/efficacy referral, 26 27 28 allowing for a well-rounded assessment of medication risks and benefits. Unsurprisingly, given their 29 30 unique inferential advantages, randomised trials are relied on more than any other evidence type to 31 32 provide evidence of drug efficacy. Current usage of non-interventional evidence for efficacy largely 33 34 occurs where clinical trial data are limited. Increasingly, however, regulators require measures of 35 36

37 drug effectiveness in routine clinical care, for which well-designed non-interventional studies and http://bmjopen.bmj.com/ 38 39 pragmatic clinical trials using routinely collected data could be highly informative.12 15 16 40 41 42 To assess safety issues non-interventional evidence is heavily relied on alongside randomised trials 43 44 and spontaneous reports. Although less frequently cited, evidence from sources such as pre-clinical 45 on September 25, 2021 by guest. Protected copyright. 46 47 studies is occasionally relied on to provide information about mechanisms of effect or 48 49 pharmacokinetics/pharmacodynamics. 50 51 52 Strengths and Limitations 53 54 55 We were able to assess almost all referrals completed between 2013 and 2017, making this the most 56 57 58 comprehensive summary of recent drug regulatory decision making. The assessment reports are a 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 15 of 31 BMJ Open

1 2

3 comprehensive summary of the evidence used in decision making, meaning we were able to BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 determine how each type of evidence contributed to the final recommendations. 7 8 9 We were unable to directly assess the quality and validity of individual studies included in the 10 11 assessments. However, by reviewing the assessment reports, we evaluated how the evidence had 12 13 been rated by the committees and how it had contributed to the overall decisions made. 14 15 Occasionally studies were mentioned in assessment reports but no reference to a publication was 16 17 18 given, or referencing Forwas incomplete, peer and there review was insufficient only detail for readers to determine 19 20 basic information such as the study design or setting. 21 22 23 Judgement about how evidence was used in an assessment is to some extent subjective and is also 24 25 reliant on what is recorded in each assessment report. However, close agreement was achieved 26 27 28 between the two reviewers in this study. 29 30 31 Previous studies of the role of different evidence types in drug regulatory decision making have 32 33 largely focused on marketing authorisation withdrawals/suspensions.3-11 These studies highlight how 34 35 the balance of evidence types has shifted over time, from heavy reliance on spontaneous reports to 36

37 http://bmjopen.bmj.com/ 38 a more comprehensive reliance on varied evidence types. Over a similar time period the overall 39 40 number of non-interventional studies conducted and published also appears to be increasing, with 41 42 studies of UK electronic primary care data a prime example of this trend.19 With the increase in 43 44 research opportunities provided by new database linkages this publication trend is likely to continue. 45 on September 25, 2021 by guest. Protected copyright. 46 47 48 Unique strengths of non-interventional evidence 49 50 51 Non-interventional evidence was mostly used to inform on safety issues. Certain aspects of a safety 52 53 assessment appear to benefit from the availability of non-interventional evidence, such as the 54 55 quantification of rare events, investigation of special populations (e.g. pregnant women and 56 57 58 children), and informing about drug usage patterns. Whilst other types of evidence are also useful in 59 60 some of these areas, our study highlighted occasions when non-interventional evidence is unique

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 16 of 31

1 2

3 and vital for regulatory decision making. The risk of developmental disability and birth defects in the BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4

5 20 6 offspring of women taking valproate in pregnancy is a key example of this. This rare outcome 7 8 occurring in a group largely excluded from randomised trials could not have been characterised and 9 10 quantified without large, well-powered non-interventional studies. Similarly, the detailed 11 12 characterisation and quantification of adverse outcomes associated with NSAIDs and the oral 13 14 contraceptive could not have been done without good quality non-interventional evidence. Where 15 16 17 media interest led to stimulated spontaneous reporting, such as in the case of HPV vaccine and 18 For peer review only 19 various adverse effects, unbiased evidence from non-interventional settings was vital in providing 20 21 reassurance of safety, enabling continued use of the vaccine with no further action required. 22 23 Randomised trials used to justify licensing of medicines are simply too small to detect even relatively 24 25 26 common adverse reactions. The median number of patients studied on a new active substance is 27 28 1,708 for standard medicines and 438 for orphan medicines in the European Union21. Rare adverse 29 30 reactions (such as those occurring in 1 in 500 patients) will not have been detected as caused by the 31 32 medicine, but such rare effects can dramatically alter the benefit/risk balance of the medicine. 33 34 35 36 Tellingly, where the EMA’s committees call for further studies to be done, they mostly require non-

37 http://bmjopen.bmj.com/ 38 interventional evidence. There is increasingly a recognition that regulatory action to minimise risks 39 40 needs to be followed up to determine how effective it has been.22 Almost all drug regulatory action 41 42 involves making changes to how medicines are used in routine clinical care, and to determine 43 44

45 whether new directives are being followed requires evidence obtained in the routine clinical care on September 25, 2021 by guest. Protected copyright. 46 47 setting. Patterns of drug usage and quantification or characterisation of adverse events following 48 49 regulatory action are often required; non-interventional studies will be most important here, and 50 51 though spontaneous reports may also be useful, they are mostly unable to give quantitative 52 53 54 information. 55 56 57 There are three key elements required to ensure a successful future for non-interventional evidence 58 59 within the framework of drug regulatory science. First, there are legitimate concerns regarding the 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 17 of 31 BMJ Open

1 2

3 use of evidence from non-interventional studies in drug regulation given the potential problems of BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4

5 23 6 missing data and residual confounding. Through high quality study design, conduct and reporting 7 24 8 these issues can in many cases be resolved . Secondly, timely evidence is needed; non- 9 10 interventional studies can be conducted rapidly in response to emerging issues, or to measure the 11 12 effectiveness of past regulatory action. Thirdly, the data used in non-interventional studies needs to 13 14 be of the highest standard. This includes both the quality of the data and its generalisability to the 15 16 25 17 population from which it comes. Data quality can be monitored and assured by data custodians. 18 For peer review only 19 Generalisability relies on research data being drawn from a representative sample of the population. 20 21 Whether data are taken from existing medical records or newly collected for a specific study, this 22 23 requires the majority of patients to consent to their data to be included. For such a transaction 24 25 26 between researchers and patients to operate successfully, maintaining anonymity and 27 28 confidentiality is paramount. 29 30 31 Conclusions 32 33 34 Regulatory decision making about the safety and efficacy of medication in the European Union relies 35 36

37 on evidence obtained from a wide range of sources; most frequently from randomised trials, http://bmjopen.bmj.com/ 38 39 spontaneous reports and non-interventional studies. Non-interventional evidence can be vital for 40 41 characterising and quantifying adverse drug reactions, is often needed for monitoring the 42 43 effectiveness of regulatory action to minimise risks, and in certain situations will be the only 44

45 on September 25, 2021 by guest. Protected copyright. 46 available evidence. 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 18 of 31

1 2

3 Funding: This study was funded by the Association of the British Pharmaceutical Industry’s (ABPI’s) BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 Pharmaceutical Industry Health Information Group. JPB is supported by the grant from the ABPI for 7 8 the study in question. KB holds a Sir Henry Dale Fellowship jointly funded by the Wellcome Trust and 9 10 the Royal Society. LS was supported by a Wellcome Trust senior research fellowship in clinical 11 12 science (098504/Z/12/Z). IJD is supported by an unrestricted grant from GlaxoSmithKline. The 13 14 funders had no role in study design, data collection and analysis, decision to publish, or preparation 15 16 17 of the manuscript. 18 For peer review only 19 20 Competing interests: All authors have completed the ICMJE uniform disclosure form at 21 22 www.icmje.org/coi_disclosure.pdf and declare: JPB had financial support from ABPI for the 23 24 submitted work; IJD has received a grant from the ABPI for the study in question, financial support 25 26 27 from GlaxoSmithKline for work unrelated to the study in question, has consulted for GlaxoSmithKline 28 29 and Gilead, and holds stock in GlaxoSmithKline; SE is an independent European Commission- 30 31 appointed expert member of EMA’s PRAC; LS reports personal fees from GSK outside the submitted 32 33 work; there are no other relationships or activities that could appear to have influenced the 34 35 36 submitted work. The views expressed in this article are personal views of the author and may not be

37 http://bmjopen.bmj.com/ 38 understood or quoted as being made on behalf of or reflecting the position of the European 39 40 Medicines Agency or one of its committees or working parties. 41 42 43 Author contributions: All authors were involved in the design of the study. JPB and IJD undertook 44

45 on September 25, 2021 by guest. Protected copyright. 46 the data collection and drafted the manuscript. All authors interpreted the results, contributed to 47 48 later drafts of the manuscript, and approved the final manuscript. 49 50 51 Data sharing: All data analysed is available publically on the European Medicines Agency 52 53 (https://www.ema.europa.eu/) and EU Register of Post-Authorisation Studies 54 55 56 (http://www.encepp.eu/) websites. 57 58 59 Patient consent: Not required. 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 19 of 31 BMJ Open

1 2

3 Ethics approval: Not required. BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 For peer review only 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 20 of 31

1 2

3 Tables BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 Table 1: Evidence leading to referral and evidence cited in assessment report for the 52 included 6 referrals 7 8 Evidence leading to referrala In assessment report 9 Type of evidence Number of % of all Number of % of all 10 referrals referrals referrals referrals 11 Pre-clinical evidence 4 8% 29 56% 12 Non - randomised trials 1 2% 33 63% 13 14 Randomised trials 22 42% 48 92% 15 Non-interventional studies 13 25% 41 79% 16 17 i. Using routinely collected data 8 15% 31 60% 18 ii. Using data collectedFor for peer6 review12% only33 63% 19 research 20 Spontaneous reports 26 50% 38 73% 21 22 Systematic review of 7 13% 27 52% 23 randomised trials 24 Systematic review of non- 1 2% 4 8% 25 interventional studies 26 Systematic review combining 0 0% 8 15% 27 randomised trials & non- 28 interventional studies 29 Unclear 11 21% 33 63% 30 a. This was primarily based on the referral notification. However, for 12 of 52 referrals no notification was 31 available and evidence leading to initiation was instead obtained from the assessment report and from the 32 description of the referral on the EMA website. 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

Page 21 of 31 BMJ Open

1 2 3 Table 2: Number and percentage of all referrals (n=52) that use each type of evidence for each purpose 4 5 6 Type of evidence Usagea 7 Effectiveness 8 of risk 9 Risk - Risk - Usage of minimisation 10 Mechanism PK/PDb Efficacy Overall subgroup product measures 11 Pre-clinical evidence 16 (31%) 6 (12%) 2 (4%) 10 (19%) 1 (2%) 0 (0%) 0 (0%) 12 For peer review only 13 Non - randomised trials 1 (2%) 10 (19%) 18 (35%) 14 (27%) 2 (4%) 0 (0%) 0 (0%) 14 15 Randomised trials 3 (6%) 9 (17%) 40 (77%) 36 (69%) 7 (13%) 0 (0%) 1 (2%) 16 Non-interventional 3 (6%) 4 (8%) 18 (35%) 31 (60%) 5 (10%) http://bmjopen.bmj.com/ 14 (27%) 0 (0%) 17 18 Non-interventional using routinely collected data 0 (0%) 1 (2%) 8 (15%) 25 (48%) 4 (8%) 10 (19%) 0 (0%) 19 20 Non-interventional using data collected for research 2 (4%) 4 (8%) 13 (25%) 20 (38%) 3 (6%) 7 (13%) 0 (0%) 21 Spontaneous reports 2 (4%) 0 (0%) 3 (6%) 37 (71%) 6 (12%) 4 (8%) 0 (0%) 22 23 Systematic review of randomised trials 0 (0%) 0 (0%) 19 (37%) 10 (19%) 1 (2%) 0 (0%) 0 (0%) 24 on September 25, 2021 by guest. Protected copyright. 25 Systematic review of non-interventional studies 0 (0%) 0 (0%) 0 (0%) 4 (8%) 1 (2%) 0 (0%) 0 (0%) 26 27 Systematic review of randomised trials & non- 0 (0%) 1 (2%) 2 (4%) 4 (8%) 0 (0%) 0 (0%) 0 (0%) 28 interventional studies 29 Unclear study design 1 (2%) 8 (15%) 12 (23%) 10 (19%) 0 (0%) 1 (2%) 0 (0%) 30 31 a. Usage was categorised, as detailed in the table, into: mechanism of adverse event with product usage, pharmacokinetics/pharmacodynamics of product, efficacy of 32 product, risk of adverse events with product, risk of adverse events with product in a subpopulation, usage/misuse of a product, and effectiveness of regulatory risk 33 minimisation measures. 34 b. Pharmacokinetics/pharmacodynamics 35 36 37 38 39 40 41 42 43 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

BMJ Open Page 22 of 31

1 2 3 Table 3: Usage of non-interventional studies in referral assessment reports 4 5 6 Usage of non-interventional studies All referrals (n=52) Referrals leading to MA Referrals leading to changes to 7 withdrawal/suspension product information (n=43) 8 (n=12) 9 Number of % of all Number of % of all Number of % of all 10 referrals referrals referrals referrals referrals referrals 11 No evidence from non-interventional studies was cited in the report 11 21% 4 33% 7 16% 12 For peer review only 13 14 Evidence from non-interventional studies was cited, but made little to 11 21% 4 33% 9 21% 15 no contribution to the decision 16 http://bmjopen.bmj.com/ 17 The decision was consistent with evidence from non-interventional 27 52% 4 33% 24 56% 18 studies, and also consistent with other evidence 19 20 The decision was consistent with evidence from non-interventional 3 6% 0 0% 3 7% 21 studies AND this evidence was the primary or only factor involved in the 22 decision e.g. there was some spontaneous reports and some large non- 23 interventional studies 24 on September 25, 2021 by guest. Protected copyright. 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 23 of 31 BMJ Open

1 2

3 BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from Table 4: Recommendations made as a result of assessment for the 52 included referrals 4 5 6 Recommendation Number of % of all referrals 7 referrals 8 No change 1 2% 9 Further evidence before decision- 2 4% 10 making 11 Suspension or withdrawal of 12 23% 12 marketing authorisation 13 Change to product information 43 83% 14 15 By section of the Summary of 16 Product Characteristics: 17 - Indication 24 46% 18 - Posology For peer28 review54% only 19 20 - Contraindications 22 42% 21 - Warnings 39 75% 22 - Interactions 14 27% 23 24 - Pregnancy 10 19% 25 - Driving/machinery 2 4% 26 27 - Undesirable effects 26 50% 28 - Overdose 3 6% 29 30 - Studies 13 25% 31 - Nature and contents 3 6% 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 24 of 31

1 2

3 References BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 1. Papoutsi C, Reed JE, Marston C, et al. Patient and public views about the security and privacy of 7 Electronic Health Records (EHRs) in the UK: results from a mixed methods study. BMC 8 medical informatics and decision making 2015;15(1):86. 9 2. Campos-Castillo C, Anthony DL. The double-edged sword of electronic health records: implications 10 for patient disclosure. Journal of the American Medical Informatics Association 11 2014;22(e1):e130-e40. 12 3. Arnaiz JA, Carne X, Codina C, et al. The use of evidence in pharmacovigilance - Case reports as the 13 reference source for drug withdrawals. European Journal of Clinical Pharmacology 14 2001;57(1):89-91. doi: DOI 10.1007/s002280100265 15 16 4. Clarke A, Deeks JJ, Shakir SAW. An assessment of the publicly disseminated evidence of safety 17 used in decisions to withdraw medicinal products from the UK and US markets. Drug Safety 18 2006;29(2):175-81.For doi: peer Doi 10.2165/00002018-200629020-00008 review only 19 5. McNaughton R, Huet G, Shakir S. An investigation into drug products withdrawn from the EU 20 market between 2002 and 2011 for safety reasons and the evidence used to support the 21 decision-making. BMJ open 2014;4(1):e004221. 22 6. Olivier P, Montastruc JL. The nature of the scientific evidence leading to drug withdrawals for 23 pharmacovigilance reasons in France. Pharmacoepidemiol Drug Saf 2006;15(11):808-12. doi: 24 25 10.1002/pds.1248 26 7. Onakpoya IJ, Heneghan CJ, Aronson JK. Post-marketing withdrawal of anti-obesity medicinal 27 products because of adverse drug reactions: a systematic review. BMC medicine 28 2016;14(1):191. 29 8. Onakpoya IJ, Heneghan CJ, Aronson JK. Post-marketing withdrawal of analgesic medications 30 because of adverse drug reactions: a systematic review. Expert opinion on drug safety 31 2018;17(1):63-72. 32 9. Paludetto MN, Olivier-Abbal P, Montastruc JL. Is spontaneous reporting always the most 33 important information supporting drug withdrawals for pharmacovigilance reasons in 34 35 France? Pharmacoepidemiology and drug safety 2012;21(12):1289-94. 36 10. Rawson NS. Drug safety: withdrawn medications are only part of the picture. BMC Med

37 2016;14(1):28. doi: 10.1186/s12916-016-0579-5 [published Online First: 2016/02/14] http://bmjopen.bmj.com/ 38 11. Lane S, Lynn E, Shakir S. Investigation assessing the publicly available evidence supporting 39 postmarketing withdrawals, revocations and suspensions of marketing authorisations in the 40 EU since 2012. BMJ open 2018;8(1):e019759. 41 12. 21st Century Cures Act - House of Representatives 34. 2015. 42 13. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-World Evidence - What Is It and What Can It 43 44 Tell Us? New England Journal of Medicine 2016;375(23):2293-97.

45 14. Kalkman S, van Thiel GJ, Zuidgeest MG, et al. Series: Pragmatic trials and real world evidence: on September 25, 2021 by guest. Protected copyright. 46 Paper 4. Informed consent. Journal of Clinical Epidemiology 2017;89:181-87. 47 15. Agency EM. Final report on the adaptive pathways pilot, 2016. 48 16. Regulation (EU) No 1235/2010 of the European Parliament and of the Council. Official Journal of 49 the European Union 2010 50 17. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for 51 Human Use. Medical Dictionary for Regulatory Activities [Available from: 52 53 https://bioportal.bioontology.org/ontologies/MEDDRA. 54 18. World Health Organization Collaborating Centre for Drug Statistics Methodology. Anatomical 55 Therapeutic Chemical Classification System [Available from: 56 https://www.whocc.no/atc_ddd_index/. 57 19. Vezyridis P, Timmons S. Evolution of primary care databases in UK: a scientometric analysis of 58 research output. BMJ open 2016;6(10):e012785. 59 20. Valproate Art. 31 Assessment Report: European Medicines Agency, 2014. 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 25 of 31 BMJ Open

1 2

3 21. Duijnhoven RG, Straus SM, Raine JM, et al. Number of patients studied prior to approval of new BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 medicines: a database analysis. PLoS medicine 2013;10(3):e1001407. 5 6 22. Commission E. Directive 2010/84/EU of the European Parliament and of the Council of 15 7 December 2010 amending, as regards pharmacovigilance, Directive 2001/83/EC on the 8 Community code relating to medicinal products for human use. 2010 9 23. Kesselheim AS, Avorn J. New “21st Century Cures” legislation: speed and ease vs science. Jama 10 2017;317(6):581-82. 11 24. Goodman SN, Schneeweiss S, Baiocchi M. Using design thinking to differentiate useful from 12 misleading evidence in observational research. Jama 2017;317(7):705-07. 13 25. Herrett E, Gallagher AM, Bhaskaran K, et al. Data resource profile: clinical practice research 14 datalink (CPRD). International journal of epidemiology 2015;44(3):827-36. 15 16 17 18 For peer review only 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 26 of 31

1 2

3 Supplementary Appendix A BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 Table 1: EMA referrals categorised by type and date 7 8 Referral category Frequency by CHMP/CMDh opinion date Total 9 2013 2014 2015 2016 2017 Jan- 10 Jun 11 Article 107i procedures 5 1 0 0 0 6 12 Article 20 procedures 2 3 1 4 1 11 13 Article 31 referrals 13 13 5 3 1 35 14 Total 20 17 6 7 2 52 15 16 17 Table 2: ATC therapeutic subgroups by frequency for the 52 included referrals 18 For peer review only 19 20 ATC subgroup Subgroup definition No. of 21 code referrals 22 G03 Sex hormones and modulators of the genital system 6 23 N02 Analgesics 6 24 A10 Drugs used in diabetes 3 25 C01 Cardiac therapy 3 26 M01 Anti-inflammatory and antirheumatic products 3 27 None Not applicable/available 3 28 R05 Cough and cold preparations 3 29 A03 Drugs for functional gastrointestinal disorders 2 30 B05 Blood substitutes and perfusion solutions 2 31 C04 Peripheral vasodilators 2 32 C10 Lipid modifying agents 2 33 G02 Other gynelogicals 2 34 L01 Antineoplastic agents 2 35 M03 Muscle relaxants 2 36 N05 Psycholeptics 2 37 http://bmjopen.bmj.com/ 38 R03 Drugs for obstructive airway diseases 2 39 B01 Antithrombotic agents 1 40 B02 Antihemorrhagics 1 41 B03 Antianemic preparations 1 42 C08 Calcium channel blockers 1 43 C09 Agents acting on the renin-angiotensin system 1 44 J01 Antibacterials for systemic use 1

45 J02 Antimycotics for systemic use 1 on September 25, 2021 by guest. Protected copyright. 46 J05 Antivirals for systemic use 1 47 L04 Immunosuppresants 1 48 M05 Drugs for treatment of bone diseases 1 49 N03 Antiepileptics 1 50 N07 Psychoanaleptics 1 51 R02 Throat preparations 1 52 R07 Other respiratory system products 1 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 27 of 31 BMJ Open

1 2

3 Table 3: ATC section by frequency for the 52 included referrals BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 Number 6 of 7 ATC section Section definition referrals 8 9 N Nervous system 13 10 C Cardiovascular system 9 11 A Alimentary tract and metabolism 8 12 G Genitourinary system and sex hormones 8 13 R Respiratory system 7 14 M Musculoskeletal system 6 15 B Blood and blood forming organs 5 16 J Anti-infectives for systemic use 3 17 L Antineoplastic and immunomodulating agents 3 18 D DermatologicalsFor peer review only0 19 Systemic hormonal preparations, excluding sex 20 H hormones and insulins 0 21 P Antiparasitic products, insecticides and repellents 0 22 S Sensory organs 0 23 V Various 0 24 25 26 27 28 Table 4: MedDRA system organ class (SOC) of adverse event by frequency for the 52 included 29 referrals 30 31 System Organ Class (SOC) Number 32 of 33 referrals 34 Cardiac disorders 16 35 36 Vascular disorders 15

45 Renal and urinary disorders 5 on September 25, 2021 by guest. Protected copyright. 46 Blood and lymphatic system disorders 4 47 Hepatobiliary disorders 4 48 Metabolism and nutrition disorders 4 49 50 Neoplasms benign, malignant and unspecified 4 51 Congenital, familial and genetic disorders 2 52 Endocrine disorders 2 53 54 Musculoskeletal and connective tissue disorders 2 55 Injury, poisoning and procedural complications 1 56 Reproductive system and breast disorders 1 57 Sexual function and fertility disorders 1 58 59 Surgical and Medical Procedures 1 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 28 of 31

1 2

3 Supplementary Appendix B: Data collection form BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 1. Basic information about EMA reference number 6 referral 7 Initiated by (e.g. MHRA, European 8 Commission) 9 Referral/procedure type (Article 107i/Article 10 11 31/Article 20) 12 Decision making model (e.g. PRAC-EC) 13 14 Cause of referral (Safety/Efficacy/Safety and 15 efficacy) 16 Cause of referral – description 17 18 For peerCHMP opinion/CMDh review position dateonly 19 20 2. Information about product Review title 21 and adverse event 22 Substance name 23 24 25 Product usage 26 27 ATC group (e.g. N03 - Antiepileptics) 28 29 30 Product class (as listed on EMA website) 31 32 33 Adverse events 34 35 MedDRA system organ classes of 36 adverse events

37 http://bmjopen.bmj.com/ 38 3. Determine the types of Source of evidence (Notification/Assessment 39 evidence leading to the referral report/EMA webpage) 40 a. Pre-clinical evidence (Yes/No/Unclear) 41 42 b. Non-randomised trials (Yes/No/Unclear) 43 44

45 c. Randomised trials (Yes/No/Unclear) on September 25, 2021 by guest. Protected copyright. 46 47 d. Observational studies (Yes/No/Unclear) 48 49 50 i. Using routinely collected real world (Yes/No/Unclear) 51 data e.g. electronic health records 52 ii. Using primary data collection e.g. (Yes/No/Unclear) 53 pregnancy registry 54 e. Spontaneous reports (Yes/No/Unclear) 55 56 57 f. Systematic review of randomised trials (Yes/No/Unclear) 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 29 of 31 BMJ Open

1 2

3 g. Systematic review of observational (Yes/No/Unclear) BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 studies 5 6 h. Systematic review combining (Yes/No/Unclear) 7 randomised trials & observational 8 studies 9 i. Unclear design (Yes/No) 10 11 4. a) Determine the types of a. Pre-clinical evidence (Yes/No/Unclear) 12 evidence used in each 13 assessment report b. Non-randomised trials (Yes/No/Unclear) 14 15 16 c. Randomised trials (Yes/No/Unclear) 17 18 For peerd. Observational review studies only(Yes/No/Unclear) 19 20 i. Using routinely collected real world (Yes/No/Unclear) 21 data e.g. electronic health records 22 23 ii. Using primary data collection e.g. (Yes/No/Unclear) 24 pregnancy registry 25 26 e. Spontaneous reports (Yes/No/Unclear) 27 28 f. Systematic review of randomised trials (Yes/No/Unclear) 29 30 g. Systematic review of observational (Yes/No/Unclear) 31 studies 32 33 h. Systematic review combining (Yes/No/Unclear) 34 randomised trials & observational 35 studies 36 i. Unclear design (Yes/No)

37 http://bmjopen.bmj.com/ 38 4. b) Summarise the types of a. Pre-clinical evidence 39 evidence used in each 40 assessment report b. Non - randomised trials 41 42 43 c. Randomised trials 44

45 on September 25, 2021 by guest. Protected copyright. d. Observational studies 46

48 e. Spontaneous reports

49 50 f. Systematic review of clinical trials 51 52 53 g. Systematic review of observational 54 studies 55 h. Systematic review combining clinical 56 trials & observational studies 57 i. Unclear design 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 30 of 31

1 2

12 d. Change to availability e.g. P to (Yes/No) 13 POM 14 e. Suspension or revocation of (Yes/No) 15 marketing authorisation 16 Summary of decision 17 18 For peer review only 19 6. If there was a 4.1 Therapeutical indications 20 recommendation for a change to 21 product information, which 4.2 Posology and method of 22 sections of the summary of administration 23 product characteristics (SmPc) 4.3 Contraindications 24 were affected? 25 4.4 Special warnings and 26 precautions for use 27 4.5 Interactions with other 28 medicinal products and other 29 forms of interaction 30 4.6 Fertility, pregnancy and 31 lactation 32 4.7 Effects on ability to drive and 33 use machines 34 4.8 Undesirable effects 35 36 4.9 Overdose

37 http://bmjopen.bmj.com/ 38 Other 39 7. Determine how observational (a. No evidence from observational studies was cited in the report/ 40 studies contributed to the b. Evidence from observational studies was cited, but made little to no 41 decision made. Judgement is contribution to the decision/ 42 involved in this step and the c. Evidence from observational studies was cited, but the decision was 43 assessment will be conducted contrary to this evidence/ 44 independently by two d. The decision was consistent with evidence from observational 45 researchers. studies, and also consistent with other evidence/ on September 25, 2021 by guest. Protected copyright. 46 e. The decision was consistent with evidence from observational 47 studies AND this evidence was the primary or only factor involved in 48 the decision/ 49 f. Unclear) 50 8. What was useful (or 51 otherwise) about the evidence 52 from observational studies? 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 31 of 31 BMJ Open

1 2

3 9. If no observational studies Yes/no? (Yes/No/Unclear) BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 were available, were such Further information 5 studies feasible and could they 6 have been useful? 7 8 9 10. Does the action taken as a Yes/no? (Yes/No/Unclear) 10 result of the referral require 11 future research? Is further non-interventional evidence (Yes/No/Unclear) 12 required? 13 Further information 14 15 16 17 18 For peer review only 19 20 Design of further non-interventional (using data collected for 21 studies in PAS register research/ using routinely 22 collected data) 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

Use of real world evidence in post-marketing medicines regulation in the European Union: a systematic assessment of European Medicines Agency referrals 2013-2017 ForJournal: peerBMJ Open review only Manuscript ID bmjopen-2018-028133.R1

Article Type: Research

Date Submitted by the 22-May-2019 Author:

Primary Subject Pharmacology and therapeutics Heading:

Secondary Subject Heading: Public health http://bmjopen.bmj.com/

Keywords: real world evidence, non-interventional studies, medicines regulation

on September 25, 2021 by guest. Protected copyright.

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 36 BMJ Open

1 2

3 Use of real world evidence in post-marketing medicines regulation in the European Union: a BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 systematic assessment of European Medicines Agency referrals 2013-2017 7 8 9 10 Jeremy P Brown1, Kevin Wing1, Stephen J Evans1, Krishnan Bhaskaran1, Liam Smeeth1, Ian J Douglas1 11 12 13 1. Electronic Health Records Group, London School of Hygiene and Tropical Medicine, Keppel Street, 14 15 London, United Kingdom WC1E 7HT 16 17 18 For peer review only 19 Correspondence to: Jeremy Brown ([email protected]), Electronic Health Records Group, 20 21 London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT. Tel: 44 (0)20 22 23 7927 2259. 24 25 26 Word count (excluding title page, abstract, references, figures and tables): 3,355 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 2 of 36

1 2

3 1 Abstract BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 Objectives: To assess the use, and evaluate the usefulness, of non-interventional studies and 7 8 3 routinely collected healthcare data in post-marketing assessments conducted by the European 9 10 4 Medicines Agency (EMA). 11 12 13 5 14 15 6 Design: We reviewed and systematically assessed all referrals to the EMA made due to safety or 16 17 7 efficacy concerns that were evaluated between 1st January 2013 and 30th June 2017. We extracted 18 For peer review only 19 8 information from the assessment report and the referral notification. Two reviewers independently 20 21 22 9 assessed the contribution of non-interventional evidence to decision-making. 23 24 10 25 26 11 Results: The preliminary evidence leading to the assessment in 52 eligible referrals was mostly from 27 28 12 spontaneous reports (cited in 26 of 52 referrals) and randomised trials (22/52). In contrast, many 29 30 31 13 evidence types were used for the full assessment. Non-interventional studies were frequently used 32 33 14 in the full assessment for the evaluation of product safety (31/52) and product efficacy (18/52). In 34 35 15 particular, non-interventional studies were relied upon for the evaluation of safety and efficacy in 36 37 16 subgroups, the evaluation of safety relating to a rare adverse event, understanding product usage http://bmjopen.bmj.com/ 38 39 40 17 and misuse, and for evaluation of the effectiveness of risk minimisation measures. The most 41 42 18 common recommendations were changes to product information (43/52) and marketing 43 44 19 authorisation withdrawal or suspension (12/52). In the majority of referrals non-interventional

45 on September 25, 2021 by guest. Protected copyright. 46 20 evidence was judged to contribute to the decision made (30/52) and in 3 referrals it was the primary 47 48 49 21 source of evidence. 50 51 22 52 53 23 Conclusions: European regulatory decision-making relies on multiple evidence types, particularly 54 55 24 randomised trials, spontaneous reports and non-interventional studies. Non-interventional studies 56 57 25 had an important role particularly for the characterisation and quantification of adverse events, the 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 3 of 36 BMJ Open

1 2

3 1 evaluation of product usage, and for evaluating the effectiveness of regulatory action to minimise BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 risk. 7 8 3 9 10 4 Keywords: real world evidence, non-interventional studies, medicines regulation 11 12 13 14 15 16 17 18 For peer review only 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 4 of 36

1 2

3 1 Strengths and limitations of this study BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 2  We assessed all safety and efficacy post-marketing authorisation referrals completed 8 9 3 through the European Medicines Agency (EMA) between January 2013 and June 2017. 10 11 4 Previous studies focused on marketing authorisation withdrawal only, but we included 12 13 5 referrals regardless of referral outcome. 14 15 16 17 6  While previous studies investigated which different evidence types are used in regulatory 18 For peer review only 19 7 decision-making, these did not look in depth at the role of these different evidence types, 20 21 8 and in particular at the role of non-interventional evidence, which we examined in detail. 22 23 24 9  Though the majority of studies cited in the referral assessment reports could be identified, 25 26 27 10 occasionally referencing was incomplete and there was insufficient detail to determine basic 28 29 11 study information. 30 31 32 12  Judgement on the role of non-interventional evidence in each assessment was to some 33 34 13 extent subjective and is dependent of what is recorded in the assessment report. However, 35 36

37 14 close agreement between two independent reviewers was observed. http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 5 of 36 BMJ Open

1 2

3 1 Introduction BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 2 There is an ongoing public debate about the use of routinely collected healthcare data in research, 8 1 2 9 3 particularly regarding concerns over patient confidentiality. Conducting research that meets strict 10 11 4 confidentiality requirements is of paramount importance, but for public trust to be established and 12 13 5 maintained there is also a need for evidence that research using patient records provides clear 14 15 6 benefits for the wider public. One potentially important and generally agreed benefit is in evaluating 16 17 18 7 the safety of drugs inFor real world peer use, though review surprisingly, there onlyis no comprehensive and systematic 19 20 8 evidence of how data from patient records is currently used in this context, with previous summaries 21 22 9 focussing largely on safety assessments resulting in marketing authorisation withdrawal or 23 24 10 suspension.3-11 25 26 27 st 28 11 Real world evidence has been defined in a number of ways. The US 21 Century Cures Act defines it 29 30 12 as “data regarding the usage, or the potential benefits or risks, of a drug derived from sources other 31 32 13 than traditional trials”. 12 An alternative definition of real world evidence, is evidence derived from 33 34 14 information collected for purposes other than research (i.e. routinely collected healthcare data such 35 36 13 37 15 as electronic healthcare records and insurance claims data). Whilst this evidence can be generated http://bmjopen.bmj.com/ 38 39 16 by (pragmatic) randomised controlled trials, currently non-interventional studies are the 40 41 17 predominant source of real-world evidence, and these are the focus of our study.13 14 42 43 44 18 Regulatory authorities increasingly require non-interventional evidence of drug effects. As a result of 45 on September 25, 2021 by guest. Protected copyright. 46 47 19 the US 21st Century Cures Act, the US Food and Drug Administration (FDA) is developing a 48 49 20 framework for the use of non-randomised “real world evidence” in the approval of new indications 50 51 21 and in post-authorisation medicinal product assessment.12 15 Similarly the European Medicines 52 53 22 Agency’s (EMA) adaptive pathway approach forms a new route of approval for medicines, blurring 54 55 56 23 the lines between pre and post-marketing data collection, it seeks to facilitate conditional approval 57 58 24 in areas of unmet need, subject to further evidence collection, particularly of non-randomised real 59 60 25 world evidence.16 EU legislation now mandates the assessment of medication effectiveness in

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 6 of 36

1 2 17 3 1 routine clinical care where warranted. The focus on using non-interventional data to evaluate the BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 expected effectiveness of medicines is relatively new; there are concerns over their validity to 7 8 3 measure causal associations, and agreed methodologies and experience are limited. 9 10 11 4 The aim of this study was to systematically assess the type of evidence used in post-authorisation 12 13 5 drug regulation by the European Medicines Agency (EMA) to give a better understanding of the 14 15 6 contribution of non-interventional evidence and routinely collected data in this setting. 16 17 18 For peer review only 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 7 of 36 BMJ Open

1 2

3 1 Methods BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 2 We identified and reviewed all EMA post-marketing authorisation referrals made for safety and/or 8 st 9 3 efficacy concerns which were evaluated by an assessment committee between 1 January 2013 and 10 11 4 30th June 2017. The EMA is the European Union (EU) agency responsible for the scientific evaluation, 12 13 5 supervision, and safety monitoring of medicines used in the EU. Its work includes the evaluation of 14 15 6 applications for marketing authorisation and the monitoring of approved medicines. We evaluated 16 17 18 7 referrals which concludedFor after peer 2012 since EUreview medicines regulation only changed that year with 19 20 8 legislation strengthening pharmacovigilance through many measures including the introduction of a 21 22 9 Pharmacovigilance Risk Assessment Committee and increased regulatory requirements.18 The 23 24 10 evaluated referrals were made in accordance with the directives of European Parliament: Article 25 26 27 11 107(i) of Directive 2001/83/EC, Article 31 of Directive 2001/83/EC, and Article 20 of Regulation No 28 29 12 726/2004 (online supplementary table 1). 30 31 32 13 When an EU member state or the European Commission has a significant concern regarding the 33 34 14 safety or efficacy of an approved medicine, a referral process is initiated. The EMA initially publishes 35 36

37 15 a notification which details the reasons for the referral. The safety and/or efficacy of the medicine is http://bmjopen.bmj.com/ 38 39 16 then assessed in depth by designated member states and subsequently evaluated by one or more of 40 41 17 the EMA committees which include the Pharmacovigilance Risk Assessment Committee, the 42 43 18 Committee for Medicinal Products for Human Use (CHMP), and the Co-ordination Group for Mutual 44

45 on September 25, 2021 by guest. Protected copyright. 46 19 Recognition and Decentralised Procedures – Human (CMDh). Finally, an assessment report is 47 48 20 published by the EMA for each referral, providing information on the recommendations made by the 49 50 21 assessment committee and the reasons for these recommendations. 51 52 53 22 Eligible referrals were identified from the EMA website. One reviewer (JPB) evaluated the 54 55 56 23 notification and assessment report of each referral using a form (available in the online 57 58 24 supplementary appendix). Information was extracted about the notification, the referral, the 59 60 25 medicinal product, the adverse events under study, and the types of evidence assessed (pre-clinical,

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 8 of 36

1 2

3 1 non-randomised trials, randomised trials, non-interventional studies, spontaneous reports and BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 systematic reviews; definitions in online supplementary appendix). In addition, the reviewer 7 8 3 assessed how different study types were used within the referral process and categorised usage 9 10 4 into: mechanism of action, pharmacokinetics/pharmacodynamics, efficacy, risk, product usage, and 11 12 5 the effectiveness of risk minimisation measures (see the online supplementary appendix for an 13 14 6 example). The referral outcome was categorised into: no change, further evidence before decision- 15 16 17 7 making, suspension or withdrawal of marketing authorisation, change to availability, and change to 18 For peer review only 19 8 product information (or a combination of these categories). 20 21 22 9 For each referral the adverse events under study were recorded and categorised into their 23 24 10 respective Medical Dictionary for Regulatory Activities (MedDRA) system organ class.19 Drugs were 25 26 20 27 11 categorised by Anatomical Therapeutic Chemical (ATC) classification system code. 28 29 30 12 Two reviewers (JPB and IJD) independently assessed the recommendations made in the assessment 31 32 13 report, and judged the extent to which non-interventional studies were both cited and contributed 33 34 14 to the recommendation made, with disagreements resolved through discussion. We aimed to 35 36

37 15 determine whether evidence from non-interventional studies, and in particular, non-interventional http://bmjopen.bmj.com/ 38 39 16 studies using routinely collected data, had an important or pivotal role in the assessment, in order to 40 41 17 determine the contribution of this type of evidence in this context. 42 43 44 18 Patient involvement 45 on September 25, 2021 by guest. Protected copyright. 46 47 48 19 No patients were involved in the development of the research question, definition of study 49 50 20 outcomes or study design. We will disseminate our study findings to patients through social media 51 52 21 and using patient groups with an interest in data. 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 9 of 36 BMJ Open

1 2

3 1 Results BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 2 Referrals 8 9 st 10 3 Sixty potentially eligible referrals were identified with a committee opinion date between 1 January 11 12 4 2013 and the 31st June 2017. Of these 60 referrals, 8 were excluded, either because they related to 13 14 5 bioequivalence (n=4) or manufacturing concerns (n=3) rather than safety/efficacy concerns, or 15 16 6 because an assessment report was not yet available as of the 31st October 2017 (n=1) (full list of 17 18 For peer review only 19 7 included referrals included in the online supplementary appendix). 20 21 22 8 The most frequent initiators of referrals were the European Commission (n=13), France (n=12), the 23 24 9 UK (n=8), Germany (n=4) and Italy (n=4). According to the referral notification and assessment 25 26 10 report, 21 of 52 referrals (40%) were made due to a combination of safety and efficacy concerns, 29 27 28 29 11 (56%) due to safety concerns only, and 2 (4%) due to efficacy concerns only. 30 31 32 12 Drug groups and adverse events 33 34 35 13 The most common drug groups defined according to ATC code were sex hormones and modulators 36 37 14 of the genital system, and analgesics (6 referrals each), followed by drugs used in diabetes, cough http://bmjopen.bmj.com/ 38 39 40 15 and cold preparations, anti-inflammatory and anti-rheumatic products, and cardiac therapies (3 41 42 16 referrals each) (online supplementary table 2). The most common body systems on which referred 43 44 17 products acted were, based on ATC code, the nervous system (n=13), the cardiovascular system

45 on September 25, 2021 by guest. Protected copyright. 46 18 (n=9), the alimentary tract and metabolism (n=8), and the genitourinary system and sex hormones 47 48 49 19 (n=8) (online supplementary table 3). 50 51 52 20 The most commonly investigated adverse events included arterial thromboembolism (n=5), venous 53 54 21 thromboembolism (n=4), hypersensitivity (n=4) and renal impairment (n=3). The most frequent 55 56 22 category of adverse events according to MedDRA system organ class were cardiac and vascular 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 10 of 36

1 2

3 1 disorders (n=16); nervous system disorders (n=15); respiratory, thoracic and mediastinal disorders BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 (n=7); and skin and subcutaneous tissue disorders (n=7) (online supplementary table 4). 7 8 9 3 Evidence usage 10 11 12 4 Evidence cited by the initial notification and the referral assessment report was categorised by type 13 14 5 (table 1). Where no notification was available (in 12 of 52 referrals) information on the evidence 15 16 6 leading to the referral was extracted from the EMA website and the assessment report. The 17 18 For peer review only 19 7 evidence leading to referral was most commonly spontaneous reports (50%, 26/52) and randomised 20 21 8 trials (42%, n=22). Assessment reports also frequently cited spontaneous reports (73%, n=38) and 22 23 9 randomised trials (92%, n=48), but frequently cited non-interventional studies (79%, n=41) too. 24 25 10 Among the 52 referrals, in the assessment report, 31 (60%) cited non-interventional studies using 26 27 28 11 pre-existing routinely collected data (e.g. electronic medical records) and 33 (63%) cited studies 29 30 12 using data collected specifically for research. Evidence was also frequently cited from non- 31 32 13 randomised trials (63%, 33/52), preclinical studies (56%, n=29) and systematic reviews of 33 34 14 randomised trials (52%, n=27). The quality of study description and referencing varied considerably 35 36

37 15 by assessment report. It was not always possible to find a corresponding study publication or to http://bmjopen.bmj.com/ 38 39 16 ascertain the design for every study mentioned in the assessment; 63% (33/52) of assessment 40 41 17 reports referred to at least one study of unclear design. 42 43 44 18 Table 2 summarises how each type of evidence contributed to different aspects of the assessments. 45 on September 25, 2021 by guest. Protected copyright. 46 47 19 The efficacy of medications was largely determined through evidence from randomised trials (cited 48 49 20 with regard to efficacy in 77% (40/52) of referrals), with non-interventional studies contributing 50 51 21 information on efficacy in 25% (13/52) of assessments. Non-interventional studies contributed to 52 53 22 the assessment of efficacy, to a limited degree, and mostly when clinical trial data was limited, such 54 55 56 23 as in a subgroup (e.g. hydroxyethyl starch in trauma patients - EMEA/H/A-107i/1376; intravenous 57 58 24 nicardipine in children and pregnant women - EMEA/H/A-31/1339), for a product developed prior to 59 60 25 current regulatory requirements (e.g. polymyxin - EMEA/H/A-31/1383), or where a clinical trial

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 11 of 36 BMJ Open

1 2

3 1 would be difficult to run due to sporadic and unpredictable need for therapy (e.g. adrenaline auto- BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 injectors - EMEA/H/A-31/1398; methysergide for cluster headache - EMEA/H/A-31/1335). 7 8 9 3 For overall risks, both randomised trials (69%, 36/52) and non-interventional studies (60%, n=31) 10 11 4 were commonly assessed, alongside evidence from spontaneous reports (71%, n=37). Product 12 13 5 usage, where assessed, was almost entirely assessed based on non-interventional evidence (27%, 14 15 6 n=14). Mechanistic evidence was largely obtained from pre-clinical sources (31%, n=16), whilst 16 17 18 7 pharmacokinetics andFor pharmacodynamics peer were review addressed through only non-randomised trials (19%, 19 20 8 n=10), randomised trials (19%, n=10) and pre-clinical studies (12%, n=6). 21 22 23 9 Investigation of product usage and misuse was almost entirely based on non-interventional data 24 25 10 (table 2). Non-interventional evidence was also cited for estimating background incidence rates of 26 27 28 11 the adverse event in the population, and for characterising the prevalence of additional risk factors 29 30 12 and effect modifiers for the outcome under study. 31 32 33 13 Role of non-interventional evidence 34 35 36 14 Over half of the assessments relied at least in part on evidence from non-interventional studies to be 37 http://bmjopen.bmj.com/ 38 39 15 able to make recommendations for regulatory action (e.g. MA suspension or change in product 40 41 16 information) (table 3). Only in 11 of 52 assessments (21%) were no non-interventional studies cited. 42 43 17 In a further 11 referrals non-interventional studies were cited, but the reports did not indicate that 44 45 18 they contributed significantly to the decision made, either because only a few pertinent non- on September 25, 2021 by guest. Protected copyright. 46 47 48 19 interventional studies were cited (n=9), or due to limitations of the non-interventional studies (n=2). 49 50 51 20 In three referrals (combined hormonal contraceptives and thromboembolism; valproate, birth 52 53 21 defects and developmental disorders (EMEA/H/A-31/1387); and Kogenate Bayer/Helixate NexGen 54 55 22 and factor VIII inhibition (EMEA/H/C/275/A20/150/ EMEA/H/C/276/A20/143) non-interventional 56 57 58 23 studies alone were the primary source of evidence. When stratified by the outcome of the 59 60 24 assessment, it appears that non-interventional evidence more often contributed to decision-making

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 12 of 36

1 2

3 1 in referrals leading to prescribing changes (64%, 27/42) than those leading to suspension (33%, BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 4/12), though only 12 assessments led to suspension or withdrawal of marketing authorisation 7 8 3 (table 3). 9 10 11 4 Non-interventional studies were used for the evaluation of safety in a subpopulation who were 12 13 5 largely or completely excluded from clinical trials, such as pregnant women. They were also used for 14 15 6 estimating the risk of rare adverse outcomes, such as venous thromboembolism with oral 16 17 18 7 contraceptives, for whichFor clinical peer trials were reviewunderpowered. Relative only to spontaneous reports, non- 19 20 8 interventional studies contributed to decision-making more when reporting was strongly influenced 21 22 9 by the media, such as with human papillomavirus (HPV) vaccines (EMEA/H/A-20/1421), and when 23 24 10 the outcome was unlikely to be picked up by case reports, such as exposure-outcome associations 25 26 27 11 with a long latency period (e.g. Caustinerf arsenical and cancer (EMEA/H/A-31/1382)). Non- 28 29 12 interventional studies using routinely collected data were mostly used in a similar way to studies 30 31 13 using data collected for research (table 2). Studies using routinely collected data were used more 32 33 14 often when the outcome was rare, whereas studies using data collected for research purposes 34 35 36 15 contributed more when the outcome was poorly recorded in clinical records (e.g. Numeta

37 http://bmjopen.bmj.com/ 38 16 G13%E/G16%E and hypermagnesemia - EMEA/H/A-107i/1373). 39 40 41 17 Referral outcomes 42 43 44 18 The majority (98%, 51/52) of referrals led to regulatory action, with the assessment committee 45 on September 25, 2021 by guest. Protected copyright. 46 47 19 recommending changes to the product information (83%, n=43) and particularly changes to the 48 49 20 warnings, posology, undesirable effects and indication sections of the Summary of Product 50 51 21 Characteristics (table 4). In 12 of 52 (23%) referrals suspension or withdrawal of marketing 52 53 22 authorisation was recommended. Only for one referral into the safety of HPV vaccines was no 54 55 56 23 change recommended. 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 13 of 36 BMJ Open

1 2

3 1 For many referrals (42%, n=22) the assessment committee required further specific studies to be BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 conducted, generally to elucidate safety, product usage and the effectiveness of risk minimisation 7 8 3 measures. From a review of the assessment reports and the EU register of post-authorisation studies 9 10 4 (EU PAS register) most of these were non-interventional studies using routinely collected data or 11 12 5 data collected for research purposes (required in 19 referrals). 13 14 15 16 17 18 For peer review only 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 14 of 36

1 2

3 1 Discussion BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 2 In this comprehensive evaluation, we have shown that a wide range of evidence sources are used to 8 9 3 aid decision making during EU drug regulatory referrals. The three types cited in the majority of 10 11 4 assessments were randomised trials, spontaneous reports and non-interventional studies. Although 12 13 5 non-interventional evidence is rarely cited in notifications leading to a referral, it is cited 14 15 6 substantially during the detailed assessment of most issues, and in a few referrals was the primary 16 17 18 7 evidence type used inFor decision-making. peer Notably, review at the end of anonly assessment when 19 20 8 recommendations were made for evidence gaps to be filled, further non-interventional evidence 21 22 9 was required more often than any other type. 23 24 25 10 Each type of evidence appears to contribute to different aspects of a drug safety/efficacy referral, 26 27 28 11 allowing for a well-rounded assessment of medication risks and benefits. Unsurprisingly, given their 29 30 12 unique inferential advantages, randomised trials are relied on more than any other evidence type to 31 32 13 provide evidence of drug efficacy. Current usage of non-interventional evidence for efficacy largely 33 34 14 occurs where clinical trial data are limited. Increasingly, however, regulators require measures of 35 36

37 15 drug effectiveness in routine clinical care, for which well-designed non-interventional studies and http://bmjopen.bmj.com/ 38 39 16 pragmatic clinical trials using routinely collected data could be highly informative.12 16 17 40 41 42 17 To assess safety issues non-interventional evidence is heavily relied on alongside randomised trials 43 44 18 and spontaneous reports. Although less frequently cited, evidence from sources such as pre-clinical 45 on September 25, 2021 by guest. Protected copyright. 46 47 19 studies is occasionally relied on to provide information about mechanisms of effect or 48 49 20 pharmacokinetics/pharmacodynamics. 50 51 52 21 Strengths and Limitations 53 54 55 22 We were able to assess almost all referrals completed between 2013 and 2017, making this the most 56 57 58 23 comprehensive summary of recent post-marketing drug regulatory decision making in Europe. The 59 60 24 assessment reports are a comprehensive summary of the evidence used in decision making,

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 15 of 36 BMJ Open

1 2

3 1 meaning we were able to determine how each type of evidence contributed to the final BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 recommendations. 7 8 9 3 We were unable to directly assess the quality and validity of individual studies included in the 10 11 4 assessments. However, by reviewing the assessment reports, we evaluated how the evidence had 12 13 5 been rated by the committees and how it had contributed to the overall decisions made. 14 15 6 Occasionally studies were mentioned in assessment reports but no reference to a publication was 16 17 18 7 given, or referencing Forwas incomplete, peer and there review was insufficient only detail for readers to determine 19 20 8 basic information such as the study design or setting. More consistent and comprehensive 21 22 9 referencing in assessment reports would increase the transparency of decision-making to the public 23 24 10 and other stakeholders. 25 26 27 28 11 Judgement about how evidence was used in an assessment is to some extent subjective and is also 29 30 12 reliant on what is recorded in each assessment report. However, close agreement was achieved 31 32 13 between the two reviewers in this study. 33 34 35 14 Previous studies of the role of different evidence types in drug regulatory decision making have 36

37 http://bmjopen.bmj.com/ 3-11 21 38 15 largely focused on marketing authorisation withdrawals/suspensions. These studies highlight 39 40 16 how the balance of evidence types has shifted over time, from heavy reliance on spontaneous 41 42 17 reports to a more comprehensive reliance on varied evidence types including non-interventional 43 44 18 studies, randomised controlled trials and meta-analyses. Over a similar time period the overall 45 on September 25, 2021 by guest. Protected copyright. 46 47 19 number of non-interventional studies conducted and published also appears to be increasing, with 48 49 20 studies of UK electronic primary care data a prime example of this trend.22 With the increase in 50 51 21 research opportunities provided by new database linkages this publication trend is likely to continue. 52 53 54 22 Unique strengths of non-interventional evidence 55 56 57 58 23 Non-interventional evidence was particularly useful for the assessment of product safety in 59 60 24 situations where evidence from randomised controlled trials was limited such as the quantification

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 16 of 36

1 2

3 1 of rare events, and the investigation of special populations (e.g. pregnant women and children). BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 Whilst other types of evidence are also useful in some of these areas, our study highlighted 7 8 3 occasions when non-interventional evidence is unique and vital for regulatory decision making. The 9 10 4 risk of developmental disability and birth defects in the offspring of women taking valproate in 11 12 5 pregnancy is a key example of this.23 This rare outcome occurring in a group largely excluded from 13 14 6 randomised trials could not have been characterised and quantified without large, well-powered 15 16 17 7 non-interventional studies. Similarly, the detailed characterisation and quantification of adverse 18 For peer review only 19 8 outcomes associated with NSAIDs and the oral contraceptive could not have been done without 20 21 9 good quality non-interventional evidence. Where media interest led to stimulated spontaneous 22 23 10 reporting, such as in the case of HPV vaccine and various adverse effects, unbiased evidence from 24 25 26 11 non-interventional settings was vital in providing reassurance of safety, enabling continued use of 27 28 12 the vaccine with no further action required. Randomised trials used to justify licensing of medicines 29 30 13 are simply too small to detect even relatively common adverse reactions. The median number of 31 32 14 patients studied on a new active substance is 1,708 for standard medicines and 438 for orphan 33 34 24 35 15 medicines in the European Union . Rare adverse reactions (such as those occurring in 1 in 500 36

37 16 patients) will not have been detected as caused by the medicine, but such rare effects can http://bmjopen.bmj.com/ 38 39 17 dramatically alter the benefit/risk balance of the medicine. 40 41 42 18 Where the EMA’s committees call for further studies to be done, they frequently require non- 43 44

45 19 interventional evidence. There is increasingly a recognition that regulatory action to minimise risks on September 25, 2021 by guest. Protected copyright. 46 47 20 needs to be followed up to determine how effective it has been.25 Almost all drug regulatory action 48 49 21 involves making changes to how medicines are used in routine clinical care, and to determine 50 51 22 whether new directives are being followed requires evidence obtained in the routine clinical care 52 53 54 23 setting. Patterns of drug usage and quantification or characterisation of adverse events following 55 56 24 regulatory action are often required; non-interventional studies will be important here, and though 57 58 25 spontaneous reports may also be useful, they are mostly unable to give quantitative information. 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 17 of 36 BMJ Open

1 2

3 1 There are three key elements required to ensure a successful future for non-interventional evidence BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 within the framework of drug regulatory science. First, there are legitimate concerns regarding the 7 8 3 use of evidence from non-interventional studies in drug regulation given the potential problems of 9 10 4 missing data and residual confounding.26 Through high quality study design, conduct and reporting 11 12 5 these issues can in many cases be resolved27. Secondly, timely evidence is needed; non- 13 14 6 interventional studies can be conducted rapidly in response to emerging issues, or to measure the 15 16 17 7 effectiveness of past regulatory action. Thirdly, the data used in non-interventional studies needs to 18 For peer review only 19 8 be of the highest standard. This includes both the quality of the data and its generalisability to the 20 21 9 population from which it comes. Data quality can be monitored and assured by data custodians.28 22 23 10 Generalisability relies on research data being drawn from a representative sample of the population. 24 25 26 11 Whether data are taken from existing medical records or newly collected for a specific study, this 27 28 12 requires the majority of patients to consent to their data to be included. For such a transaction 29 30 13 between researchers and patients to operate successfully, maintaining anonymity and 31 32 14 confidentiality is paramount. 33 34 35 36 15 Conclusions

37 http://bmjopen.bmj.com/ 38 39 16 Regulatory decision making about the safety and efficacy of medication in the European Union relies 40 41 17 on evidence obtained from a wide range of sources; most frequently from randomised trials, 42 43 18 spontaneous reports and non-interventional studies. Non-interventional evidence can be vital for 44

45 on September 25, 2021 by guest. Protected copyright. 46 19 characterising and quantifying adverse drug reactions, is often needed for monitoring the 47 48 20 effectiveness of regulatory action to minimise risks, and in certain situations will be the only 49 50 21 available evidence. 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 18 of 36

1 2

3 1 Funding: This study was funded by the Association of the British Pharmaceutical Industry’s (ABPI’s) BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 Pharmaceutical Industry Health Information Group. JPB is supported by the grant from the ABPI for 7 8 3 the study in question. KB holds a Sir Henry Dale Fellowship jointly funded by the Wellcome Trust and 9 10 4 the Royal Society. LS was supported by a Wellcome Trust senior research fellowship in clinical 11 12 5 science (098504/Z/12/Z). IJD is supported by an unrestricted grant from GlaxoSmithKline. The 13 14 6 funders had no role in study design, data collection and analysis, decision to publish, or preparation 15 16 17 7 of the manuscript. 18 For peer review only 19 20 8 Competing interests: All authors have completed the ICMJE uniform disclosure form at 21 22 9 www.icmje.org/coi_disclosure.pdf and declare: JPB had financial support from ABPI for the 23 24 10 submitted work; IJD has received a grant from the ABPI for the study in question, financial support 25 26 27 11 from GlaxoSmithKline for work unrelated to the study in question, has consulted for GlaxoSmithKline 28 29 12 and Gilead, and holds stock in GlaxoSmithKline; SE is an independent European Commission- 30 31 13 appointed expert member of EMA’s Pharmacovigilance Risk Assessment Committee; LS reports 32 33 14 personal fees from GSK outside the submitted work; KW and KB declare no potential conflicts of 34 35 36 15 interest; there are no other relationships or activities that could appear to have influenced the

37 http://bmjopen.bmj.com/ 38 16 submitted work. The views expressed in this article are personal views of the author and may not be 39 40 17 understood or quoted as being made on behalf of or reflecting the position of the European 41 42 18 Medicines Agency or one of its committees or working parties. 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 19 Author contributions: IJD conceived the study. All authors (JPB, KW, SE, KB, LS, IJD) were involved 47 48 20 substantially in the design and planning of the study. JPB and IJD undertook the data collection and 49 50 21 wrote the initial draft of the manuscript. JPB conducted the data analyses. All authors interpreted 51 52 22 the results, contributed to later drafts of the manuscript, and approved the final manuscript. 53 54 55 56 23 Data sharing: All data analysed is available publically on the European Medicines Agency 57 58 24 (https://www.ema.europa.eu/) and EU Register of Post-Authorisation Studies 59 60 25 (http://www.encepp.eu/) websites.

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 19 of 36 BMJ Open

1 2

3 1 Patient consent: Not required. BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 2 Ethics approval: Not required. 8 9 10 11 12 13 14 15 16 17 18 For peer review only 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 20 of 36

1 2

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

Page 21 of 36 BMJ Open

1 2 3 Table 2: Number and percentage of all referrals (n=52) that use each type of evidence for each purpose 4 5 6 Type of evidence Usagea 7 Effectiveness 8 of risk 9 Risk - Risk - Usage of minimisation 10 Mechanism PK/PDb Efficacy Overall subgroup product measures 11 Pre-clinical evidence 16 (31%) 6 (12%) 2 (4%) 10 (19%) 1 (2%) 0 (0%) 0 (0%) 12 For peer review only 13 Non - randomised trials 1 (2%) 10 (19%) 18 (35%) 14 (27%) 2 (4%) 0 (0%) 0 (0%) 14 15 Randomised trials 3 (6%) 9 (17%) 40 (77%) 36 (69%) 7 (13%) 0 (0%) 1 (2%) 16 Non-interventional 3 (6%) 4 (8%) 18 (35%) 31 (60%) 5 (10%) http://bmjopen.bmj.com/ 14 (27%) 0 (0%) 17 18 Non-interventional using routinely collected data 0 (0%) 1 (2%) 8 (15%) 25 (48%) 4 (8%) 10 (19%) 0 (0%) 19 20 Non-interventional using data collected for research 2 (4%) 4 (8%) 13 (25%) 20 (38%) 3 (6%) 7 (13%) 0 (0%) 21 Spontaneous reports 2 (4%) 0 (0%) 3 (6%) 37 (71%) 6 (12%) 4 (8%) 0 (0%) 22 23 Systematic review of randomised trials 0 (0%) 0 (0%) 19 (37%) 10 (19%) 1 (2%) 0 (0%) 0 (0%) 24 on September 25, 2021 by guest. Protected copyright. 25 Systematic review of non-interventional studies 0 (0%) 0 (0%) 0 (0%) 4 (8%) 1 (2%) 0 (0%) 0 (0%) 26 27 Systematic review of randomised trials & non- 0 (0%) 1 (2%) 2 (4%) 4 (8%) 0 (0%) 0 (0%) 0 (0%) 28 interventional studies 29 Unclear study design 1 (2%) 8 (15%) 12 (23%) 10 (19%) 0 (0%) 1 (2%) 0 (0%) 30 31 32 Legend 33 Percentage of referrals that use Colour 34 evidence type for each purpose 35 <10% 36 10-19% 37 20-29% 38 30-39% 39 40%+ 40 41 42 43 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

BMJ Open Page 22 of 36

1 2 3 4 a. Usage was categorised, as detailed in the table, into: mechanism of adverse event with product usage, pharmacokinetics/pharmacodynamics of product, efficacy of 5 product, risk of adverse events with product, risk of adverse events with product in a subpopulation, usage/misuse of a product, and effectiveness of regulatory risk 6 minimisation measures. 7 b. Pharmacokinetics/pharmacodynamics 8 9 10 11 12 For peer review only 13 14 15 16 http://bmjopen.bmj.com/ 17 18 19 20 21 22 23 24 on September 25, 2021 by guest. Protected copyright. 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

Page 23 of 36 BMJ Open

1 2 3 Table 3: Usage of non-interventional studies in referral assessment reports 4 5 a 6 Usage of non-interventional studies All referrals (n=52) Referrals leading to MA Referrals leading to changes to 7 withdrawal/suspension product information (n=43) 8 (n=12) 9 Number of % of all Number of % of all Number of % of all 10 referrals referrals referrals referrals referrals referrals 11 No evidence from non-interventional studies was cited in the report 11 21% 4 33% 7 16% 12 For peer review only 13 14 Evidence from non-interventional studies was cited, but made little to 11 21% 4 33% 9 21% 15 no contribution to the decision 16 http://bmjopen.bmj.com/ 17 The decision was consistent with evidence from non-interventional 27 52% 4 33% 24 56% 18 studies, and also consistent with other evidence 19 20 The decision was consistent with evidence from non-interventional 3 6% 0 0% 3 7% 21 studies AND this evidence was the primary or only factor involved in the 22 decision e.g. there was some spontaneous reports and some large non- 23 interventional studies 24 a. Marketing authorisation on September 25, 2021 by guest. Protected copyright. 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open Page 24 of 36

1 2

3 BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from Table 4: Recommendations made as a result of assessment for the 52 included referrals 4 5 6 Recommendation Number of % of all referrals 7 referrals 8 No change 1 2% 9 Further evidence before decision- 2 4% 10 making 11 Suspension or withdrawal of 12 23% 12 marketing authorisation 13 Change to availability 0 0% 14 15 Change to product information 43 83% 16 By section of the Summary of 17 Product Characteristics: 18 - Indication For peer24 review46% only 19 20 - Posology 28 54% 21 - Contraindications 22 42% 22 - Warnings 39 75% 23 24 - Interactions 14 27% 25 - Pregnancy 10 19% 26 27 - Driving/machinery 2 4% 28 - Undesirable effects 26 50% 29 30 - Overdose 3 6% 31 - Studies 13 25% 32 - Nature and contents 3 6% 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 25 of 36 BMJ Open

1 2

37 2016;14(1):28. doi: 10.1186/s12916-016-0579-5 [published Online First: 2016/02/14] http://bmjopen.bmj.com/ 38 11. Lane S, Lynn E, Shakir S. Investigation assessing the publicly available evidence supporting 39 postmarketing withdrawals, revocations and suspensions of marketing authorisations in the 40 EU since 2012. BMJ open 2018;8(1):e019759. 41 12. 21st Century Cures Act H.R.34, 2015. 42 13. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-World Evidence - What Is It and What Can It 43 44 Tell Us? New England Journal of Medicine 2016;375(23):2293-97.

45 14. Kalkman S, van Thiel GJ, Zuidgeest MG, et al. Series: Pragmatic trials and real world evidence: on September 25, 2021 by guest. Protected copyright. 46 Paper 4. Informed consent. Journal of Clinical Epidemiology 2017;89:181-87. 47 15. Avorn J, Kesselheim ASJNEJoM. The 21st Century Cures Act—will it take us back in time? 48 2015;372(26):2473-75. 49 16. Agency EM. Final report on the adaptive pathways pilot, 2016. 50 17. Regulation (EU) No 1235/2010 of the European Parliament and of the Council. Official Journal of 51 the European Union 2010 52 53 18. Commision E. Commission Implementing Regulation (EU) No 520/2012 of 19 June 2012 on the 54 performance of pharmacovigilance activities provided for in Regulation (EC) No 726/2004 of 55 the European Parliament and of the Council and Directive 2001/83/EC of the European 56 Parliament and of the Council Official Journal of the European Union 2012 57 19. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for 58 Human Use. Medical Dictionary for Regulatory Activities [Available from: 59 https://bioportal.bioontology.org/ontologies/MEDDRA. 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 26 of 36

1 2

3 20. World Health Organization Collaborating Centre for Drug Statistics Methodology. Anatomical BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 Therapeutic Chemical Classification System [Available from: 5 6 https://www.whocc.no/atc_ddd_index/. 7 21. Ishiguro C, Hall M, Neyarapally GA, et al. Post-market drug safety evidence sources: an analysis of 8 FDA drug safety communications. 2012;21(10):1134-36. 9 22. Vezyridis P, Timmons S. Evolution of primary care databases in UK: a scientometric analysis of 10 research output. BMJ open 2016;6(10):e012785. 11 23. Valproate Art. 31 Assessment Report: European Medicines Agency, 2014. 12 24. Duijnhoven RG, Straus SM, Raine JM, et al. Number of patients studied prior to approval of new 13 medicines: a database analysis. PLoS medicine 2013;10(3):e1001407. 14 25. Commission E. Directive 2010/84/EU of the European Parliament and of the Council of 15 15 16 December 2010 amending, as regards pharmacovigilance, Directive 2001/83/EC on the 17 Community code relating to medicinal products for human use. 2010 18 26. Kesselheim AS, AvornFor J. New peer “21st Century review Cures” legislation: only speed and ease vs science. Jama 19 2017;317(6):581-82. 20 27. Goodman SN, Schneeweiss S, Baiocchi M. Using design thinking to differentiate useful from 21 misleading evidence in observational research. Jama 2017;317(7):705-07. 22 28. Herrett E, Gallagher AM, Bhaskaran K, et al. Data resource profile: clinical practice research 23 datalink (CPRD). International journal of epidemiology 2015;44(3):827-36. 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 27 of 36 BMJ Open

1 2

3 BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 Table 1: European Medicines Agency referrals categorised by type of referral procedure and date 5

6 a 7 Referral category Frequency by CHMP/CMDh opinion date Total 8 2013 2014 2015 2016 2017 Jan- 9 Jun 10 Article 107i procedures 5 1 0 0 0 6 11 Article 20 procedures 2 3 1 4 1 11 12 Article 31 referrals 13 13 5 3 1 35 13 Total 20 17 6 7 2 52 14 a) Committee for Medicinal Products for Human Use/Coordination Group for Mutual Recognition and 15 Decentralised Procedures – Human 16 17 Table 2: ATCa therapeutic subgroup of medicinal product by frequency for the 52 included referrals 18 ATC subgroup SubgroupFor definition peer review onlyNo. of 19 code referrals 20 G03 Sex hormones and modulators of the genital system 6 21 N02 Analgesics 6 22 A10 Drugs used in diabetes 3 23 C01 Cardiac therapy 3 24 25 M01 Anti-inflammatory and antirheumatic products 3 26 None Not applicable/available 3 27 R05 Cough and cold preparations 3 28 A03 Drugs for functional gastrointestinal disorders 2 29 B05 Blood substitutes and perfusion solutions 2 30 C04 Peripheral vasodilators 2 31 C10 Lipid modifying agents 2 32 G02 Other gynelogicals 2 33 L01 Antineoplastic agents 2 34 M03 Muscle relaxants 2 35 N05 Psycholeptics 2 36 R03 Drugs for obstructive airway diseases 2 37 B01 Antithrombotic agents 1 http://bmjopen.bmj.com/ 38 B02 Antihemorrhagics 1 39 B03 Antianemic preparations 1 40 C08 Calcium channel blockers 1 41 42 C09 Agents acting on the renin-angiotensin system 1 43 J01 Antibacterials for systemic use 1 44 J02 Antimycotics for systemic use 1

45 J05 Antivirals for systemic use 1 on September 25, 2021 by guest. Protected copyright. 46 L04 Immunosuppresants 1 47 M05 Drugs for treatment of bone diseases 1 48 N03 Antiepileptics 1 49 N07 Psychoanaleptics 1 50 R02 Throat preparations 1 51 R07 Other respiratory system products 1 52 a) Anatomical Therapeutic Chemical (ATC) Classification System 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 28 of 36

1 2 a 3 Table 3: ATC section of medicinal product by frequency for the 52 included referrals BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 Number 6 of 7 ATC section Section definition referrals 8 9 N Nervous system 13 10 C Cardiovascular system 9 11 A Alimentary tract and metabolism 8 12 G Genitourinary system and sex hormones 8 13 R Respiratory system 7 14 M Musculoskeletal system 6 15 B Blood and blood forming organs 5 16 J Anti-infectives for systemic use 3 17 L Antineoplastic and immunomodulating agents 3 18 D DermatologicalsFor peer review only0 19 Systemic hormonal preparations, excluding sex 20 H hormones and insulins 0 21 P Antiparasitic products, insecticides and repellents 0 22 S Sensory organs 0 23 V Various 0 24 a) Anatomical Therapeutic Chemical (ATC) Classification System 25

26 27 a 28 Table 4: MedDRA system organ class (SOC) of adverse event by frequency for the 52 included 29 referrals 30 31 System Organ Class (SOC) Number 32 of 33 referrals 34 Cardiac disorders 16 35 Vascular disorders 15 36

37 Nervous system disorders 9 http://bmjopen.bmj.com/ 38 Respiratory, thoracic and mediastinal disorders 7 39 Skin and subcutaneous tissue disorders 7 40 41 Gastrointestinal disorders 6 42 Immune system disorders 5 43 Infections and infestations 5 44 Renal and urinary disorders 5 45 on September 25, 2021 by guest. Protected copyright. 46 Blood and lymphatic system disorders 4 47 Hepatobiliary disorders 4 48 Metabolism and nutrition disorders 4 49 50 Neoplasms benign, malignant and unspecified 4 51 Congenital, familial and genetic disorders 2 52 Endocrine disorders 2 53 Musculoskeletal and connective tissue disorders 2 54 55 Injury, poisoning and procedural complications 1 56 Reproductive system and breast disorders 1 57 Sexual function and fertility disorders 1 58 Surgical and Medical Procedures 1 59 60 a) Medical Dictionary for Regulatory Activities

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 29 of 36 BMJ Open

1 2

3 European Medicines Agency referrals included in the study BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 a CHMPb Referral Title 6 EMA Reference No. opinion/CMDhc 7 position date 8 9 EMEA/H/C/889/A20/37 17/01/2013 Tredaptive, Pelzont and Trevaclyn 10 EMEA/H/C/903/A20/38 11 EMEA/H/C/897/A20/38 12 EMEA/H/A-31/1306 21/03/2013 Cilostazol-containing medicines 13 EMEA/H/A107i/1352 24/04/2013 Tetrazepam-containing medicines 14 EMEA/H/A-107i/1357 29/05/2013 Cyproterone and ethinylestradiol containing 15 medicinal products 16 EMEA/H/A-31/1346 29/05/2013 Almitrine-containing medicines 17 EMEA/H/A-107i/1363 26/06/2013 Flupirtine-containing medicines 18 EMEA/H/A-31/1342 For peer26/06/2013 review Codeine-containing only medicines 19 EMEA/H/A-31/1344 26/06/2013 Diclofenac-containing medicines 20 EMEA/H/A-31/1325 27/06/2013 Ergot derivatives 21 EMEA/H/A-31/1322 27/06/2013 Intravenous iron-containing medicinal products 22 EMEA/H/A-31/1314 25/07/2013 Ketoconazole-containing medicines 23 EMEA/H/A-107i/1373 18/09/2013 Numeta G13E and Numeta G16E emulsion for 24 infusion 25 EMEA/H/A-107i/1376 23/10/2013 Hydroxyethyl starch solutions for infusion 26 EMEA/H/A-31/1348 23/10/2013 Hydroxyethyl starch solutions for infusion 27 EMEA/H/A-31/1347 23/10/2013 Short-acting beta-agonists 28 EMEA/H/A-31/1339 24/10/2013 Intravenous nicardipine medicines 29 30 EMEA/H/A-31/1321 24/10/2013 Metoclopramide-containing medicines 31 EMA/H/A-31/1361 21/11/2013 Thiocolchicoside-containing medicines 32 EMEA/H/A-31/1366 18/12/2013 Substances related to nicotinic acid 33 EMEA/H/C/275/A20/150 19/12/2013 Kogenate Bayer and Helixate NexGen 34 EMEA/H/C/276/A20/143 35 EMEA/H/A-31/1356 16/01/2014 Combined hormonal contraceptives 36 EMEA/H/A20/1371/C/00560- 20/02/2014 Protelos and Osseor

37 561/0039-0034 http://bmjopen.bmj.com/ 38 EMEA/H/A-31/1335 20/02/2014 Methysergide-containing medicines 39 EMEA/H/A-31/1349 19/03/2014 Diacerein-containing medicines for oral 40 administration 41 EMEA/H/A-31/1365 24/04/2014 Domperidone-containing medicines 42 EMEA/H/A-31/1377 24/04/2014 Zolpidem-containing medicines 43 EMEA/H/A-31/1382 25/04/2014 Caustinerf arsenical and Yranicid arsenical 44 EMEA/H/A-31/1336 25/04/2014 Linoladiol N and Linoladiol HN 45 EMEA/H/A-31/1370 22/05/2014 Renin-angiotensin-system (RAS)-acting agents on September 25, 2021 by guest. Protected copyright. 46 EMEA/H/A-107i/1395 23/07/2014 Methadone medicinal products for oral use 47 containing povidone 48 EMEA/H/A-31/1391 24/07/2014 Emergency contraceptives 49 EMEA/H/A-31/1379 20/08/2014 Bromocriptine-containing medicines indicated in 50 51 the prevention or suppression of physiological 52 lactation post-partum 53 EMEA/H/C/2695/A20/0003 23/10/2014 Iclusig 54 EMEA/H/A-31/1383 23/10/2014 Polymyxin-containing medicines 55 EMEA/H/A-31/1396 19/11/2014 Testosterone-containing medicines 56 EMEA/H/A-31/1387 19/11/2014 Valproate and related substances 57 EMEA/H/A20/1404/C/000598/0031 20/11/2014 Corlentor and Procoralan 58 EMEA/H/A20/1404/C/000597/0032 59 EMEA/H/A-31/1400 25/03/2015 Hydroxyzine-containing medicinal products 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 30 of 36

1 2

3 EMEA/H/A-31/1394 22/04/2015 Codeine-containing medicinal products for the BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 treatment of cough or cold in paediatric patients 5 EMEA/H/A-31/1401 20/05/2015 Ibuprofen- and dexibuprofen-containing medicines 6 EMEA/H/A-31/1398 25/06/2015 Adrenaline auto-injectors 7 EMEA/H/A-31/1397 18/11/2015 Ambroxol and bromhexine-containing medicines 8 EMEA/H/A-20/1421 19/11/2015 Human papillomavirus vaccines 9 10 EMEA/H/A-20/1419 25/02/2016 SGLT2 (sodium-glucose co-transporter 2) inhibitors 11 EMEA/H/A- 25/02/2016 Tysabri 12 20/1416/C/000603/0083 13 EMEA/H/A-31/1420 31/03/2016 Fusafungine containing medicinal products for 14 oromucosal and nasal use 15 EMEA/H/A-31/1415 28/04/2016 Inhaled corticosteroids containing medicinal 16 products indicated in the treatment of chronic 17 obstructive pulmonary disease 18 EMEA/H/A-20/1439/C/3843/0023For peer 21/07/2016 review Zydelig only 19 EMEA/H/A-31/1432 13/10/2016 Metformin and metformin-containing medicines 20 EMEA/H/A-20/1438 15/12/2016 Direct-acting antivirals indicated for treatment of 21 hepatitis C (interferon-free) 22 EMEA/H/A-31/1435 26/01/2017 Dienogest/ethinylestradiol-containing medicinal 23 products indicated in acne 24 EMEA/H/A-20/1442 23/02/2017 SGLT2 (sodium-glucose co-transporter 2) inhibitors 25 (previously canagliflozin) 26 a) European Medicines Agency 27 b) Committee for Medicinal Products for Human Use 28 c) Coordination Group for Mutual Recognition and Decentralised Procedures – Human 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 31 of 36 BMJ Open

1 2

3 Definition of key terms in study BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 Study type Definition 5 Pre-clinical evidence Evidence from in-vitro and in-vivo (non-human 6 animals) experimentation. 7 Non-randomised trials Interventional studies where assignment to therapy 8 was not at random or where there was only one trial 9 arm (e.g. Phase 1 and Phase 2 trials). 10 Randomised trials Interventional studies where assignment to therapy 11 versus control was random (including both 12 traditional multi-arm randomised controlled trials 13 and randomised crossover trials). 14 15 Interventional studies Clinical studies where the study investigators 16 intervene on patient therapy. 17 Non-interventional studies Clinical studies where there is no intervention by 18 For peer reviewstudy investigators. only Alternatively termed 19 observational studies. 20 Spontaneous reports Unsolicited reports of adverse outcomes reported 21 by consumers or healthcare professionals. 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 32 of 36

1 2

3 Primary Data collection form BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5

6 1. Basic information about EMAa reference number 7 referral 8 b 9 Initiated by (e.g. MHRA , European 10 Commission) 11 Referral/procedure type (Article 107i/Article 12 31/Article 20) 13 Decision making model (e.g. PRAC- c 14 CMDh-EC ) 15 Cause of referral (Safety/Efficacy/Safety and 16 efficacy) 17 Cause of referral – description 18 For peer review only 19 CHMPd opinion/CMDhe position date 20 21 2. Information about product Review title 22 and adverse event 23 Substance name 24 25 26 Product usage 27 28 f 29 ATC group (e.g. N03 - Antiepileptics) 30 31 Product class (as listed on EMA website) 32 33 34 Adverse events 35 36 MedDRAg system organ classes of 37 adverse events http://bmjopen.bmj.com/ 38 39 3. Determine the types of Source of evidence (Notification/Assessment 40 evidence leading to the referral report/EMA webpage) 41 a. Pre-clinical evidence (Yes/No/Unclear) 42 43 44 b. Non-randomised trials (Yes/No/Unclear)

45 on September 25, 2021 by guest. Protected copyright. 46 c. Randomised trials (Yes/No/Unclear) 47 48 d. Non-interventional studies (Yes/No/Unclear) 49 50 51 i. Using routinely collected real world (Yes/No/Unclear) 52 data e.g. electronic health records 53 ii. Using primary data collection e.g. (Yes/No/Unclear) 54 pregnancy registry 55 56 e. Spontaneous reports (Yes/No/Unclear) 57 58 f. Systematic review of randomised trials (Yes/No/Unclear) 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 33 of 36 BMJ Open

1 2

3 g. Systematic review of non- (Yes/No/Unclear) BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 interventional studies 5 6 h. Systematic review combining (Yes/No/Unclear) 7 randomised trials & non-interventional 8 studies 9 i. Unclear design (Yes/No) 10 11 4. a) Determine the types of a. Pre-clinical evidence (Yes/No/Unclear) 12 evidence used in each 13 assessment report b. Non-randomised trials (Yes/No/Unclear) 14 15 16 c. Randomised trials (Yes/No/Unclear) 17 18 For peerd. Non-interventional review studies only(Yes/No/Unclear) 19 20 i. Using routinely collected real world (Yes/No/Unclear) 21 data e.g. electronic health records 22 23 ii. Using primary data collection e.g. (Yes/No/Unclear) 24 pregnancy registry 25 26 e. Spontaneous reports (Yes/No/Unclear) 27 28 f. Systematic review of randomised trials (Yes/No/Unclear) 29 30 g. Systematic review of non- (Yes/No/Unclear) 31 interventional studies 32 33 h. Systematic review combining (Yes/No/Unclear) 34 randomised trials & non-interventional 35 studies 36 i. Unclear design (Yes/No)

37 http://bmjopen.bmj.com/ 38 4. b) Summarise the types of a. Pre-clinical evidence 39 evidence used in each 40 assessment report b. Non - randomised trials 41 42 43 c. Randomised trials 44

45 on September 25, 2021 by guest. Protected copyright. d. Non-interventional studies 46

48 e. Spontaneous reports

49 50 f. Systematic review of randomised trials 51 52 53 g. Systematic review of non- 54 interventional studies 55 h. Systematic review combining 56 randomised trials & non-interventional 57 studies 58 i. Unclear design 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 34 of 36

1 2

26 use 27 4.5 Interactions with other medicinal 28 products and other forms of interaction 29 4.6 Fertility, pregnancy and lactation 30 31 4.7 Effects on ability to drive and use 32 machines 33 4.8 Undesirable effects 34 4.9 Overdose 35 36 Other 37 http://bmjopen.bmj.com/ 38 7. Determine how non- (a. No evidence from non-interventional studies was cited in the 39 interventional studies report/ 40 contributed to the decision b. Evidence from non-interventional studies was cited, but made little 41 made. Judgement is involved in to no contribution to the decision/ 42 this step and the assessment will c. Evidence from non-interventional studies was cited, but the decision 43 be conducted independently by was contrary to this evidence/ 44 two researchers. d. The decision was consistent with evidence from non-interventional

45 studies, and also consistent with other evidence/ on September 25, 2021 by guest. Protected copyright. 46 e. The decision was consistent with evidence from non-interventional 47 studies AND this evidence was the primary or only factor involved in 48 the decision/ 49 f. Unclear) 50 8. What was useful (or 51 otherwise) about the evidence 52 from non-interventional studies? 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 35 of 36 BMJ Open

1 2

3 9. If no non-interventional Yes/no? (Yes/No/Unclear) BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 studies were available, were Further information 5 such studies feasible and could 6 they have been useful? 7 8 9 10. Does the action taken as a Yes/no? (Yes/No/Unclear) 10 result of the referral require 11 future research? Is further non-interventional evidence (Yes/No/Unclear) 12 required? 13 Further information 14 15 16 17 18 For peer review only 19 20 Design of further non-interventional (using data collected for 21 studies in PAS register research/ using routinely 22 collected data) 23

24 25 a) European Medicines Agency 26 b) Medicines and Healthcare Regulatory Agency 27 c) Pharmacovigilance Risk Assessment Committee – Coordination Group for Mutual Recognition and 28 Decentralised Procedures (Human) - European Commission 29 d) Committee for Medicinal Products for Human Use 30 e) Coordination Group for Mutual Recognition and Decentralised Procedures – Human 31 f) Anatomical Therapeutic Chemical (ATC) Classification System 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

BMJ Open Page 36 of 36

1 2 3 4 5 Secondary data collection form - Example 6 7 EMEA/H/A- Pre- Non - Randomised NIa NI NI studies Spontaneous Systematic Systematic Systematic Unclear 8 107i/1395 clinical randomised trials studies studies using reports review of review of review study design 9 evidence trials using primary randomised NI studies combining 10 RCDb data trials randomised 11 collection trials & NI 12 For peer review only studies 13 c 14 Mechanism of AE No No No No No No Yes No No No No 15 with product 16 Pharmacokinetics/ Yes Yes No No No No No No No No Yes http://bmjopen.bmj.com/ 17 Pharmacodynamics 18 Efficacy No No Yes No No No No Yes No No No 19 Risk - Overall Yes No No No No No Yes No No No Yes 20 Risk - No No No No No No No No No No No 21 Subpopulation 22 Usage of product No No No Yes Yes Yes No No No No No 23 24 Effectiveness of risk No No No No No No No No No No No on September 25, 2021 by guest. Protected copyright. 25 minimisation 26 a. Non-interventional studies 27 b. Routinely collected data 28 c. Adverse event 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

Article Type: Research

Date Submitted by the 15-Jul-2019 Author:

Primary Subject Pharmacology and therapeutics Heading:

Secondary Subject Heading: Public health http://bmjopen.bmj.com/

Keywords: real world evidence, non-interventional studies, medicines regulation

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 37 BMJ Open

1 2

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 2 of 37

1 2

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 3 of 37 BMJ Open

1 2

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 4 of 37

1 2

37 14 close agreement between two independent reviewers was observed. http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 5 of 37 BMJ Open

1 2

3 1 Introduction BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 2 There is an ongoing public debate about the use of routinely collected healthcare data in research, 8 1 2 9 3 particularly regarding concerns over patient confidentiality. Conducting research that meets strict 10 11 4 confidentiality requirements is of paramount importance, but for public trust to be established and 12 13 5 maintained there is also a need for evidence that research using patient records provides clear 14 15 6 benefits for the wider public. One potentially important and generally agreed benefit is in evaluating 16 17 18 7 the safety of drugs inFor real world peer use, though review surprisingly, there onlyis no comprehensive and systematic 19 20 8 evidence of how data from patient records is currently used in this context, with previous summaries 21 22 9 focussing largely on safety assessments resulting in marketing authorisation withdrawal or 23 24 10 suspension.3-14 25 26 27 st 28 11 Real world evidence has been defined in a number of ways. The US 21 Century Cures Act defines it 29 30 12 as “data regarding the usage, or the potential benefits or risks, of a drug derived from sources other 31 32 13 than traditional trials”. 15 An alternative definition of real world evidence, is evidence derived from 33 34 14 information collected for purposes other than research (i.e. routinely collected healthcare data such 35 36 16 37 15 as electronic healthcare records and insurance claims data). Whilst this evidence can be generated http://bmjopen.bmj.com/ 38 39 16 by (pragmatic) randomised controlled trials, currently non-interventional studies are the 40 41 17 predominant source of real-world evidence, and these are the focus of our study.16 17 42 43 44 18 Regulatory authorities increasingly require non-interventional evidence of drug effects. As a result of 45 on September 25, 2021 by guest. Protected copyright. 46 47 19 the US 21st Century Cures Act, the US Food and Drug Administration (FDA) is developing a 48 49 20 framework for the use of non-randomised “real world evidence” in the approval of new indications 50 51 21 and in post-authorisation medicinal product assessment.15 18 Similarly the European Medicines 52 53 22 Agency’s (EMA) adaptive pathway approach forms a new route of approval for medicines, blurring 54 55 56 23 the lines between pre and post-marketing data collection, it seeks to facilitate conditional approval 57 58 24 in areas of unmet need, subject to further evidence collection, particularly of non-randomised real 59 60 25 world evidence.19 EU legislation now mandates the assessment of medication effectiveness in

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 6 of 37

1 2 20 3 1 routine clinical care where warranted. The focus on using non-interventional data to evaluate the BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 expected effectiveness of medicines is relatively new; there are concerns over their validity to 7 8 3 measure causal associations, and agreed methodologies and experience are limited. 9 10 11 4 The aim of this study was to systematically assess the type of evidence used in post-authorisation 12 13 5 drug regulation by the European Medicines Agency (EMA) to give a better understanding of the 14 15 6 contribution of non-interventional evidence and routinely collected data in this setting. 16 17 18 For peer review only 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 7 of 37 BMJ Open

1 2

3 1 Methods BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 7 2 We identified and reviewed all EMA post-marketing authorisation referrals made for safety and/or 8 st 9 3 efficacy concerns which were evaluated by an assessment committee between 1 January 2013 and 10 11 4 30th June 2017. The EMA is the European Union (EU) agency responsible for the scientific evaluation, 12 13 5 supervision, and safety monitoring of medicines used in the EU. Its work includes the evaluation of 14 15 6 applications for marketing authorisation and the monitoring of approved medicines. We evaluated 16 17 18 7 referrals which concludedFor after peer 2012 since EUreview medicines regulation only changed that year with 19 20 8 legislation strengthening pharmacovigilance through many measures including the introduction of a 21 22 9 Pharmacovigilance Risk Assessment Committee and increased regulatory requirements.21 The 23 24 10 evaluated referrals were made in accordance with the directives of European Parliament: Article 25 26 27 11 107(i) of Directive 2001/83/EC, Article 31 of Directive 2001/83/EC, and Article 20 of Regulation No 28 29 12 726/2004 (online supplementary table 1). 30 31 32 13 When an EU member state or the European Commission has a significant concern regarding the 33 34 14 safety or efficacy of an approved medicine, a referral process is initiated. The EMA initially publishes 35 36

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 8 of 37

1 2

3 1 non-randomised trials, randomised trials, non-interventional studies, spontaneous reports and BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 systematic reviews; definitions in online supplementary appendix). In addition, the reviewer 7 8 3 assessed how different study types were used within the referral process and categorised usage 9 10 4 into: mechanism of action, pharmacokinetics/pharmacodynamics, efficacy, risk, product usage, and 11 12 5 the effectiveness of risk minimisation measures (see the online supplementary appendix for an 13 14 6 example). The referral outcome was categorised into: no change, further evidence before decision- 15 16 17 7 making, suspension or withdrawal of marketing authorisation, change to availability, and change to 18 For peer review only 19 8 product information (or a combination of these categories). 20 21 22 9 For each referral the adverse events under study were recorded and categorised into their 23 24 10 respective Medical Dictionary for Regulatory Activities (MedDRA) system organ class.22 Drugs were 25 26 23 27 11 categorised by Anatomical Therapeutic Chemical (ATC) classification system code. 28 29 30 12 Two reviewers (JPB and IJD) independently assessed the recommendations made in the assessment 31 32 13 report, and judged the extent to which non-interventional studies were both cited and contributed 33 34 14 to the recommendation made, with disagreements resolved through discussion. We aimed to 35 36

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 9 of 37 BMJ Open

1 2

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 10 of 37

1 2

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 11 of 37 BMJ Open

1 2

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 12 of 37

1 2

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 13 of 37 BMJ Open

1 2

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 14 of 37

1 2

37 15 drug effectiveness in routine clinical care, for which well-designed non-interventional studies and http://bmjopen.bmj.com/ 38 39 16 pragmatic clinical trials using routinely collected data could be highly informative.15 19 20 40 41 42 17 To assess safety issues non-interventional evidence is heavily relied on alongside randomised trials 43 44 18 and spontaneous reports. Although less frequently cited, evidence from sources such as pre-clinical 45 on September 25, 2021 by guest. Protected copyright. 46 47 19 studies is occasionally relied on to provide information about mechanisms of effect or 48 49 20 pharmacokinetics/pharmacodynamics. 50 51 52 21 Real world evidence can be generated from trials, such as from pragmatic trials conducted using 53 54 22 routinely collected data. We did not identify any such trials in the assessment reports. This study 55 56 57 23 design could, however, be of considerable utility given the potential for increased generalisability 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 15 of 37 BMJ Open

1 2

3 1 relative to traditional trials, and the minimisation of confounding, through randomisation, relative to BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4

5 24 6 2 non-interventional studies. 7 8 9 3 Strengths and Limitations 10 11 12 4 We were able to assess almost all referrals completed between 2013 and 2017, making this the most 13 14 5 comprehensive summary of recent post-marketing drug regulatory decision making in Europe. The 15 16 6 assessment reports are a comprehensive summary of the evidence used in decision making, 17 18 For peer review only 19 7 meaning we were able to determine how each type of evidence contributed to the final 20 21 8 recommendations. 22 23 24 9 We were unable to directly assess the quality and validity of individual studies included in the 25 26 10 assessments. However, by reviewing the assessment reports, we evaluated how the evidence had 27 28 29 11 been rated by the committees and how it had contributed to the overall decisions made. 30 31 12 Occasionally studies were mentioned in assessment reports but no reference to a publication was 32 33 13 given, or referencing was incomplete, and there was insufficient detail for readers to determine 34 35 14 basic information such as the study design or setting. For example, for the assessment report on 36

37 http://bmjopen.bmj.com/ 38 15 combined hormonal contraceptives (EMEA/H/A-31/1356) it was not clear whether some of the trials 39 40 16 mentioned were randomised or not. More consistent and comprehensive referencing in assessment 41 42 17 reports would increase the transparency of decision-making to the public and other stakeholders. 43 44 45 18 Judgement about how evidence was used in an assessment is to some extent subjective and is also on September 25, 2021 by guest. Protected copyright. 46 47 48 19 reliant on what is recorded in each assessment report. However, close agreement was achieved 49 50 20 between the two reviewers in this study. 51 52 53 21 Previous studies of the role of different evidence types in drug regulatory decision making have 54 55 22 largely focused on marketing authorisation withdrawals/suspensions.3-11 25 These studies highlight 56 57 58 23 how the balance of evidence types has shifted over time, from heavy reliance on spontaneous 59 60 24 reports to a more comprehensive reliance on varied evidence types including non-interventional

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 16 of 37

1 2

3 1 studies, randomised controlled trials and meta-analyses. Over a similar time period the overall BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 number of non-interventional studies conducted and published also appears to be increasing, with 7 26 8 3 studies of UK electronic primary care data a prime example of this trend. With the increase in 9 10 4 research opportunities provided by new database linkages this publication trend is likely to continue. 11 12 13 5 Unique strengths of non-interventional evidence 14 15 16 6 Non-interventional evidence was particularly useful for the assessment of product safety in 17 18 For peer review only 19 7 situations where evidence from randomised controlled trials was limited such as the quantification 20 21 8 of rare events, and the investigation of special populations (e.g. pregnant women and children). 22 23 9 Whilst other types of evidence are also useful in some of these areas, our study highlighted 24 25 10 occasions when non-interventional evidence is unique and vital for regulatory decision making. The 26 27 28 11 risk of developmental disability and birth defects in the offspring of women taking valproate in 29 30 12 pregnancy is a key example of this.27 This rare outcome occurring in a group largely excluded from 31 32 13 randomised trials could not have been characterised and quantified without large, well-powered 33 34 14 non-interventional studies. Similarly, the detailed characterisation and quantification of adverse 35 36

37 15 outcomes associated with NSAIDs and the oral contraceptive could not have been done without http://bmjopen.bmj.com/ 38 39 16 good quality non-interventional evidence. Where media interest led to stimulated spontaneous 40 41 17 reporting, such as in the case of HPV vaccine and various adverse effects, unbiased evidence from 42 43 18 non-interventional settings was vital in providing reassurance of safety, enabling continued use of 44

45 on September 25, 2021 by guest. Protected copyright. 46 19 the vaccine with no further action required. Randomised trials used to justify licensing of medicines 47 48 20 are simply too small to detect even relatively common adverse reactions. The median number of 49 50 21 patients studied on a new active substance is 1,708 for standard medicines and 438 for orphan 51 52 22 medicines in the European Union28. Rare adverse reactions (such as those occurring in 1 in 500 53 54 55 23 patients) will not have been detected as caused by the medicine, but such rare effects can 56 57 24 dramatically alter the benefit/risk balance of the medicine. 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 17 of 37 BMJ Open

1 2

3 1 Where the EMA’s committees call for further studies to be done, they frequently require non- BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 interventional evidence. There is increasingly a recognition that regulatory action to minimise risks 7 29 8 3 needs to be followed up to determine how effective it has been. Almost all drug regulatory action 9 10 4 involves making changes to how medicines are used in routine clinical care, and to determine 11 12 5 whether new directives are being followed requires evidence obtained in the routine clinical care 13 14 6 setting. Patterns of drug usage and quantification or characterisation of adverse events following 15 16 17 7 regulatory action are often required; non-interventional studies will be important here, and though 18 For peer review only 19 8 spontaneous reports may also be useful, they are mostly unable to give quantitative information. 20 21 22 9 There are three key elements required to ensure a successful future for non-interventional evidence 23 24 10 within the framework of drug regulatory science. First, there are legitimate concerns regarding the 25 26 27 11 use of evidence from non-interventional studies in drug regulation given the potential problems of 28 29 12 missing data and residual confounding.30 Through high quality study design, conduct and reporting 30 31 13 these issues can in many cases be resolved31. Secondly, timely evidence is needed; non- 32 33 14 interventional studies can be conducted rapidly in response to emerging issues, or to measure the 34 35 36 15 effectiveness of past regulatory action. Thirdly, the data used in non-interventional studies needs to

37 http://bmjopen.bmj.com/ 38 16 be of the highest standard. This includes both the quality of the data and its generalisability to the 39 40 17 population from which it comes. Data quality can be monitored and assured by data custodians.32 41 42 18 Generalisability relies on research data being drawn from a representative sample of the population. 43 44

45 19 Whether data are taken from existing medical records or newly collected for a specific study, this on September 25, 2021 by guest. Protected copyright. 46 47 20 requires the majority of patients to consent to their data to be included. For such a transaction 48 49 21 between researchers and patients to operate successfully, maintaining anonymity and 50 51 22 confidentiality is paramount. 52 53 54 55 23 Conclusions 56 57 58 24 Regulatory decision making about the safety and efficacy of medication in the European Union relies 59 60 25 on evidence obtained from a wide range of sources; most frequently from randomised trials,

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 18 of 37

1 2

3 1 spontaneous reports and non-interventional studies. Non-interventional evidence can be vital for BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 characterising and quantifying adverse drug reactions, is often needed for monitoring the 7 8 3 effectiveness of regulatory action to minimise risks, and in certain situations will be the only 9 10 4 available evidence. 11 12 13 14 15 16 17 18 For peer review only 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 19 of 37 BMJ Open

1 2

3 1 Funding: This study was funded by the Association of the British Pharmaceutical Industry’s (ABPI’s) BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 6 2 Pharmaceutical Industry Health Information Group. JPB is supported by the grant from the ABPI for 7 8 3 the study in question. KB holds a Sir Henry Dale Fellowship jointly funded by the Wellcome Trust and 9 10 4 the Royal Society. LS was supported by a Wellcome Trust senior research fellowship in clinical 11 12 5 science (098504/Z/12/Z). IJD is supported by an unrestricted grant from GlaxoSmithKline. The 13 14 6 funders had no role in study design, data collection and analysis, decision to publish, or preparation 15 16 17 7 of the manuscript. 18 For peer review only 19 20 8 Competing interests: All authors have completed the ICMJE uniform disclosure form at 21 22 9 www.icmje.org/coi_disclosure.pdf and declare: JPB had financial support from ABPI for the 23 24 10 submitted work; IJD has received a grant from the ABPI for the study in question, financial support 25 26 27 11 from GlaxoSmithKline for work unrelated to the study in question, has consulted for GlaxoSmithKline 28 29 12 and Gilead, and holds stock in GlaxoSmithKline; SE is an independent European Commission- 30 31 13 appointed expert member of EMA’s Pharmacovigilance Risk Assessment Committee; LS reports 32 33 14 personal fees from GSK outside the submitted work; there are no other relationships or activities 34 35 36 15 that could appear to have influenced the submitted work. The views expressed in this article are

37 http://bmjopen.bmj.com/ 38 16 personal views of the author and may not be understood or quoted as being made on behalf of or 39 40 17 reflecting the position of the European Medicines Agency or one of its committees or working 41 42 18 parties. 43 44

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 20 of 37

1 2

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 21 of 37 BMJ Open

1 2

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

BMJ Open Page 22 of 37

1 2 3 Table 2: Number and percentage of all referrals (n=52) that use each type of evidence for each purpose 4 5 6 Type of evidence Usagea 7 Effectiveness 8 of risk 9 Risk - Risk - Usage of minimisation 10 Mechanism PK/PDb Efficacy Overall subgroup product measures 11 Pre-clinical evidence 16 (31%) 6 (12%) 2 (4%) 10 (19%) 1 (2%) 0 (0%) 0 (0%) 12 For peer review only 13 Non - randomised trials 1 (2%) 10 (19%) 18 (35%) 14 (27%) 2 (4%) 0 (0%) 0 (0%) 14 15 Randomised trials 3 (6%) 9 (17%) 40 (77%) 36 (69%) 7 (13%) 0 (0%) 1 (2%) 16 Non-interventional 3 (6%) 4 (8%) 18 (35%) 31 (60%) 5 (10%) http://bmjopen.bmj.com/ 14 (27%) 0 (0%) 17 18 Non-interventional using routinely collected data 0 (0%) 1 (2%) 8 (15%) 25 (48%) 4 (8%) 10 (19%) 0 (0%) 19 20 Non-interventional using data collected for research 2 (4%) 4 (8%) 13 (25%) 20 (38%) 3 (6%) 7 (13%) 0 (0%) 21 Spontaneous reports 2 (4%) 0 (0%) 3 (6%) 37 (71%) 6 (12%) 4 (8%) 0 (0%) 22 23 Systematic review of randomised trials 0 (0%) 0 (0%) 19 (37%) 10 (19%) 1 (2%) 0 (0%) 0 (0%) 24 on September 25, 2021 by guest. Protected copyright. 25 Systematic review of non-interventional studies 0 (0%) 0 (0%) 0 (0%) 4 (8%) 1 (2%) 0 (0%) 0 (0%) 26 27 Systematic review of randomised trials & non- 0 (0%) 1 (2%) 2 (4%) 4 (8%) 0 (0%) 0 (0%) 0 (0%) 28 interventional studies 29 Unclear study design 1 (2%) 8 (15%) 12 (23%) 10 (19%) 0 (0%) 1 (2%) 0 (0%) 30 31 32 Legend 33 Percentage of referrals that use Colour 34 evidence type for each purpose 35 <10% 36 10-19% 37 20-29% 38 30-39% 39 40%+ 40 41 42 43 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

Page 23 of 37 BMJ Open

BMJ Open Page 24 of 37

1 2

3 BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from Table 4: Recommendations made as a result of assessment for the 52 included referrals 4 5 6 Recommendation Number of % of all referrals 7 referrals 8 No change 1 2% 9 Further evidence before decision- 2 4% 10 making 11 Suspension or withdrawal of 12 23% 12 marketing authorisation 13 Change to availability 0 0% 14 15 Change to product information 43 83% 16 By section of the Summary of 17 Product Characteristics: 18 - Indication For peer24 review46% only 19 20 - Posology 28 54% 21 - Contraindications 22 42% 22 - Warnings 39 75% 23 24 - Interactions 14 27% 25 - Pregnancy 10 19% 26 27 - Driving/machinery 2 4% 28 - Undesirable effects 26 50% 29 30 - Overdose 3 6% 31 - Studies 13 25% 32 - Nature and contents 3 6% 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 26 of 37

1 2

37 2016;14(1):28. doi: 10.1186/s12916-016-0579-5 [published Online First: 2016/02/14] http://bmjopen.bmj.com/ 38 11. Lane S, Lynn E, Shakir S. Investigation assessing the publicly available evidence supporting 39 postmarketing withdrawals, revocations and suspensions of marketing authorisations in the 40 EU since 2012. BMJ open 2018;8(1):e019759. 41 12. Downing NS, Shah ND, Aminawung JA, et al. Postmarket safety events among novel therapeutics 42 approved by the US Food and Drug Administration between 2001 and 2010. Jama 43 44 2017;317(18):1854-63.

45 13. Zeitoun JD, Lefèvre JH, Downing NS, et al. Regulatory review time and post-market safety events on September 25, 2021 by guest. Protected copyright. 46 for novel medicines approved by the EMA between 2001 and 2010: a cross-sectional study. 47 British journal of clinical pharmacology 2015;80(4):716-26. 48 14. Lester J, Neyarapally GA, Lipowski E, et al. Evaluation of FDA safety-related drug label changes in 49 2010. Pharmacoepidemiology and drug safety 2013;22(3):302-05. 50 15. 21st Century Cures Act H.R.34, 2015. 51 16. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-World Evidence - What Is It and What Can It 52 53 Tell Us? New England Journal of Medicine 2016;375(23):2293-97. 54 17. Kalkman S, van Thiel GJ, Zuidgeest MG, et al. Series: Pragmatic trials and real world evidence: 55 Paper 4. Informed consent. Journal of Clinical Epidemiology 2017;89:181-87. 56 18. Avorn J, Kesselheim ASJNEJoM. The 21st Century Cures Act—will it take us back in time? 57 2015;372(26):2473-75. 58 19. Agency EM. Final report on the adaptive pathways pilot, 2016. 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 27 of 37 BMJ Open

1 2

3 20. Regulation (EU) No 1235/2010 of the European Parliament and of the Council. Official Journal of BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 the European Union 2010 5 6 21. Commision E. Commission Implementing Regulation (EU) No 520/2012 of 19 June 2012 on the 7 performance of pharmacovigilance activities provided for in Regulation (EC) No 726/2004 of 8 the European Parliament and of the Council and Directive 2001/83/EC of the European 9 Parliament and of the Council Official Journal of the European Union 2012 10 22. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for 11 Human Use. Medical Dictionary for Regulatory Activities [Available from: 12 https://bioportal.bioontology.org/ontologies/MEDDRA. 13 23. World Health Organization Collaborating Centre for Drug Statistics Methodology. Anatomical 14 Therapeutic Chemical Classification System [Available from: 15 16 https://www.whocc.no/atc_ddd_index/. 17 24. Knottnerus JA, Tugwell P. Research methods must find ways of accommodating clinical reality, 18 not ignoring it:For the need peer for pragmatic review trials. Journal of onlyclinical epidemiology 2017;88:1-3. 19 25. Ishiguro C, Hall M, Neyarapally GA, et al. Post-market drug safety evidence sources: an analysis of 20 FDA drug safety communications. 2012;21(10):1134-36. 21 26. Vezyridis P, Timmons S. Evolution of primary care databases in UK: a scientometric analysis of 22 research output. BMJ open 2016;6(10):e012785. 23 27. Valproate Art. 31 Assessment Report: European Medicines Agency, 2014. 24 25 28. Duijnhoven RG, Straus SM, Raine JM, et al. Number of patients studied prior to approval of new 26 medicines: a database analysis. PLoS medicine 2013;10(3):e1001407. 27 29. Commission E. Directive 2010/84/EU of the European Parliament and of the Council of 15 28 December 2010 amending, as regards pharmacovigilance, Directive 2001/83/EC on the 29 Community code relating to medicinal products for human use. 2010 30 30. Kesselheim AS, Avorn J. New “21st Century Cures” legislation: speed and ease vs science. Jama 31 2017;317(6):581-82. 32 31. Goodman SN, Schneeweiss S, Baiocchi M. Using design thinking to differentiate useful from 33 misleading evidence in observational research. Jama 2017;317(7):705-07. 34 35 32. Herrett E, Gallagher AM, Bhaskaran K, et al. Data resource profile: clinical practice research 36 datalink (CPRD). International journal of epidemiology 2015;44(3):827-36.

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 28 of 37

1 2

3 Supplementary Table 1: European Medicines Agency referrals categorised by type of referral BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 procedure and date 5 6 Referral category Frequency by CHMP/CMDha opinion date Total 7 2013 2014 2015 2016 2017 8 Jan-Jun 9 Article 107i procedures 5 1 0 0 0 6 10 Article 20 procedures 2 3 1 4 1 11 11 Article 31 referrals 13 13 5 3 1 35 12 Total 20 17 6 7 2 52 13 a) Committee for Medicinal Products for Human Use/Coordination Group for Mutual Recognition and 14 Decentralised Procedures – Human 15 16 Supplementary Table 2: ATCa therapeutic subgroup of medicinal product by frequency for the 17 52 included referrals 18 For peer review only 19 ATC Subgroup definition No. of 20 subgroup referrals 21 code 22 G03 Sex hormones and modulators of the genital system 6 23 N02 Analgesics 6 24 A10 Drugs used in diabetes 3 25 C01 Cardiac therapy 3 26 M01 Anti-inflammatory and antirheumatic products 3 27 None Not applicable/available 3 28 R05 Cough and cold preparations 3 29 A03 Drugs for functional gastrointestinal disorders 2 30 B05 Blood substitutes and perfusion solutions 2 31 C04 Peripheral vasodilators 2 32 C10 Lipid modifying agents 2 33 G02 Other gynelogicals 2 34 L01 Antineoplastic agents 2 35 M03 Muscle relaxants 2 36 N05 Psycholeptics 2 37 http://bmjopen.bmj.com/ R03 Drugs for obstructive airway diseases 2 38 B01 Antithrombotic agents 1 39 B02 Antihemorrhagics 1 40 B03 Antianemic preparations 1 41 C08 Calcium channel blockers 1 42 C09 Agents acting on the renin-angiotensin system 1 43 J01 Antibacterials for systemic use 1 44 J02 Antimycotics for systemic use 1 45 on September 25, 2021 by guest. Protected copyright. J05 Antivirals for systemic use 1 46 L04 Immunosuppresants 1 47 M05 Drugs for treatment of bone diseases 1 48 N03 Antiepileptics 1 49 N07 Psychoanaleptics 1 50 R02 Throat preparations 1 51 R07 Other respiratory system products 1 52 a) Anatomical Therapeutic Chemical (ATC) Classification System 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 29 of 37 BMJ Open

1 2 a 3 Supplementary Table 3: ATC section of medicinal product by frequency for the 52 included BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 referrals 5 6 Number 7 of 8 ATC section Section definition referrals 9 N Nervous system 13 10 C Cardiovascular system 9 11 A Alimentary tract and metabolism 8 12 G Genitourinary system and sex hormones 8 13 R Respiratory system 7 14 M Musculoskeletal system 6 15 B Blood and blood forming organs 5 16 J Anti-infectives for systemic use 3 17 L Antineoplastic and immunomodulating agents 3 18 D DermatologicalsFor peer review only0 19 Systemic hormonal preparations, excluding sex 20 H hormones and insulins 0 21 P Antiparasitic products, insecticides and repellents 0 22 S Sensory organs 0 23 V Various 0 24 a) Anatomical Therapeutic Chemical (ATC) Classification System 25 26 Supplementary Table 4: MedDRAa system organ class (SOC) of adverse event by frequency for 27 the 52 included referrals 28 29 System Organ Class (SOC) Number 30 of 31 referrals 32 Cardiac disorders 16 33 34 Vascular disorders 15 35 Nervous system disorders 9 36 Respiratory, thoracic and mediastinal disorders 7 37 http://bmjopen.bmj.com/ Skin and subcutaneous tissue disorders 7 38 39 Gastrointestinal disorders 6 40 Immune system disorders 5 41 Infections and infestations 5 42 43 Renal and urinary disorders 5 44 Blood and lymphatic system disorders 4 45 Hepatobiliary disorders 4 on September 25, 2021 by guest. Protected copyright. 46 47 Metabolism and nutrition disorders 4 48 Neoplasms benign, malignant and unspecified 4 49 Congenital, familial and genetic disorders 2 50 Endocrine disorders 2 51 52 Musculoskeletal and connective tissue disorders 2 53 Injury, poisoning and procedural complications 1 54 Reproductive system and breast disorders 1 55 56 Sexual function and fertility disorders 1 57 Surgical and Medical Procedures 1 58 a) Medical Dictionary for Regulatory Activities 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 30 of 37

1 2

3 European Medicines Agency referrals included in the study BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 EMAa Reference No. CHMPb Referral Title 6 opinion/CMDhc 7 position date 8 EMEA/H/C/889/A20/37 17/01/2013 Tredaptive, Pelzont and Trevaclyn 9 EMEA/H/C/903/A20/38 10 EMEA/H/C/897/A20/38 11 EMEA/H/A-31/1306 21/03/2013 Cilostazol-containing medicines 12 EMEA/H/A107i/1352 24/04/2013 Tetrazepam-containing medicines 13 EMEA/H/A-107i/1357 29/05/2013 Cyproterone and ethinylestradiol containing 14 medicinal products 15 EMEA/H/A-31/1346 29/05/2013 Almitrine-containing medicines 16 EMEA/H/A-107i/1363 26/06/2013 Flupirtine-containing medicines 17 EMEA/H/A-31/1342 26/06/2013 Codeine-containing medicines 18 EMEA/H/A-31/1344 For peer26/06/2013 reviewDiclofenac-containing only medicines 19 EMEA/H/A-31/1325 27/06/2013 Ergot derivatives 20 EMEA/H/A-31/1322 27/06/2013 Intravenous iron-containing medicinal products 21 EMEA/H/A-31/1314 25/07/2013 Ketoconazole-containing medicines 22 EMEA/H/A-107i/1373 18/09/2013 Numeta G13E and Numeta G16E emulsion for 23 infusion 24 EMEA/H/A-107i/1376 23/10/2013 Hydroxyethyl starch solutions for infusion 25 EMEA/H/A-31/1348 23/10/2013 Hydroxyethyl starch solutions for infusion 26 EMEA/H/A-31/1347 23/10/2013 Short-acting beta-agonists 27 EMEA/H/A-31/1339 24/10/2013 Intravenous nicardipine medicines 28 EMEA/H/A-31/1321 24/10/2013 Metoclopramide-containing medicines 29 EMA/H/A-31/1361 21/11/2013 Thiocolchicoside-containing medicines 30 EMEA/H/A-31/1366 18/12/2013 Substances related to nicotinic acid 31 EMEA/H/C/275/A20/150 19/12/2013 Kogenate Bayer and Helixate NexGen 32 EMEA/H/C/276/A20/143 33 EMEA/H/A-31/1356 16/01/2014 Combined hormonal contraceptives 34 EMEA/H/A20/1371/C/00560- 20/02/2014 Protelos and Osseor 35 561/0039-0034 36 EMEA/H/A-31/1335 20/02/2014 Methysergide-containing medicines

37 EMEA/H/A-31/1349 19/03/2014 Diacerein-containing medicines for oral http://bmjopen.bmj.com/ 38 administration 39 EMEA/H/A-31/1365 24/04/2014 Domperidone-containing medicines 40 EMEA/H/A-31/1377 24/04/2014 Zolpidem-containing medicines 41 EMEA/H/A-31/1382 25/04/2014 Caustinerf arsenical and Yranicid arsenical 42 EMEA/H/A-31/1336 25/04/2014 Linoladiol N and Linoladiol HN 43 EMEA/H/A-31/1370 22/05/2014 Renin-angiotensin-system (RAS)-acting agents 44 EMEA/H/A-107i/1395 23/07/2014 Methadone medicinal products for oral use

45 containing povidone on September 25, 2021 by guest. Protected copyright. 46 EMEA/H/A-31/1391 24/07/2014 Emergency contraceptives 47 EMEA/H/A-31/1379 20/08/2014 Bromocriptine-containing medicines indicated in 48 the prevention or suppression of physiological 49 lactation post-partum 50 EMEA/H/C/2695/A20/0003 23/10/2014 Iclusig 51 EMEA/H/A-31/1383 23/10/2014 Polymyxin-containing medicines 52 EMEA/H/A-31/1396 19/11/2014 Testosterone-containing medicines 53 EMEA/H/A-31/1387 19/11/2014 Valproate and related substances 54 EMEA/H/A20/1404/C/000598/0031 20/11/2014 Corlentor and Procoralan 55 EMEA/H/A20/1404/C/000597/0032 56 EMEA/H/A-31/1400 25/03/2015 Hydroxyzine-containing medicinal products 57 EMEA/H/A-31/1394 22/04/2015 Codeine-containing medicinal products for the 58 treatment of cough or cold in paediatric patients 59 EMEA/H/A-31/1401 20/05/2015 Ibuprofen- and dexibuprofen-containing medicines 60 EMEA/H/A-31/1398 25/06/2015 Adrenaline auto-injectors

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 31 of 37 BMJ Open

1 2

3 EMEA/H/A-31/1397 18/11/2015 Ambroxol and bromhexine-containing medicines BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 EMEA/H/A-20/1421 19/11/2015 Human papillomavirus vaccines 5 EMEA/H/A-20/1419 25/02/2016 SGLT2 (sodium-glucose co-transporter 2) 6 inhibitors 7 EMEA/H/A- 25/02/2016 Tysabri 8 20/1416/C/000603/0083 9 EMEA/H/A-31/1420 31/03/2016 Fusafungine containing medicinal products for 10 oromucosal and nasal use 11 EMEA/H/A-31/1415 28/04/2016 Inhaled corticosteroids containing medicinal 12 products indicated in the treatment of chronic 13 obstructive pulmonary disease 14 EMEA/H/A-20/1439/C/3843/0023 21/07/2016 Zydelig 15 EMEA/H/A-31/1432 13/10/2016 Metformin and metformin-containing medicines 16 EMEA/H/A-20/1438 15/12/2016 Direct-acting antivirals indicated for treatment of 17 hepatitis C (interferon-free) 18 EMEA/H/A-31/1435 For peer26/01/2017 reviewDienogest/ethinylestradiol-containing only medicinal 19 products indicated in acne 20 EMEA/H/A-20/1442 23/02/2017 SGLT2 (sodium-glucose co-transporter 2) 21 inhibitors (previously canagliflozin) 22 a) European Medicines Agency 23 b) Committee for Medicinal Products for Human Use 24 c) Coordination Group for Mutual Recognition and Decentralised Procedures – Human 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 32 of 37

1 2

3 Definition of key terms in study BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 Study type Definition 6 Pre-clinical evidence Evidence from in-vitro and in-vivo (non-human 7 animals) experimentation. 8 Non-randomised trials Interventional studies where assignment to therapy 9 was not at random or where there was only one trial 10 arm (e.g. Phase 1 and Phase 2 trials). 11 Randomised trials Interventional studies where assignment to therapy 12 versus control was random (including both 13 traditional multi-arm randomised controlled trials 14 and randomised crossover trials). 15 Interventional studies Clinical studies where the study investigators 16 intervene on patient therapy. 17 Non-interventional studies Clinical studies where there is no intervention by 18 For peer reviewstudy investigators. only Alternatively termed 19 observational studies. 20 Spontaneous reports Unsolicited reports of adverse outcomes reported by 21 consumers or healthcare professionals. 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 33 of 37 BMJ Open

1 2

3 Primary Data collection form BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 5 1. Basic information about EMAa reference number 6 referral 7 Initiated by (e.g. MHRAb, European 8 Commission) 9 Referral/procedure type (Article 107i/Article 10 31/Article 20) 11 Decision making model (e.g. PRAC- 12 CMDh-ECc) 13 14 Cause of referral (Safety/Efficacy/Safety and 15 efficacy) 16 Cause of referral – description 17 18 For peerCHMPd opinion/CMDh reviewe position only date 19 20 2. Information about product Review title 21 and adverse event

22 Substance name 23 24 25 Product usage 26 27 ATCf group (e.g. N03 - Antiepileptics) 28 29 30 Product class (as listed on EMA website) 31 32 Adverse events 33 34 35 MedDRAg system organ classes of 36 adverse events

37 http://bmjopen.bmj.com/ 3. Determine the types of Source of evidence (Notification/Assessment 38 evidence leading to the referral report/EMA webpage) 39

40 a. Pre-clinical evidence (Yes/No/Unclear) 41 42 b. Non-randomised trials (Yes/No/Unclear) 43 44

45 c. Randomised trials (Yes/No/Unclear) on September 25, 2021 by guest. Protected copyright. 46 47 d. Non-interventional studies (Yes/No/Unclear) 48 49 50 i. Using routinely collected real world (Yes/No/Unclear) data e.g. electronic health records 51 52 ii. Using primary data collection e.g. (Yes/No/Unclear) 53 pregnancy registry 54 e. Spontaneous reports (Yes/No/Unclear) 55 56 57 f. Systematic review of randomised trials (Yes/No/Unclear) 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 34 of 37

1 2

18 For peerd. Non-interventional review studies only(Yes/No/Unclear) 19

20 i. Using routinely collected real world (Yes/No/Unclear) 21 data e.g. electronic health records 22 23 ii. Using primary data collection e.g. (Yes/No/Unclear) 24 pregnancy registry 25 e. Spontaneous reports (Yes/No/Unclear) 26 27 28 f. Systematic review of randomised trials (Yes/No/Unclear) 29 30 g. Systematic review of non- (Yes/No/Unclear) 31 interventional studies 32 33 h. Systematic review combining (Yes/No/Unclear) randomised trials & non-interventional 34 studies 35 i. Unclear design (Yes/No) 36

37 http://bmjopen.bmj.com/ 38 4. b) Summarise the types of a. Pre-clinical evidence 39 evidence used in each assessment report 40 b. Non - randomised trials 41 42 43 c. Randomised trials 44

on September 25, 2021 by guest. Protected copyright. 45 d. Non-interventional studies

47 e. Spontaneous reports 48 49 50 f. Systematic review of randomised trials 51 52 g. Systematic review of non- 53 interventional studies 54 55 h. Systematic review combining 56 randomised trials & non-interventional 57 studies 58 i. Unclear design 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 35 of 37 BMJ Open

1 2

3 5. Determine the a. No change – the available (Yes/No) BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 recommendation made in the evidence dismisses any concern 5 report. b. Further evidence before decision- (Yes/No) 6 making 7 c. Change to product information e.g. (Yes/No) 8 restriction of use, addition of new 9 adverse drug reaction, restriction of dose 10 etc. 11 d. Change to availability e.g. P to (Yes/No) 12 POM 13 e. Suspension or revocation of (Yes/No) 14 marketing authorisation 15 Summary of decision 16 17 18 For peer review only 19 6. If there was a 4.1 Therapeutical indications 20 recommendation for a change 4.2 Posology and method of 21 to product information, which sections of the summary of administration 22 product characteristics (SmPc) 4.3 Contraindications 23 were affected? 24 4.4 Special warnings and precautions for 25 use 26 4.5 Interactions with other medicinal 27 products and other forms of interaction 28 4.6 Fertility, pregnancy and lactation 29 30 4.7 Effects on ability to drive and use 31 machines 32 4.8 Undesirable effects 33 4.9 Overdose 34 35 Other 36

37 7. Determine how non- (a. No evidence from non-interventional studies was cited in the report/ http://bmjopen.bmj.com/ 38 interventional studies b. Evidence from non-interventional studies was cited, but made little to 39 contributed to the decision no contribution to the decision/ 40 made. Judgement is involved in c. Evidence from non-interventional studies was cited, but the decision 41 this step and the assessment will was contrary to this evidence/ be conducted independently by d. The decision was consistent with evidence from non-interventional 42 two researchers. studies, and also consistent with other evidence/ 43 e. The decision was consistent with evidence from non-interventional 44 studies AND this evidence was the primary or only factor involved in 45 the decision/ on September 25, 2021 by guest. Protected copyright. 46 f. Unclear) 47 8. What was useful (or 48 otherwise) about the evidence 49 from non-interventional 50 studies? 51 52 53 54 55 56 57 9. If no non-interventional Yes/no? (Yes/No/Unclear) 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 36 of 37

1 2

3 studies were available, were Further information BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from 4 such studies feasible and could 5 they have been useful? 6 7 10. Does the action taken as a Yes/no? (Yes/No/Unclear) 8 result of the referral require 9 future research? 10 Is further non-interventional evidence (Yes/No/Unclear) required? 11 Further information 12 13 14 15 16 17 18 For peer review only 19 Design of further non-interventional (using data collected for 20 studies in PAS register research/ using routinely 21 collected data) 22 23 a) European Medicines Agency 24 b) Medicines and Healthcare Regulatory Agency c) Pharmacovigilance Risk Assessment Committee – Coordination Group for Mutual Recognition and 25 Decentralised Procedures (Human) - European Commission 26 d) Committee for Medicinal Products for Human Use 27 e) Coordination Group for Mutual Recognition and Decentralised Procedures – Human 28 f) Anatomical Therapeutic Chemical (ATC) Classification System 29 30 31 32 33 34 35 36

37 http://bmjopen.bmj.com/ 38 39 40 41 42 43 44

45 on September 25, 2021 by guest. Protected copyright. 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2018-028133 on 28 October 2019. Downloaded from

Page 37 of 37 BMJ Open

1 2 3 4 5 Secondary data collection form - Example 6 7 EMEA/H/A- Pre- Non - Randomise NIa NI NI studies Spontaneous Systematic Systematic Systematic Unclear 8 107i/1395 clinical randomised d trials studies studies using reports review of review of review study design 9 evidence trials using primary randomised NI studies combining 10 RCDb data trials randomised 11 collection trials & NI 12 For peer review only studies 13 Mechanism of AEc No No No No No No Yes No No No No 14 with product 15 Pharmacokinetics/ Yes Yes No No No No No No No No Yes 16 Pharmacodynamics http://bmjopen.bmj.com/ 17 Efficacy No No Yes No No No No Yes No No No 18 19 Risk - Overall Yes No No No No No Yes No No No Yes 20 Risk - No No No No No No No No No No No 21 Subpopulation 22 Usage of product No No No Yes Yes Yes No No No No No 23 Effectiveness of No No No No No No No No No No No 24 risk minimisation on September 25, 2021 by guest. Protected copyright. 25 a. Non-interventional studies 26 b. Routinely collected data 27 c. Adverse event 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60