Benefit-Risk Assessment: IMI-PROTECT Case Study Example

Benefit-Risk Assessment: IMI-PROTECT case study example

Juhaeri Juhaeri, Ph.D. Head of Pharmacoepidemiology, Sanofi Adjunct Assistant Professor, UNC Chapel Hill

30th International Conference on Pharmacoepidemiology & Therapeutic Risk Management, Taipei, 24 October 2014 Disclaimers

Juhaeri Juhaeri is an employee of Sanofi. The views and opinions represented in this presentation are solely those of the presenter and are not endorsed by nor necessarily representative of those of Sanofi

The 5th Annual Meeting: Patient Safety - A Sustained Focus from Scientific Ideas to Innovative Medicines May 12-15, 2013 2 Objectives • To understand existing frameworks for a structured BR assessment

• To describe quantitative BR methods to integrate benefits and risks

Using rimonabant as an example

3 FDA Structured approach

Decision Evidence and Conclusions Factor Uncertainties and Reasons

Analysis of Condition

Current Treatment Options

Benefit

Risk

Risk Management

Benefit-Risk Summary Assessment http://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm326192.htm 4 EMA Project: 2 level approach

• Qualitative approach • Quantitative approach:  Multi-Criteria Decision Analysis (MCDA) method to derive a numerical value for the benefit–risk balance  recommended for more complex situations

*Zafiropoulos N, Phillips L, Pignatti F, Luria X. Evaluating benefit-risk: an agency perspective. Regulatory Rapporteur 2012 (9): 5-8 5 ISPE: Benefit-Risk Assessment, Communication and Evaluation (BRACE)

6 The Innovative Medicines Initiative (IMI)

• The largest public- private initiative in biopharmaceutical research – € 1 billion from the European Commission (7th Framework Program for Research) – € 1 billion in kind contribution by EFPIA

7 IMI PROTECT Public Regulators: Private EMA (Co-ordinator) EFPIA companies: DKMA (DK) GSK (Deputy Co- AEMPS (ES) ordinator) MHRA (UK) Sanofi Roche Academic Institutions: Novartis University of Munich Pfizer FICF (Barcelona) Amgen INSERM (Paris) Genzyme Mario Negri Institute Merck Serono (Milan) Others: Poznan University of Bayer Medical Sciences WHO UMC Astra Zeneca University of Groningen GPRD Lundbeck University of Utrecht SMEs: IAPO NovoNordisk Imperial College London Outcome Europe CEIFE Takeda University of Newcastle PGRx (LA-SER) Eli Lilly

8 Recommendation Roadmap

Exploration

Evidence Conclusion and gathering and dissemination data preparation Analysis

Planning

http://www.imi- protect.eu/documents/HughesetalRecommendationsforthemethodologyandvi sualisationtechniquestobeusedintheassessmento.pdf 9 Classifications of approaches

Mt-Isa S, et al. Balancing benefit and risk of medicines: a systematic review and classification of available methodologies. Pharmacoepidemiol Drug Saf. 2014 Jul;23(7):667-78. 10 Case studies: Methodologies Rimonabant Telithromycin Efalizumab Natalizumab Rosiglitazone Warfarin

PrOACT-URL ✓ ✓ ✓ ✓ ✓ BRAT ✓ ✓ ✓ ✓ ✓ MCDA ✓ ✓ ✓ ✓ ✓ SMAA ✓ ✓ ✓ NNT & NNH ✓ ✓ Impact ✓ Number QALY Q-TWiST INHB ✓ BRR ✓ ✓ ✓ ✓ PSM ✓ ✓ ✓ MTC ✓ ✓ DCE ✓ Other: Direct utility SBRAM, Decision Decision Decision Individual elicitation, Swing- conferencing conferencing conferencing benefit risk Dashboard, weighting MACBETH, Probabilistic assessment MCDA AHP MCDA model (NCB) simulations 11 Rimonabant Case Study: Disclaimers • “The processes described and conclusions drawn from the work presented herein relate solely to the testing of methodologies and representations for the evaluation of benefit and risk of medicines. • This report neither replaces nor is intended to replace or comment on any regulatory decisions made by national regulatory agencies, nor the European Medicines Agency.”

12 Descriptive framework: PrOACT - URL

Problem Uncertainty

Objective Risk tolerance

Alternatives Linked decisions

Consequences Key references:

Hunink M, Glasziou P, Siegel J, Weeks J, Pliskin J, Elstein A, et al. Decision making in health and medicine: Integrating evidence and values. Trade-off Cambridge: Cambridge University Press; 2001. Hammond JS, Keeney RL, Raiffa H. Smart choices. A practical guide to making better life decisions. New York: Broadway Books; 2002. 13 Descriptive framework: PhRMA BRAT

(6) (1) (3) (5) (2) (4) Display & Decision & Define Identify Assess Identify Customise interpret communication of decision data outcome outcomes framework key B-R B-R assessment context sources importance metrics

Key references: Coplan PM, Noel RA, Levitan BS, Ferguson J, Mussen F. Development of a framework for enhancing the transparency, reproducibility and communication of the benefit-risk balance of medicines. Clin Pharmacol Ther 2011 Feb;89(2):312-5 Levitan BS, Andrews EB, Gilsenan A, Ferguson J, Noel RA, Coplan PM, et al. Application of the BRAT framework to case studies: observations and insights. Clin Pharmacol Ther 2011 Feb;89(2):217-24

14 BRAT vs. PrOACT-URL vs. Regulatory

PrOACT-URL BRAT Regulatory landscape Problem Define decision context Define decision context

Objective Identify benefit and risk Identify important outcomes relevant information

Alternative Define the decision Define decision context context Consequence Extract source data Identify data

Customise framework Identify important relevant information Trade-off Assess outcome Weigh the information importance Uncertainty Weigh the information Risk tolerance Display & interpret key BR metrics

Linked decisions 15 Decision context: Rimonabant

Indication Weight loss in obese and overweight patients with co-morbidities in adults (>18y)

Regulatory history Approved June 2006, Voluntary withdrawal in January 2009

Safety issue Increased risk of depression Data source EPAR Published clinical trials

Methodologies PrOACT-URL, BRAT, MCDA, SMAA, tested NNT&NNH, Impact numbers, INHB, BRR, PSM + direct utility elicitation via survey

16 Rimonabant: value tree

17 Rimonabant: effect table Outcome Rimonabant Placebo Difference

Benefits Weight loss (kg) -6.3 kg -1.6 kg 4.7 kg [4.1-5.3]

Weight loss >10% 25.5% [23.8 , 6.6% [5.5 , 7.9] 19% [17 , 22] 27.3] OR=5.1 [3.6 – 7.3]

Waist -6.2 [-7.2 , -5.2)] -1.9 [-2.3 , -1.4)] -4.3 [-5.5 , -3.0] circumference changes (cm) Systolic blood -1.3 [-2.0 , -0.5] 0.5 [-0.6 , 1.6] -1.8 [-2.8 , -0.8] pressure

Risks Psychiatric 26.2% [24.5 , 14.1% [12.4 , 15.9] 12.1% [10 , 15] adverse event 28.0] OR=1.9 [1.5 , 2.3]

Neurological 27.4% [25.7 , 24.4% [22.3 , 26.6] 3.0% [0.5 , 5.5] Adverse Event 29.2] OR=1.7 [1.1 , 2.7]*

Serious adverse 5.9% [5.0 , 6.9] 4.2% [3.3, 5.3] 1.7% [0.4 , 3.0] event OR=1.43 [1.03 , 1.98]

18 Rimonabant: forest plot

19 Quantitative Methods • Quantitative frameworks: MCDA, SMAA

• Utility survey techniques, eliciting stakeholders’ preference information: DCE

• Estimation techniques: ITC/MTC, PSM

• Metric indices: Impact numbers (including NNT/NNH), BRR

20 Multi-Criteria Decision Analysis (MCDA)

MCDA has 3 ingredients:

B-R evidence Value data functions

Preference weights

Overall benefit-risk balance

Reference: Walker S, Phillips L, Cone M. Benefit-Risk Assessment Model for Medicines: Developing a Structured Approach to Decision Making. Epsom, Surrey: CMR International Institute for Regulatory Science; 2006. 21 MCDA

1. Select benefits and risks 2. Identify data sources and get the data 3. Apply a score to each benefit and risk (based on the data) 4. Apply a weight to each benefit and risk depending on their importance in the decision – Different perspectives 5. Calculate MCDA score for each drug 6. Assess sensitivity of the analysis: – Change in weight – Change in perspective

23 Wave 2 value tree: simplified

24 2. Get the data

Criteria Placebo Rimonabant 10% Weight lost 0.11 [0.10,0.14] 0.40 [0.22,0.62] (proportion) HDL Cholesterol (mg/dL) 1.02 [0.95,1.10] 2.12 [0.26,3.93]

Cardiovascular death 0.02 [0.01,0.02] 0.03 [0.00,0.99] (proportion) Depression (proportion) 0.01 [0.01,0.01] 0.02 [0.00,0.06]

Constipation/Diarrhoea 0.05 [0.05,0.06] 0.07 [0.04,0.15] (proportion)

25 3. Apply score: rimonabant

Weight change

0.40 0.02

Psychiatric disorders 10% weight loss

* Apply to comparison groups 4. Apply weights (medical/regulatory perspective)

Benefits and Risks Weight Normalized weight Overall benefit 73 42.5

Overall risk 98 57.5

10% Weight lost 70 47.9

HDL Cholesterol 76 52.1

Cardiovascular death 100 35.7

Psychiatric disorders 90 32.1

GI disorders 90 32.1 5. Assess B/R - calculate MCDA Score: weighted sum model (WSM)

′

● A: Total MCDA score of alternative i ● w′: normalized weight

● a: score/value of an event

| 28 Rimonabant: Overall BR profile, MCDA model (wave 1)

29 Rimonabant: Benefit score

30 Rimonabant: Risk score

31 Weighted difference in score: rimonabant - placebo

32 6. Assess sensitivity: benefit

33 Sensitivity analysis: risk

34 Stochastic Multi-criteria Acceptability Analysis (SMAA) Data sample from distribution on each criterion

Which alternatives is Utility score more preferred

Repeat for n iterations (to account for variations in input)

Weighted Weight average on sample from a each distribution alternatives

Weighted utility score

• Lahdelma R, Hokkanen J, Salminen P. SMAA - Stochastic multi objective acceptability analysis. European Journal of Operational Research 1998 Apr 1;106(1):137-43. • Tervonen, T. and Lahdelma, R. Implementing stochastic multicriteria acceptability analysis. European Journal of 35 Operational Research, 178 (2): 500-513, 2007 SMAA: illustration

Iteration Value, Ranks (1,2) # Weight 1 (v, w)1 Placebo, Rimonabant 2 (v, w)2 Rimonabant, Placebo 3 (v, w)3 Rimonabant, Placebo 4 (v, w)4 Placebo, Rimonabant 5 (v, w)5 Rimonabant, Placebo 6 (v, w)6 Placebo, Rimonabant 7 (v, w)7 Rimonabant, Placebo 8 (v, w)8 Rimonabant, Placebo 9 (v, w)9 Rimonabant, Placebo 10 (v, w)10 Rimonabant, Placebo

36 Rimonabant: SMAA score (wave 1)

Acceptability index alternative i is ranked r

37 What types of visualisations were useful at the Exploration stage? Rimonabant example

38 DCE: Discrete Choice Experiments

• Stated preference methods, similar to conjoint analysis • Two different options to be compared – Different (multi) criteria – A specific value for each criterion • Assessors are asked to compare two different options at a time – which of the two they prefer. • From many preference statements about many pairs of hypothetical drugs, criterion weights and utilities or preference values can be calculated

39 Rimonabant: example of a choice set

40 Rimonabant: preferences (probit model)

Table shows impact on choice for one unit increment

Attribute Coefficient 95% CI 10% Weight loss 0.034** 0.030 to 0.039 (%) Psychiatric -0.134** -0.158 to -0.110 disorders (%) Cardiovascular -0.097** -0.114 to -0.080 disorders (%) Gastrointestinal -0.035** -0.053 to -0.018 disorders (%) Physician’s view on HDL cholesterol level 0.306** 0.224 to 0.387 (per level) * p≤0.05, ** p≤0.001; Log-likelihood = -577.88 41 ITC/MTC: Indirect/Mixed Treatment Comparison

• Indirect Treatment Comparison Placebo – Used in evidence synthesis when the relevant direct evidence between 2 treatments Treatment A Treatment B are not available • Mixed Treatment Comparison ?

– … when direct and Both require common indirect evidence are denominator, e.g. comparison available and relevant to placebo

42 MTC – Rimonabant example

Rimonabant V Placebo Rimonabant S

Sibutramine V Placebo Sibutramine V Placebo S S

Orlistat V Placebo Orlistat S

43 Summary table

Criteria Placebo Orlistat Sibutramine Rimonabant

10% Weight lost 0.11 [0.10,0.14] 0.24 [0.15,0.34] 0.47 [0.16,0.80] 0.40 [0.22,0.62] (proportion)

HDL Cholesterol 1.02 [0.95,1.10] 1.92 [1.12,2.75] 2.63 [1.19,4.11] 2.12 [0.26,3.93] (mg/dL)

Cardiovascular death 0.02 [0.01,0.02] 0.11 [0.00,1.00] 0.04 [0.00,1.00] 0.03 [0.00,0.99] (proportion)

Depression 0.01 [0.01,0.01] 0.00 [0.00,0.03] 0.02 [0.00,0.11] 0.02 [0.00,0.06] (proportion)

Constipation/Diarrhoe 0.05 [0.05,0.06] 0.19 [0.06,0.50] 0.09 [0.02,0.35] 0.07 [0.04,0.15] a (proportion)

44 Impact Numbers Metric index Definition Formula the difference in risk between exposed and Attributable risk (AR) × unexposed groups Population attributable the attributable risk in the whole population × − 1 risk (PAR) 1 + × Attributable fraction the attributable risk of exposure among − 1 among exposed (AFE) exposed cases − 1 the number of people with the medical − 1 Disease impact number 1 condition in question amongst whom one event (DIN) AR × is attributable to exposure to the risk factor the number of people in the whole population Population impact 1 amongst whom one case is attributable to DIN × number (PIN) exposure to the risk factor the number of people with the case for whom Case impact number 1 one case will be attributable to the exposure or (CIN) risk factor PAR the number of people with the exposure Exposure impact number 1 amongst whom one excess case is due to the (EIN) or NNT/NNH exposure AR Exposed cases impact the number of exposed cases amongst whom 1 number (ECIN) one case is due to the exposure AFE Population impact the potential number of cases prevented in the number of eliminating a study population over the next years by × × PAR risk factor over time t eliminating a risk factor (PIN-ER-t) Number of events the number of cases prevented by the prevented in a population intervention in the study population × × × × (NEPP) − 1 45 Impact Numbers

Ie Iu

ECIN AFE

PAR

PIN-ER-t

CIN

NEPP n Pe

DIN

PIN EIN

AR Pd

46 NNT and NNH: What does (NNT/NNH)<1 mean?

2006 2008 Criterion Mean Median 95% CI Mean Median 95% CI 10% weight loss at 1 year 4.04 4.01 (3.26, 4.98) 4.04 4.01 (3.26, 4.98) Reduction in metabolic 6.60 6.45 (4.60, 9.51) 6.60 6.45 (4.60, 9.51) syndrome Diarrhoea 96.82 65.03 (22.34, 438.56) 62.25 49.77 (25.30, 160.65) Nausea 14.85 14.29 (9.26, 23.59) 14.09 13.69 (9.32, 21.08) Vomiting 114.78 56.20 (25.69, 189.97) 55.53 48.90 (24.89, 123.24) Asthenia/Fatigue 123.91 86.02 (-833.38, 100.90 65.84 (26.95, 381.92) 1050.61) Influenza 40.62 71.50 (-1690.07, 40.62 71.50 (-1690.07, 1728.07) 1728.07) Gastroenteritis viral 226.56 113.82 (-1284.52, 226.56 113.82 (-1284.52, 1573.11) 1573.11) Upper respiratory tract -39.92 68.23 (-997.91, -39.92 68.23 (-997.91, infections 1074.40) 1074.40)

Holden WL, Juhaeri J, Dai W. Benefit-risk analysis: a proposal using quantitative methods. Pharmacoepidemiology and Drug Safety 2003;12:611-6 47 NEPP: Number of events prevented in the population

2006 2008 Criterion Mean Median 95% CI Mean Median 95% CI

10% weight loss at 1 year 715.86 712.55 (573.09, 862.17 858.18 (690.22, 877.76) 1057.17) Reduction in metabolic 447.36 442.78 (299.82, 538.80 533.27 (361.10, syndrome 621.13) 748.08) Diarrhoea 44.65 42.56 (1.11, 99.92) 71.33 68.98 (20.83, 134.93) Nausea 203.48 199.61 (121.04, 254.98 251.03 (162.91, 308.28) 368.89) Vomiting 53.83 50.75 (14.83, 73.60 70.29 (27.91, 110.45) 138.20) Asthenia/Fatigue 30.08 28.18 (-10.94, 53.20 51.17 (4.32, 113.38) 81.48) Influenza 9.81 8.43 (-38.37, 11.82 10.15 (-46.21, 79.12) 65.70) Gastroenteritis viral 22.33 20.31 (-9.47, 26.89 24.46 (-11.40, 78.71) 65.35) NEPP: the number of people in the trials in whom the events would

have been prevented if everybody had received placebo 48 PSM: Probabilistic simulation method • An estimation technique to incorporate statistical uncertainties in the simulated parameters in different methods

• How PSM can be used to evaluate uncertainty in NEPP

49 PSM: assumptions

Parameter

Baseline risk In general, , for criteria is formulated as follows: ~ , , , , = / , , , is then the simulated proportion with Binomial errors. This two-stage , parameterisation is to accommodate evidence data more directly as they are often reported as percentages. The same parameterisation are also used for and , so we only show the first line.

Relative risk s correspond to the criteria, and sampled as follows for each criterion : = log log , , , = 0.975 − log , ~ , , 2 × ϕ = , Λ is then the simulated relative risk with the specified log-normal errors.

Proportion eligible for ~ 2168, 0.6956 Acomplia Proportion of overweight 2006: ~ 12088,0.616 and obese 2008: ~ (12835,0.614) Size of population as constants as constants 50 PSM: The effect of halving and doubling baseline rates for psychiatric adverse events on the NEPP in 2006

Double risk Anxiety Same risk Half risk Double risk Depressivedisorders Samerisk Half risk Double risk Insomnia Same risk Half risk Double risk Irritability Same risk Half risk Double risk Mood alterations with depressive symptoms Same risk Half risk Double risk Nervousness Same risk Half risk Double risk Parasomnia Same risk Half risk Double risk Sleep disorder Same risk Half risk

-200 0 200 400 600 800 1000 1200 1400 1600 1800 2000 NEPP in 2006 for different baseline risks

51 BRR: Benefit Risk Ratio • Impact numbers does not integrate benefits and risks • BRR combines benefits and risks

is the scaling factor such that benefit risk • How much a decision-maker prefers to avoid the risk more than gaining the benefit • A balance is achieved when BRR equals one that is when benefit equals risk

52 BRR: weight loss and anxiety

2006 2008 150 When a DM prefers to lose weight twice as much than avoiding anxiety i.e. relative importance of 0.5, the ratio of benefit to risk is

y 11.1 (95% CI 7.4 – 19.9). This t e

i means 11 people would have lost 1 x - 100 weight by taking Acomplia for n R

A every person who experienced E . - anxiety due to Acomplia. When s N v I

weight loss is valued as important P s

as avoiding anxiety, the BRR drops f s o o

l to 5.6 (95% CI 3.7 – 9.0).

t R h R g

i 50 B e W

.1.125 .25 .5 1 2 4 8 10 .1.125 .25 .5 1 2 4 8 10 Relative importance of benefit to risk Median 95% CI

53 BRR: weight loss and severe depression s r

e 2006 2008 d r

o 6000 s i d

c i r t 4000 a i h c y 1 s -

p 2000 R

e E r - e N I v e P

0 s f

o y

n R A

s -2000 B u s r e v

s -4000 s o l

t h g

i -6000 e .1.125 .25 .5 1 2 4 8 10 .1.125 .25 .5 1 2 4 8 10 W Relative importance of benefit to risk Median 95% CI

54 Final remarks • BR methods support decision making process, but do not replace clinical judgement • Website of PROTECT BR Group rimonabant study case (wave 1 and wave 2 reports) – http://www.imi-protect.eu/benefitsRep.shtml • Website of PROTECT BR Group findings with interactive features: – http://www.imi-protect.eu/benefit-risk.html • More PROTECT BR manuscripts will be published

55 Acknowledgements

• The research leading to these results was conducted as part of the PROTECT consortium (Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium, www.imi- protect.eu) which is a public-private partnership coordinated by the European Medicines Agency. • The PROTECT project has received support from the Innovative Medicines Initiative Joint Undertaking (www.imi.europa.eu) under Grant Agreement n° 115004, resources of which are composed of financial contribution from the European Union's Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.