ANALYSIS Oseltamivir: the To coincide with the publication of an updated by the Cochrane group the BMJ invited Nick Freemantle and colleagues to consider the current status of observational studies of oseltamivir and their influence on policy and practice

seltamivir has become a mainstay in factors or confounders and are at the mercy of the treatment of influenza, although how accurately these factors are recorded. For there is no evidence from clini- example clinicians use their judgment to decide cal trials that it reduces mortality.1 which treatment participants should receive, and In 2009 we published a review of this will be influenced by whom they recognise to Oobservational studies examining the effects of be at higher risk. Their judgment therefore forms oseltamivir in influenza. It was based on a list unrecorded information on risk that cannot be of studies provided by Roche, the manufacturer, captured in a statistical model,4 which gives rise that it claimed provided evidence of the drug’s to what is known as confounding by indication. “real life” value.2 The evidence supported a role This confounding is relevant to observational for oseltamivir in reducing pneumonia and other data on influenza patients because doctors may complications of influenza in otherwise healthy be more likely to treat patients who are sicker. adults. But we found no evidence that it was Similarly, treatment may be given only to associated with mortality, little information those who survive long enough to receive on safety, and none on pregnancy related it, or remained ill for longer, termed survi-

MALCOLM WILLETT MALCOLM outcomes. We have now updated our evidence vorship . by doing a systematic search of published obser- example, because of a small P value) then the may play out in different vational studies. Here we explain the advantages experimental treatment must be the explana- ways in observational studies of oseltamivir. Jain and drawbacks of observational data and what tion. However, even impeccably designed ran- and colleagues describe a multivariable analy- they tell us about oseltamivir. domised controlled trials have limitations. For sis where receiving antiviral therapy within two Worldwide influenza is a major public health example, trials often include participants who days after the onset of illness was associated with concern. During an epidemic about 5-15% of are relatively healthy, young, male, lacking reduced mortality, but their analysis was open the global population develop an upper res- comorbidities, or more motivated and adher- to survivorship bias and confounding by indi- piratory tract infection, 3 to 5 million people ent to treatment than patients seen in clinical cation.5 It is likely that a patient who presented contract severe disease, and there are between practice, so the results may be not be able to be atypically and did not received early antivirals 250 000 and 500 000 influenza related deaths.3 generalised to a wider population. Furthermore, but recovered well might be spared oseltamivir Vaccination is the leading public health con- trials of relatively short duration in selected par- and perhaps not included in the study or data trol measure to prevent disease and reduce the ticipants without other risk factors may not pro- set. On the other hand a sicker or deteriorating effect of epidemics, but neuraminidase inhibi- vide evidence of real world safety. patient might be treated even though late treat- tors (oseltamivir or zanamivir) or M2 ion channel ment was not recommended. When a sicker blockers (amantadine or rimatidine) may be pre- What do observational data have to offer? group of patients is treated late, or when subjects scribed to reduce the duration of symptoms and Observational studies have some advantages who die early (before the opportunity to receive improve clinical outcomes. The use of neurami- over randomised trials. They can be conducted antiviral treatment) are included in an analysis nidase inhibitors increased substantially during in actual populations receiving treatment and as “untreated,” survivorship bias will make the 2009 pandemic of influenza A/H1N1, partly can be conducted rapidly with accruing data, treatment appear more efficacious. because there was no effective vaccine but also avoiding the long lead times and substantial A recent study used a statistical approach that because of concerns over increasing drug resist- costs associated with randomised trials. They claimed to overcome survivorship bias by includ- ance to amantadine and rimatidine.3 can also be conducted when it is considered ing treatment with antivirals as a time depend- unethical to randomise patients between treat- ent explanatory variable.6 This approach can Why randomised trials aren’t the be all and ment conditions, such as in pregnancy.4 Obser- address the survivorship bias associated with end all vational studies may also provide information early deaths, but the method does nothing to Randomised controlled trials are the gold stand- on the safety of interventions because they can recover information about patients missing from ard in and have a central role in drug include more participants and follow them up the analysis. evaluation and health policy. Randomisation over a longer period. Statistical techniques are available to address ensures that participants differ only by the play The challenge for observational studies is that other types of confounding in observational stud- of chance and treatment allocation. This means treatment is not allocated by the play of chance ies, including multivariable statistical modelling that in a well designed randomised trial, the only (randomisation) so they are subject to substan- and propensity scores.4 Propensity scores, which explanations for a difference in outcome are the tial bias.4 Observational studies should adopt use patient characteristics to estimate the likeli- effect of the experimental treatment or chance, approaches to reduce bias and potential con- hood they received a certain treatment, can be and if chance is not a plausible explanation (for founders, but they can account only for known used to match groups or included in a multivari-

22 BMJ | 12 APRIL 2014 | VOLUME 348 ANALYSIS

able model to adjust for or weight different pro- Identification of observational studies 7 pensities to receiving the treatments. Propensity Search methods scores are not as robust as randomisation, as they We designed a sensitive search to retrieve observational studies from electronic bibliographic incorporate only available risk information and databases. In order to retrieve non-randomised studies, we used no study design filter. No date often in a somewhat crude way. In the observa- limitation or language restriction was applied. tional studies of oseltamivir, propensity scores We identified 7523 items from the following databases on 25 February 2014 adjusting for patient risk may not account for MEDLINE via OVID (1946 to 25 February 2013) other factors such as patient entitlement to care, EMBASE via OVID (1947 to 24 Feb 2013) staffing levels, or high dependency care facilities, via Wiley (Issue 2 of 12, 2014) including the Database of Reviews of Effects (DARE), Health Technology Assessment (HTA), and Economic Evaluations Database (EED) to scan the reference which can partly explain the outcomes measured. lists of relevant systematic reviews PUBMED via NLM ‘Related Articles’ search in PUBMED using the previous review of observational Taking stock of the evidence studies (Freemantle, 2009) as a seed paper2 Following the 2009 H1N1 pandemic the World The search strategy was devised on OVID MEDLINE and then adapted for the other databases. The Health Organization commissioned Hsu and col- search strategies are available in appendix 1 on bmj.com. Screening of the search was limited to studies leagues to review the observational evidence for published from January 2010 to update the searches conducted by Hsu et al.8 In addition we screened oseltamivir, including studies up to November the 51 studies included in Hsu et al to see whether they would indeed meet our more stringent criteria. 8 2010. They found some evidence that oseltami- Inclusion and exclusion criteria vir might reduce mortality in high risk popula- We included studies published in English that compared outcomes for patients prescribed oseltamivir tions (=0.23, 95% confidence interval versus patients prescribed no drug. Patients of all ages with or without comorbidities were eligible for 0.13 to 0.43) but they included low quality stud- inclusion. Randomised trials were excluded, as were observational studies without a “no treatment” ies in which the investigators did not use best comparator population. We excluded studies that did not use multivariable models or other methods to address bias. This makes the finding appropriate methods to condition for observed confounders. Thus, studies that use propensity score matching or adjustment were included, but studies that adjusted for a single confounding factor (age), of these studies unreliable, a point recognised or undertook no adjustment, were excluded. by the authors, leaving it unclear whether the pooled results are reliable. Study assessment and review methods DK and LS screened the titles and abstracts of publications identified in the search for potential We performed a systematic search of pub- inclusion. MC, NF, and LS screened full text articles. Where there was uncertainty about the inclusion lished observational studies to find new evidence of a study this was resolved through discussion. The major design characteristics of included studies published after 2012 and review this together were tabulated. Studies that included mortality outcomes, harms (neuropsychiatric outcomes), or that with older studies under stricter inclusion and were conducted in pregnancy were summarised in narrative review. quality criteria. We included only those studies We identified 18 potentially adequately designed observational studies (table 1, appendix that prespecified a statistical method for deal- on bmj.com).9‑26 These studies fell naturally into two types: those based on large scale medical ing with bias (box). We focused on the effects records or insurance claims based databases10‑13 16 17 21 25 and those based on tailored disease 9 14 15 18 19 22‑ 24 26 of oseltamivir in patients with influenza and registries, with the largest study including more than 80 000 subjects and the its association with mortality. We collated and smallest studies including fewer than 300 subjects. reviewed narratively those studies found to have Five studies used propensity score based methods to address observed sources of confounding,9 12‑ 14 16 with the remainder using multivariable regression models. Four studies included used potentially adequate methods to address an assessment of the effects of oseltamivir on mortality.9 14 18 19 Three studies examined pregnant confounding. In addition, we identified studies 11 24 25 10 14 21‑ 23 women, five studies included only children and adolescents, nine included adults and that examined the effect of oseltamivir in preg- children,9 12 15‑ 20 26 and two included only adults.11 13 Four studies included only patients admitted to nancy and on neuropsychiatric events. Below, we hospital with influenza virus.14 18 19 26 summarise the evidence. Liem and colleagues (2009) included 67 predict the outcome in new data sets. Coupled Mortality (72%) of 93 cases of influenza A/H5N1 infec- with (only studies with positive Only three modestly sized studies met our inclu- tion diagnosed in Vietnam from a retrospective results being published) overfitting causes major sion criteria and provided estimates of the effect notes review of laboratory confirmed cases problems because it may lead to exaggerated esti- of oseltamivir on mortality.9 18 19 Even though between January 2004 and December 2006.18 mates of treatment effect in some studies. the studies met our inclusion criteria they There were 26 deaths among the 67 included McGeer and colleagues studied 322 adults were all open to bias because of inadequacies cases (18 deaths among the 55 patients treated admitted to Toronto Invasive Bacterial Diseases in design, analysis, and reporting. Each study with oseltamivir). Given the small number of Network hospitals with laboratory confirmed points towards oseltamivir reducing mortality deaths the multivariable regression models are influenza from 1 January 2005 to 31 May 2006.19 (fig 1),9 18 19 and there was no evidence of het- likely to be overfitted and may have provided Again, given the number of deaths (27 deaths by erogeneity between study results (P=0.77). Thus unstable results. Overfitting is a problem associ- 15 days), the parameter estimates may be biased the studies included provided reasonably con- ated with multiple testing where including too by overfitting. sistent evidence, which points towards a benefit many explanatory variables means that a posi- A fourth study, by Coffin and colleagues, also for oseltamivir in this setting. tive result can be expected simply on the basis examined risk of death but did not report overall Adisasmito and colleagues’ (2010) study of chance. Harrell and colleagues suggested that results. The authors studied 252 children admit- of people who were and were not treated with as a rule of thumb for survival models the maxi- ted to one of 41 participating US hospitals with oseltamivir included 221 people with con- mum number of explanatory variables should influenza and treated with oseltamivir within 24 firmed influenza A/H5N1, of whom 140 died, be one tenth the number of observed deaths.27 hours of admission; these were matched with drawn from a registry of 12 countries spon- The big problem with overfitted models is that children who were not treated with oseltami- sored by Roche.9 they can appear to fit the data well but will fail to vir using a propensity score.14 They found no

BMJ | 12 APRIL 2014 | VOLUME 348 23 ANALYSIS

Odds ratio Odds ratio P value (classifying patients into quintiles of the pro- (% CI) (% CI) pensity score), and the time dependent covari-

Adisasmito  . (. to .) . ate survival approach used may not adequately Liem  . (. to .) .  deal with likely survivorship bias. The authors McGeer . (. to .) . did not describe whether there was an interac- tion between the propensity score and outcome, . . . .   which as we have previously described is a useful 4 Fig 1 | Estimated effects of oseltamivir in reducing mortality. (Odds ratio for Adisasmito9 derived from the check on how well the method addresses bias. reported and control exposure event rate) Those hoping to be informed by the observa- tional studies might question why randomised trials in higher risk populations, specifically Odds ratio Odds ratio P value (% CI) (% CI) those with comorbidities, have not been con- ducted. As health systems have been happy  Greene  . (. to . ) . to purchase substantial stockpiles of antiviral Blumentals  .  (.  to . ) . agents against the perceived risk of an influ- .  enza pandemic, it has not been in the interests Fig 2 | Estimated effects of oseltamivir in reducing psychiatric events of manufacturers to undertake such trials, and others with a research funding interest have not d­ifference in the rate of death between these employed by Roche and Thompson Healthcare, taken up the challenge to support trials in this two groups and did not provide a summary esti- and the work of Greene and colleagues was area despite the clear importance of such trials. mate of risk or the actual numbers of deaths in sponsored by America’s Health Insurance Plans We might also wonder why drug regulators the matched cohorts. Somewhat oddly they did (AHIP) under contract from the Centers for have not taken the licensing of antiviral drugs report the predicted risk of death, and it seems Disease Control and Prevention (CDC). more seriously, requiring high quality evidence likely that there were no more than two deaths of effect in a range of populations as they do for in each group. Their failure to publish the overall What is needed? other conditions, particularly among those at results is evidence of publication bias. We have summarised the better designed obser- high risk because of comorbidities. vational evidence on the effects of oseltamivir in Influenza is a predictable threat that occurs Pregnancy the treatment of influenza. The effects on new every year, and people with comorbidities face Since 2009 three observational studies have psychiatric and neurological conditions and evi- potentially serious consequences as a result. been published that focus on the association dence on the harms of treatment in pregnancy Requiring or facilitating adequately designed between treatment with oseltamivir during preg- remain unclear. research would be in the public interest, and nancy and maternal infant outcomes.11 24 25 No The health of pregnant women is difficult to public funding mechanisms have failed in their significant associations between adverse preg- examine in randomised trials but data are impor- duty of care towards patients. nancy outcomes and the use of oseltamivir were tant because pregnant women may be at higher Nick Freemantle professor of clinical observed, but the confidence intervals are wide risk of illness and complications associated and biostatistics, Research Department of Primary Care and do not preclude harm. One study indicated with influenza.29‑32 Pregnant women could be and Population Health, PRIMENT , UCL Medical School (Royal Free Campus), London NW3 2PF, UK higher rates of transient hypoglycaemia in included in future randomised trials investigat- L J Shallcross clinical lecturer public health medicine, exposed infants, but again the confidence inter- ing oseltamivir to provide more robust estimates Research Department of Infection and Population Health, vals were wide (odds ratio=4.0, 95% confidence of efficacy and safety with “a plan for monitoring UCL Medical School 24 interval 1.23 to 12.76). We could not ascertain the outcome of the pregnancy with regard to both D Kyte doctoral researcher, School of Sport, Exercise and whether pregnant women were included in the the health of the woman and the short-term and Rehabilitation, University of Birmingham, Birmingham, published observational studies evaluating the long-term health of the child.”33 UK and Primary Care Clinical Sciences, University of effects of oseltamivir on mortality.9 18 19 How persuasive should we find the results of Birmingham T Rader knowledge translation specialist, Cochrane mortality studies described here? The studies Musculoskeletal Group, Centre for Global Health, University Neuropsychiatric events seem to show that oseltamivir reduces mortality. of Ottawa, Ottawa, Canada Three studies report rates of neuropsychiatric However, they are based on relatively small num- M J Calvert professor of outcomes methodology, Primary events (psychiatric or neurological events such bers of participants, use designs that are known Care Clinical Sciences, University of Birmingham as depression or seizures) with oseltamivir.12 16 17 to be open to substantial ,4 and were not Correspondence to N Freemantle Funch and colleagues (2012) found no sig- optimally designed or conducted. We consider [email protected] nificant difference in reported adverse events the findings interesting but inconclusive. Contributors and sources: NF and MC have skills and experience in the design, conduct, and evaluation of between treated and untreated adults but found Recently, a large funded observational and randomised evaluative studies, and in a modest increase among treated adolescents by Roche examined the effects of neuramini- the conduct of systematic overviews. LS has expertise in (=3.14, 1.05 to 9.67; P=0.046).16 dase inhibitors on mortality in patients admit- infection and systematic overviews. DK has expertise in the conduct of systematic overviews. TR has expertise in the However, subgroup analyses such as this are ted to hospital with influenza A H1N1pdm09 design and implementation of systematic search strategies well known to be potentially misleading.28 The virus infection, based on individual patient and of systematic reviews. studies by Blumentals and colleagues (2007) data (and pooling many previous studies).6 This Competing interests: TR received support from the BMJ to and Greene and colleagues (2013) both point study provides data that support a reduction in conduct the searches. Provenance and : Commissioned; externally peer towards oseltamivir reducing neuropsychiatric mortality and the advantages of early treatment reviewed. events, although only Blumentals is statisti- over late treatment and in pregnancy. However, References are in the version on bmj.com cally significant (fig 2). Blumentals studied staff the propensity score method used was weak Cite this as: BMJ 2014;348:g2371

24 BMJ | 12 APRIL 2014 | VOLUME 348