ANALYSIS Oseltamivir: the real world data To coincide with the publication of an updated systematic review by the Cochrane group the BMJ invited Nick Freemantle and colleagues to consider the current status of observational studies of oseltamivir and their influence on policy and practice seltamivir has become a mainstay in factors or confounders and are at the mercy of the treatment of influenza, although how accurately these factors are recorded. For there is no evidence from clini- example clinicians use their judgment to decide cal trials that it reduces mortality.1 which treatment participants should receive, and In 2009 we published a review of this will be influenced by whom they recognise to Oobservational studies examining the effects of be at higher risk. Their judgment therefore forms oseltamivir in influenza. It was based on a list unrecorded information on risk that cannot be of studies provided by Roche, the manufacturer, captured in a statistical model,4 which gives rise that it claimed provided evidence of the drug’s to what is known as confounding by indication. “real life” value.2 The evidence supported a role This confounding is relevant to observational for oseltamivir in reducing pneumonia and other data on influenza patients because doctors may complications of influenza in otherwise healthy be more likely to treat patients who are sicker. adults. But we found no evidence that it was Similarly, treatment may be given only to associated with mortality, little information those who survive long enough to receive on safety, and none on pregnancy related it, or remained ill for longer, termed survi- MALCOLM WILLETT MALCOLM outcomes. We have now updated our evidence vorship bias. by doing a systematic search of published obser- example, because of a small P value) then the Survivorship bias may play out in different vational studies. Here we explain the advantages experimental treatment must be the explana- ways in observational studies of oseltamivir. Jain and drawbacks of observational data and what tion. However, even impeccably designed ran- and colleagues describe a multivariable analy- they tell us about oseltamivir. domised controlled trials have limitations. For sis where receiving antiviral therapy within two Worldwide influenza is a major public health example, trials often include participants who days after the onset of illness was associated with concern. During an epidemic about 5-15% of are relatively healthy, young, male, lacking reduced mortality, but their analysis was open the global population develop an upper res- comorbidities, or more motivated and adher- to survivorship bias and confounding by indi- piratory tract infection, 3 to 5 million people ent to treatment than patients seen in clinical cation.5 It is likely that a patient who presented contract severe disease, and there are between practice, so the results may be not be able to be atypically and did not received early antivirals 250 000 and 500 000 influenza related deaths.3 generalised to a wider population. Furthermore, but recovered well might be spared oseltamivir Vaccination is the leading public health con- trials of relatively short duration in selected par- and perhaps not included in the study or data trol measure to prevent disease and reduce the ticipants without other risk factors may not pro- set. On the other hand a sicker or deteriorating effect of epidemics, but neuraminidase inhibi- vide evidence of real world safety. patient might be treated even though late treat- tors (oseltamivir or zanamivir) or M2 ion channel ment was not recommended. When a sicker blockers (amantadine or rimatidine) may be pre- What do observational data have to offer? group of patients is treated late, or when subjects scribed to reduce the duration of symptoms and Observational studies have some advantages who die early (before the opportunity to receive improve clinical outcomes. The use of neurami- over randomised trials. They can be conducted antiviral treatment) are included in an analysis nidase inhibitors increased substantially during in actual populations receiving treatment and as “untreated,” survivorship bias will make the 2009 pandemic of influenza A/H1N1, partly can be conducted rapidly with accruing data, treatment appear more efficacious. because there was no effective vaccine but also avoiding the long lead times and substantial A recent study used a statistical approach that because of concerns over increasing drug resist- costs associated with randomised trials. They claimed to overcome survivorship bias by includ- ance to amantadine and rimatidine.3 can also be conducted when it is considered ing treatment with antivirals as a time depend- unethical to randomise patients between treat- ent explanatory variable.6 This approach can Why randomised trials aren’t the be all and ment conditions, such as in pregnancy.4 Obser- address the survivorship bias associated with end all vational studies may also provide information early deaths, but the method does nothing to Randomised controlled trials are the gold stand- on the safety of interventions because they can recover information about patients missing from ard in research and have a central role in drug include more participants and follow them up the analysis. evaluation and health policy. Randomisation over a longer period. Statistical techniques are available to address ensures that participants differ only by the play The challenge for observational studies is that other types of confounding in observational stud- of chance and treatment allocation. This means treatment is not allocated by the play of chance ies, including multivariable statistical modelling that in a well designed randomised trial, the only (randomisation) so they are subject to substan- and propensity scores.4 Propensity scores, which explanations for a difference in outcome are the tial bias.4 Observational studies should adopt use patient characteristics to estimate the likeli- effect of the experimental treatment or chance, approaches to reduce bias and potential con- hood they received a certain treatment, can be and if chance is not a plausible explanation (for founders, but they can account only for known used to match groups or included in a multivari- 22 BMJ | 12 APRIL 2014 | VOLUME 348 ANALYSIS able model to adjust for or weight different pro- Identification of observational studies 7 pensities to receiving the treatments. Propensity Search methods scores are not as robust as randomisation, as they We designed a sensitive search to retrieve observational studies from electronic bibliographic incorporate only available risk information and databases. In order to retrieve non-randomised studies, we used no study design filter. No date often in a somewhat crude way. In the observa- limitation or language restriction was applied. tional studies of oseltamivir, propensity scores We identified 7523 items from the following databases on 25 February 2014 adjusting for patient risk may not account for MEDLINE via OVID (1946 to 25 February 2013) other factors such as patient entitlement to care, EMBASE via OVID (1947 to 24 Feb 2013) staffing levels, or high dependency care facilities, Cochrane Library via Wiley (Issue 2 of 12, 2014) including the Database of Reviews of Effects (DARE), Health Technology Assessment (HTA), and Economic Evaluations Database (EED) to scan the reference which can partly explain the outcomes measured. lists of relevant systematic reviews PUBMED via NLM ‘Related Articles’ search in PUBMED using the previous review of observational Taking stock of the evidence studies (Freemantle, 2009) as a seed paper2 Following the 2009 H1N1 pandemic the World The search strategy was devised on OVID MEDLINE and then adapted for the other databases. The Health Organization commissioned Hsu and col- search strategies are available in appendix 1 on bmj.com. Screening of the search was limited to studies leagues to review the observational evidence for published from January 2010 to update the searches conducted by Hsu et al.8 In addition we screened oseltamivir, including studies up to November the 51 studies included in Hsu et al to see whether they would indeed meet our more stringent criteria. 8 2010. They found some evidence that oseltami- Inclusion and exclusion criteria vir might reduce mortality in high risk popula- We included studies published in English that compared outcomes for patients prescribed oseltamivir tions (odds ratio=0.23, 95% confidence interval versus patients prescribed no drug. Patients of all ages with or without comorbidities were eligible for 0.13 to 0.43) but they included low quality stud- inclusion. Randomised trials were excluded, as were observational studies without a “no treatment” ies in which the investigators did not use best comparator population. We excluded studies that did not use multivariable models or other methods to address bias. This makes the finding appropriate methods to condition for observed confounders. Thus, studies that use propensity score matching or adjustment were included, but studies that adjusted for a single confounding factor (age), of these studies unreliable, a point recognised or undertook no adjustment, were excluded. by the authors, leaving it unclear whether the pooled results are reliable. Study assessment and review methods DK and LS screened the titles and abstracts of publications identified in the search for potential We performed a systematic search of pub- inclusion. MC, NF, and LS screened full text articles.