Epidemiology and Econometrics: Two Sides of the Same Coin Or Different Currencies?
Total Page:16
File Type:pdf, Size:1020Kb
Epidemiology and Econometrics: Two Sides of the Same Coin or Different Currencies? Marcella Vigneri1 , Edoardo Masset1 , Josephine Exley2, Howard White3 1CEDIL, 2London School of Hygiene and Tropical Medicine, 3Campbell Collaboration Centre of Excellence in Impact Evaluation and Learning (CEDIL) Inception Paper Draft version: 19 January 2018 Abstract This paper discusses differences observed in the practice, conduct and reporting of evaluation studies conducted by economists and clinical epidemiologists. There has been much informal debate around the reasons for these differences and the missed opportunities that they lead to, but we are not aware of any formal review of the key areas where these disciplines diverge and why. This paper fills this gap, drawing on a workshop organized by the Centre of Excellence in Impact Evaluation and Learning (CEDIL) with evaluation specialists from both disciplines, published papers and a systematic approach to illustrate idiosyncrasies from each discipline. The paper also includes an epidemiology-economics evaluation glossary that speaks to both disciplines to facilitate cross learning and synergies between the evaluations that they do 1. Introduction Economics and epidemiology are disciplines that share a research interest in investigating and measuring the impact of interventions in international development. In a 2016 contribution to the American Journal of Public Health, Spiegelman (Spiegelman, 2016) claimed that similarities in the conduct of impact evaluations across disciplines like epidemiology and economics overwhelm the differences, and that the distinctions that have been made confuse the underlying common ground. She broadly defined impact evaluation as a method for assessing the efficacy and effectiveness of an intervention in terms of intended and unintended health, social, and economic outcomes, and involving the explicit statement of a counterfactual. While the principles of Spiegelman’s definition are clear, it is also important to recognize and understand the boundaries that separate how economists and epidemiologists conduct evaluations. Although the principles of their analytical methods are similar (by research questions, type of intervention, intended goal of the intervention), their practice for conducting and reporting evaluations is often different. This paper addresses these boundaries by reviewing evaluation practices of economists and epidemiologists. For the purposes of this paper, we would include the work done by health economists within epidemiological evaluations (for instance to estimate the cost effectiveness of an intervention) as “epidemiology” rather than “economics”. The paper was initially motivated by two debates around differences in these practices. The first debate - led by CEDIL Intellectual Leadership Team members - looked at how the increase in research on global health and international development by scholars in economics (who typically have greater influence on LMIC1 government policymakers in using evidence than researchers in clinical epidemiology), has generated a critical gap in the research methods and practice of evaluation tools between the impact evaluations they do and those within the discipline of clinical epidemiology. This paper builds on this debate to compare the two different approaches, highlighting similarities and differences, and areas for cross learning. The second debate comes from a research paper written by economists and epidemiologists based at the London School of Hygiene and Tropical Medicine (LSHTM) (Powell-Jackson et al., forthcoming 2018) on the practice of randomized trials by the two disciplines. Those authors suggest that evidence-based public health can be strengthened by a better understanding of the differences in the use, practice and reporting of results from trials, and in so doing promote cross-disciplinary learning. In response to these debates, the CEDIL organized a workshop in November 2017 gathering economists and epidemiologists specialising in evaluation research to discuss areas of overlap and of divergence. In preparation for the workshop we collected some evidence of differences in approaches between economists and epidemiologists in two ways. First, we selected a number of systematic reviews of interventions in early child development/nutrition specific programmes and in Conditional Cash Transfer (CCT) programmes, and used the discussion and results of the reviews to reflect on approaches and differences between primary researchers and reviewers in the two disciplines. Second, we identified a few publications of project types evaluated by economists and by epidemiologists to identify main methodological differences in the conduct of impact evaluations. 1 Low and Middle-Income Countries 1 This paper resulted as the critical elaboration of what emerged in the workshop. Section 2 identifies areas where economist and epidemiologist differ in the practice of evaluation studies. After a brief description of the issues relevant in each area, examples from peer reviewed papers in each discipline are used to illustrate differences in practice. The common themes selected for this exercise are conditional cash transfers (CCT), bednets, WASH (Water, Sanitation and Hygiene), anti- retroviral treatment (ART) for the reduction of HIV incidence, and gender interventions to reduce the risk of Intimate Partner Violence (IPV). Section 3 draws reflections from comparing systematic reviews in health insurance, school feeding and conditional and unconditional cash transfers programmes to illustrate further differences in the evaluation methods used in the two disciplines. Section 4 explains the rationale for collating a unified “epi-econ” evaluation glossary (included in the annex) to compare terms used by the two disciplines and try to reduce the clashes which often cloud cross-discipline understanding of evaluation studies. This glossary will help evaluation researchers to identify a common lexicon of their different approaches. Section 5 concludes the paper, reflecting on learning opportunities across the two approaches. 2. Comparing economic and epidemiology approaches to evaluation How do economists and epidemiologists conduct evaluation studies? In general, economists are interested in measuring the success of a program—where success can be broadly or narrowly defined. Their goal is to differentiate less effective interventions from successful ones, and help improve existing programs. The primary purpose of impact evaluation for economists is to determine whether a program has an impact (on a set of key outcomes), and more specifically, to quantify how large that impact is (insert ref. from J-Pal). On the other hand, epidemiologists operating in public health services conduct evaluations as a process for determining, as systematically and objectively as possible, the likely future relevance, effectiveness, efficiency, and impact of activities with respect to established health outcomes (insert ref. from Centres for Disease Control and Prevention). One could say that although evaluations conducted in both disciplines share the common goal to identify presence or absence of impact, their ‘journey’ to an estimate of impact differs in many ways. In the November 2017 workshop, we identified eight key areas where differences in evaluation practices are prominent: 1. The use of theory to underpin model evaluation; 2. The generalizability and transferability of findings from the specifics of an individual case study; 3. The definition of research questions; 4. Study design, and endpoint/outcome measure of impact; 5. Power calculations and powered studies; 6. Publication of protocols and pre-specified outcomes; 7. Reporting findings, and timing of publications; and 8. Replicability. We discuss each of these below to highlight differences. A word of caution is warranted in reading the discussion that follows: while the paper reviews issues that often contrast how economist and epidemiologists conduct evaluation studies, by no means it intends to claim universal truths. There are, of course, evaluations done by economists that more closely resemble those carried out by epidemiologists than those completed by other economists, and vice versa. 2.1 Using Theory to underpin the evaluation problem In ‘best practice’ economics, it is not unusual to find that impact evaluations are the empirical translation of some theory of economic behavior. Economists use theory both to model behaviour, and to identify ex-ante predictions about changes in behavior that can be attributed to an 2 intervention. While the use of theory is not universal among economists, it is generally seen as giving stronger grounding to impact evaluations; theory informs the design of the study, and guides the empirical testing. For instance, economists sometimes use structural models to investigate the mechanisms behind the impacts of a given intervention and to guide the interpretation of the results after estimation. The practice of using theory is much less common in epidemiology, where modelling is rather employed to look at transmission mechanisms (process evaluation) that capture the dynamic nature and spread of – for example - diseases from infectious to susceptible individuals, and to incorporate positive and negative feedback characteristics of infectious processes. Epidemiology also uses theory to inform clinical decision analysis (for example by using decision trees, Markov models to assess the probability of transitions from one state