Health Research Policy and Systems Provided by Pubmed Central Biomed Central
Total Page:16
File Type:pdf, Size:1020Kb
View metadata, citation and similar papers at core.ac.uk brought to you by CORE Health Research Policy and Systems provided by PubMed Central BioMed Central Guide Open Access SUPPORT Tools for evidence-informed health Policymaking (STP) 17: Dealing with insufficient research evidence Andrew D Oxman*1, John N Lavis2, Atle Fretheim3 and Simon Lewin4 Address: 1Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway, 2Centre for Health Economics and Policy Analysis, Department of Clinical Epidemiology and Biostatistics, and Department of Political Science, McMaster University, 1200 Main St. West, HSC-2D3, Hamilton, ON, Canada, L8N 3Z5, 3Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway; Section for International Health, Institute of General Practice and Community Medicine, Faculty of Medicine, University of Oslo, Norway and 4Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway; Health Systems Research Unit, Medical Research Council of South Africa Email: Andrew D Oxman* - [email protected]; John N Lavis - [email protected]; Atle Fretheim - [email protected]; Simon Lewin - [email protected] * Corresponding author Published: 16 December 2009 <supplement>byof thethe fundersEuropean had<title> Commission's a role <p>SUPPORT in drafting, 6th Framework revising Tools for or evidence-informedINCOapproving programme, the content health contract of thisPolicymaking 031939. series.</note> The (STP)</p> Norwegian </sponsor> </title> Agency <note>Guides</note> <editor>Andy for Development Oxman Cooperation<url>http://www.biomedcentral.com/content/pdf/1478-4505-7-S1-info.pdf</url> and Stephan (N Hanney</editor>orad), the Alliance <sponsor> for Health <note>This Policy and seriesSystems of articlesResearch, was and prepared the </supplement> Milbank as part Memorial of the SUPPORT Fund provided project, additional which wasfunding. supported None Health Research Policy and Systems 2009, 7(Suppl 1):S17 doi:10.1186/1478-4505-7-S1-S17 This article is available from: http://www.health-policy-systems.com/content/7/S1/S17 © 2009 Oxman et al; licensee BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. In this article, we address the issue of decision making in situations in which there is insufficient evidence at hand. Policymakers often have insufficient evidence to know with certainty what the impacts of a health policy or programme option will be, but they must still make decisions. We suggest four questions that can be considered when there may be insufficient evidence to be confident about the impacts of implementing an option. These are: 1. Is there a systematic review of the impacts of the option? 2. Has inconclusive evidence been misinterpreted as evidence of no effect? 3. Is it possible to be confident about a decision despite a lack of evidence? 4. Is the option potentially harmful, ineffective or not worth the cost? About STP improve the tools in this series is welcome and should be sent to: This article is part of a series written for people responsible for [email protected]. making decisions about health policies and programmes and for those who support these decision makers. The series is intended Scenario to help such people ensure that their decisions are well informed The Ministry of Health is considering strategies to recruit and by the best available research evidence. The SUPPORT tools retain health professionals in underserved rural areas. You have and the ways in which they can be used are described in more been asked to advise the Minister of Health about these strate- detail in the Introduction to this series [1]. A glossary for the gies. You have found many articles describing strategies that entire series is attached to each article (see Additional File 1). have been used in other settings, but no reliable evaluations of Links to Spanish, Portuguese, French and Chinese translations the impacts of such strategies [2]. of this series can be found on the SUPPORT website http:// www.support-collaboration.org. Feedback about how to Page 1 of 7 (page number not for citation purposes) Health Research Policy and Systems 2009, 7(Suppl 1):S17 http://www.health-policy-systems.com/content/7/S1/S17 Background For many questions related to health systems it is not pos- In this article, we present five questions that policymakers sible to find relevant and up-to-date systematic reviews. and those who support them can ask when considering There is widespread recognition, for example, that health scenarios in which there may be insufficient evidence to workers are critical to achieving the Millennium Develop- inform judgements about the impacts of policy and pro- ment Goals (MDGs) and other health goals. Yet despite gramme options. this, an overview of systematic reviews of options to address human resources for health found only a small It is unrealistic to assume that one can predict the impacts amount of high-quality, synthesised research evidence of a health policy or programme with certainty. Many gov- regarding the effects of a few options for the improvement ernance, financial and delivery arrangements have not of human resources for health [5]. Other overviews of been rigorously evaluated. Neither have many of the pro- reviews have found similar gaps [e.g. [6]]. A lack of sys- grammes, services and drugs that these arrangements sup- tematic reviews may not necessarily reflect a lack of evi- port. But policymakers must still make decisions dence. But under such circumstances it is difficult for regardless of the availability (or paucity) of evidence to policymakers to know what evidence is available (see inform such decisions. Table 1, for example). In this article, we focus on decision making undertaken in Rapid assessments may need to be undertaken when time instances in which there is insufficient evidence available or resources are limited. These assessments should be to be able to know whether an option will have the transparent about the methods used, as well as any impor- impacts intended, or whether it may have unintended tant methodological limitations or related uncertainties. (and undesirable) impacts. Common mistakes made They should also address the need for, and urgency of, when there is insufficient evidence at hand include mak- undertaking a full systematic review at a later date [7]. ing assumptions about the evidence without a systematic Consideration should also be given to commissioning a review, confusing a lack of evidence with evidence of no new review whenever a relevant, up-to-date review of effect, assuming that insufficient evidence necessarily good quality is unavailable. Appropriate processes should implies uncertainty about a decision, and the assumption be used, including setting priorities for systematic reviews that it is politically expedient to feign certainty. We [8]. Building and strengthening international collabora- present four questions in this article that can help to avoid tions, such as the Cochrane Collaboration http:// these. www.cochrane.org, can help to avoid unnecessary dupli- cations of effort involved in producing systematic reviews Questions to consider and help to ensure that up-to-date reviews are more read- If there is insufficient evidence at hand to allow one to be ily available. confident about the impacts of implementing a policy or programme option, the following questions can be con- 2. Has inconclusive evidence been misinterpreted as sidered: evidence of no effect? Another common mistake made in instances when evi- 1. Is there a systematic review of the impacts of the dence is inconclusive is the confusion of a lack of evidence option? of an effect with 'evidence of no effect' [9]. It is wrong to claim that inconclusive evidence shows that a policy or 2. Has inconclusive evidence been misinterpreted as evi- programme has had 'no effect'. 'Statistical significance' dence of no effect? should not be confused with importance. 3. Is it possible to be confident about a decision despite a When results are not 'statistically significant' it cannot be lack of evidence? assumed that there was no impact. Typically a cut-off of 5% is used to indicate statistical significance. This means 4. Is the option potentially harmful, ineffective or not that the results are considered to be 'statistically non-sig- worth the cost? nificant' if the analysis shows that differences as large as (or larger than) the observed difference would be expected 1. Is there a systematic review of the impacts of the option? to occur by chance more than one out of twenty times (p The first step in addressing a perceived lack of evidence is ≥0.05). There are, however, two problems with this to find out what evidence is available. It is risky to make assumption. Firstly, the cut-off point of 5% is arbitrary. assumptions about the availability of evidence without Secondly, 'statistically non-significant' results (often mis- referring to systematic reviews. Considerations related to labelled as 'negative'), might or might not be inconclu- finding and critically appraising systematic reviews are sive. Table 2 contains a further discussion of this point addressed in Articles 5 and 6 in this series [3,4]. Page 2 of 7 (page number not for citation purposes) Health Research Policy and Systems 2009, 7(Suppl 1):S17 http://www.health-policy-systems.com/content/7/S1/S17 Table 1: An independent inquiry into inequalities in health - an example of the need for up-to-date systematic reviews to know what evidence there is In 1997, the incoming British Labour government was keen to reduce inequalities in health.