Designing Efficient Systematic Reviews Using Economical Allocation, Creation and Synthesis of Medical Evidence
Total Page:16
File Type:pdf, Size:1020Kb
CHILDREN AND FAMILIES The RAND Corporation is a nonprofit institution that helps improve policy and EDUCATION AND THE ARTS decisionmaking through research and analysis. ENERGY AND ENVIRONMENT HEALTH AND HEALTH CARE This electronic document was made available from www.rand.org as a public service INFRASTRUCTURE AND of the RAND Corporation. TRANSPORTATION INTERNATIONAL AFFAIRS LAW AND BUSINESS Skip all front matter: Jump to Page 16 NATIONAL SECURITY POPULATION AND AGING PUBLIC SAFETY Support RAND SCIENCE AND TECHNOLOGY Browse Reports & Bookstore TERRORISM AND Make a charitable contribution HOMELAND SECURITY For More Information Visit RAND at www.rand.org Explore the Pardee RAND Graduate School View document details Limited Electronic Distribution Rights This document and trademark(s) contained herein are protected by law as indicated in a notice appearing later in this work. This electronic representation of RAND intellectual property is provided for non- commercial use only. Unauthorized posting of RAND electronic documents to a non-RAND website is prohibited. RAND electronic documents are protected under copyright law. Permission is required from RAND to reproduce, or reuse in another form, any of our research documents for commercial use. For information on reprint and linking permissions, please see RAND Permissions. This product is part of the Pardee RAND Graduate School (PRGS) dissertation series. PRGS dissertations are produced by graduate fellows of the Pardee RAND Graduate School, the world’s leading producer of Ph.D.’s in policy analysis. The dissertation has been supervised, reviewed, and approved by the graduate fellow’s faculty committee. Dissertation Designing Efficient Systematic Reviews Using Economical Allocation, Creation and Synthesis of Medical Evidence Mike Scarpati Dissertation Designing Efficient Systematic Reviews Using Economical Allocation, Creation and Synthesis of Medical Evidence Mike Scarpati This document was submitted as a dissertation in June 2014 in partial fulfillment of the requirements of the doctoral degree in public policy analysis at the Pardee RAND Graduate School. The faculty committee that supervised and approved the dissertation consisted of Siddhartha Dalal (Chair), Kanaka Shety, and Jeffrey Wasserman. PARDEE RAND GRADUATE SCHOOL Published 2014 by the RAND Corporation 1776 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138 1200 South Hayes Street, Arlington, VA 22202-5050 4570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665 RAND URL: http://www.rand.org/ To order RAND documents or to obtain additional information, contact Distribution Services: Telephone: (310) 451-7002; Email: [email protected] Contents Abstract 1 Acknowledgments 3 1. Introduction 5 1.1. Overview .................................... 5 1.2. Policy Importance ............................... 6 1.3. The Systematic Review ............................ 9 1.3.1. How Reviews are Utilized ...................... 11 1.3.2. Factors Influencing the Quality of Systematic Reviews ...... 13 1.4. A Brief Economic Model of Systematic Reviews .............. 15 1.4.1. Single Review Model ......................... 15 1.4.2. Allocating Resources to Multiple Reviews ............. 17 1.4.3. Implications .............................. 18 1.4.4. Limitations .............................. 19 1.5. Alternatives to Traditional Systematic Reviews .............. 20 1.6. Initial Recommendations ........................... 21 1.7. Conclusion ................................... 21 2. Estimating the Value of Systematic Reviews: Osteoporosis Case Study 23 2.1. Introduction .................................. 23 2.1.1. Brief Background on Cost Effectiveness Analysis .......... 25 2.1.2. Other Benefits of Systematic Reviews ................ 27 2.2. Flexible Osteoporosis Cost and Utility Simulation (FOCUS) ....... 28 2.2.1. Initialization .............................. 28 2.2.2. Changes During Model Execution .................. 31 2.2.3. Transition Probabilities ........................ 31 2.2.4. Treatment Strategies ......................... 35 2.2.5. Utility and Costs ........................... 37 2.2.6. Summary of Architecture and Parameters Used .......... 38 2.3. Identifying Optimal Treatment Strategies .................. 40 2.4. Costs and Utility Effects of Individual Fractures .............. 41 2.5. Combining the Pieces to Calculate the Population Value of Information . 44 2.5.1. Limitations .............................. 46 2.6. US Benefits of Computer Aided Screening ................. 48 2.7. Conclusion ................................... 49 i Contents Contents 3. Screening Times in Systematic Reviews 51 3.1. Introduction .................................. 51 3.2. Background .................................. 52 3.3. Data ...................................... 53 3.4. Descriptive Statistics ............................. 56 3.5. Factors Associated with Screening Times .................. 59 3.6. Application of Results ............................ 59 3.7. Conclusion ................................... 62 4. De Novo Reviews 65 4.1. The Systematic Review Process ....................... 65 4.2. Domain-Specific Concerns .......................... 66 4.3. Proposed Approach .............................. 67 4.3.1. Notation and Background ...................... 67 4.3.2. Differential Error Costs ........................ 69 4.3.3. Classification as Bipartite Ranking ................. 69 4.3.4. Active Learning ............................ 71 4.4. Performance Evaluation ........................... 74 4.5. Data ...................................... 75 4.6. Results .................................... 76 4.6.1. OHSU Experiments .......................... 76 4.6.2. Low Bone Density Experiments ................... 79 4.7. Conclusions .................................. 82 5. Updating Existing Reviews 83 5.1. Introduction .................................. 83 5.1.1. Prior Work .............................. 84 5.1.2. Different Types of Updates ...................... 84 5.2. Surveillance Updates ............................. 85 5.2.1. Approach ............................... 85 5.2.2. Experiments .............................. 86 5.3. Full Updates .................................. 88 5.3.1. Approach ............................... 88 5.3.2. Experiments .............................. 90 5.4. Discussion ................................... 93 6. Conclusion 95 6.1. Implementation Considerations ....................... 95 6.1.1. Challenges ............................... 95 6.1.2. Opportunities ............................. 97 6.2. How This Work Could Affect Policy Development ............. 98 6.2.1. Reframing the Goals of Evidence Creation and Synthesis ..... 99 6.2.2. Organizational Implications .....................101 6.3. How Stakeholders Benefit from this Work ..................103 ii Contents 6.4. Conclusion ...................................106 Appendix 107 A. Acronym List 109 B. Value of Information and Low Bone Density 111 B.1. Conversion of Maternal Hip Fracture Risk to Parental Hip Fracture Risk 111 B.2. Initialization Regression Results .......................112 B.3. Group Risks from Relative Risks ......................112 B.4. Treatment Strategies Implemented .....................113 B.5. Meta-Analysis Forest Plots ..........................113 B.6. Recent AHRQ EPC Evidence Reports ...................113 C. Timing Details 117 C.1. Empirical Distributions of Screening Times .................117 iii Abstract Medical literature and the actions of policymakers have emphasized the importance of evidence-based medicine in recent years, but basing clinical practice on an exploding base of evidence is challenging. Systematic reviews, which are very resource-intensive, are a crucial channel in the pathway from medical literature to clinical practice. This thesis begins by estimating the value of one systematic review, finding that synthesized evidence regarding treatments to prevent osteoporotic fractures generated a net benefit of approximately $450M. Next, the time taken to screen articles in systematic reviews is analyzed, showing that user interface changes can result in significant reductions in resource requirements. Presenting multiple articles on one screen while reviewing titles leads to a seven-fold reduction in time taken per article. Experience and mental state are also related to screening times, with abstracts reviewed at ideal session lengths requiring 33% less time than those at the beginning of a session. To further increase the speed at which articles can be screened and decrease the cost of preparing systematic reviews, machine learning techniques allow avoidance of up to 80% of articles. When updating an existing review, savings are increased by utilizing the information present in original screening decisions to train the machine learning model. Finally, implementation issues are addressed, paying attention to technical, organiza- tional, and institutional challenges and opportunities. 1 Acknowledgments When beginning the process of writing this thesis, I did not realize the amount of work that would be required of those whom I asked for advisement. I truly appreciate the effort of the people who helped me reach this point. The program itself has given me flexibility to study challenging technical problems while remaining connected to practical considerations. Combining the valuable work and the incredible colleagues, PRGS was ideal. I am deeply indebted to my dissertation chair, Sid Dalal, who has devoted substantial time to developing my research