Comparative Effectiveness Review Number 204 Methods for Evaluating Natural Experiments in Obesity: Systematic Evidence Review e Comparative Effectiveness Review Number 204 Methods for Evaluating Natural Experiments in Obesity: Systematic Evidence Review Prepared for: Agency for Healthcare Research and Quality U.S. Department of Health and Human Services 5600 Fishers Lane Rockville, MD 20857 www.ahrq.gov Contract No. 290-2012-00007-I Prepared by: Johns Hopkins University Evidence-based Practice Center Baltimore, MD Investigators: Wendy L. Bennett, M.D., M.P.H. Lawrence J. Cheskin, M.D. Renee F. Wilson, M.S. Allen Zhang, B.S. Eva Tseng, M.D., M.P.H. Oluwaseun Shogbesan, M.D. Emily A. Knapp, M.H.S. Elizabeth A. Stuart, Ph.D. Eric B. Bass, M.D., M.P.H. Hadi Kharrazi, M.D. AHRQ Publication No. 18-EHC006-EF December 2017 Purpose of Review To characterize studies of programs and policies in obesity prevention and control in terms of data sources, data linkages, measures reported, study designs, and analytic approaches, and to identify needed methodological advances. Key Messages • Relevant programs, policies, or built environment changes were evaluated in 156 natural experiments, 118 experimental studies, and 20 other studies. • Criteria for a data system (source exists, is available for research, is sharable, and has outcomes of interest) were met by 106 data sources. • Thirty-seven percent of U.S. data systems were linked to secondary data. • Outcome measures included dietary behavior (148 studies), physical activity (152 studies), childhood weight (112 studies), and adult weight (32 studies). • Natural experiments most commonly used regression models comparing exposed and unexposed groups at one time. • Natural experiments generally had moderate risk of selection bias and high risk of bias for losses to follow-up. • Research could be advanced by more use of data dictionaries, reporting standards on data linkage, long-term obesity-related outcomes, and study designs with multiple pre- and post- exposure time points. ii This report is based on research conducted by the Johns Hopkins University Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2012-00007-I). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services. None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report. The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients. This report is made available to the public under the terms of a licensing agreement between the author and the Agency for Healthcare Research and Quality. This report may be used and reprinted without permission except those copyrighted materials that are clearly noted in the report. Further reproduction of those copyrighted materials is prohibited without the express permission of copyright holders. AHRQ or U.S. Department of Health and Human Services endorsement of any derivative products that may be developed from this report, such as clinical practice guidelines, other quality enhancement tools, or reimbursement or coverage policies, may not be stated or implied. This report may periodically be assessed for the currency of conclusions. If an assessment is done, the resulting surveillance report describing the methodology and findings will be found on the Effective Health Care Program Web site at www.effectivehealthcare.ahrq.gov. Search on the title of the report. Persons using assistive technology may not be able to fully access information in this report. For assistance contact [email protected]. Suggested citation: Bennett WL, Cheskin LJ, Wilson RF, Zhang A, Tseng E, Shogbesan O, Knapp EA, Stuart EA, Bass EB, Kharazzi H. Methods for Evaluating Natural Experiments in Obesity: Systematic Evidence Review. Comparative Effectiveness Review No. 204. (Prepared by the Johns Hopkins University Evidence-based Practice Center under Contract No. 290-2012- 00007-I.) AHRQ Publication No. 18-EHC006-EF. Rockville, MD: Agency for Healthcare Research and Quality; December 2017. Posted final reports are located on the Effective Health Care Program search page. DOI: http://doi.org/10.23970/AHRQEPCCER204. iii Preface The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-based Practice Centers (EPCs), sponsors the development of systematic reviews to assist public- and private-sector organizations in their efforts to improve the quality of health care in the United States. These reviews provide comprehensive, science-based information on common, costly medical conditions, and new health care technologies and strategies. Systematic reviews are the building blocks underlying evidence-based practice; they focus attention on the strength and limits of evidence from research studies about the effectiveness and safety of a clinical intervention. In the context of developing recommendations for practice, systematic reviews can help clarify whether assertions about the value of the intervention are based on strong evidence from clinical studies. For more information about AHRQ EPC systematic reviews, see www.effectivehealthcare.ahrq.gov/reference/purpose.cfm. AHRQ expects that these systematic reviews will be helpful to health plans, providers, purchasers, government programs, and the health care system as a whole. Transparency and stakeholder input are essential to the Effective Health Care Program. Please visit the Web site (www.effectivehealthcare.ahrq.gov) to see draft research questions and reports or to join an email list to learn about new program products and opportunities for input. If you have comments on this systematic review, they may be sent by mail to the Task Order Officer named below at: Agency for Healthcare Research and Quality, 5600 Fishers Lane, Rockville, MD 20857, or by email to [email protected]. Gopal Khanna, M.B.A. Arlene S. Bierman, M.D., M.S. Director Director Agency for Healthcare Research and Quality Center for Evidence and Practice Improvement Agency for Healthcare Research and Quality Stephanie Chang, M.D., M.P.H. Lionel L. Bañez, M.D. Director Task Order Officer Evidence-based Practice Center Program Center for Evidence and Practice Center for Evidence and Practice Improvement Improvement Agency for Healthcare Research and Quality Agency for Healthcare Research and Quality iv Acknowledgments The authors gratefully acknowledge the following individuals for their contributions to this project: Jessica Gayleard, Ritu Sharma, Catalina Suarez, and Lisa Wilson. Technical Expert Panel In designing the study questions and methodology at the outset of this report, the EPC consulted several technical and content experts. Broad expertise and perspectives were sought. Divergent and conflicted opinions are common and perceived as healthy scientific discourse that results in a thoughtful, relevant systematic review. Therefore, in the end, study questions, design, methodological approaches, and/or conclusions do not necessarily represent the views of individual technical and content experts. Technical Experts must disclose any financial conflicts of interest greater than $10,000 and any other relevant business or professional conflicts of interest. Because of their unique clinical or content expertise, individuals with potential conflicts may be retained. The TOO and the EPC work to balance, manage, or mitigate any potential conflicts of interest identified. The list of Technical Experts who participated in developing this report follows: David Allison, Ph.D.* Lisa Sanbonmatsu, Ph.D.* Dean and Provost Professor National Bureau of Economic Research Indiana University School of Public Health Cambridge, MA Bloomington, IN Brandon Simms Cheryl Anderson, Ph.D., M.P.H.* Greater Gethsemane Missionary Baptist Associate Professor Church Department of Family Medicine and Public Baltimore, MD Health University of California, San Diego Xuemei Zhu, Ph.D.* La Jolla, CA Associate Professor University of Texas College of Architecture Donna Coffman, Ph.D.* Texas A&M University Dept. of Epidemiology and Biostatistics College Station, TX Temple University Philadelphia, PA Trina Histon, Ph.D. Senior Principal Consultant Kaiser Permanente Oakland, CA *Provided input on Draft Report. v Peer Reviewers Prior to publication of the final evidence report, EPCs sought input from independent Peer Reviewers without financial conflicts of interest. However, the conclusions and synthesis of the scientific literature presented in this report do not necessarily represent the views of individual reviewers. Peer Reviewers must disclose any financial conflicts of interest greater than $10,000 and any other relevant business or professional