Final Report Evaluation of AHRQ’s National Guideline Clearinghouse™ (NGC) Agency for Healthcare Research and Quality August 31, 2011 AFYA, Inc. 8101 Sandy Spring Road, Third Floor Laurel, MD 20707 Final Contract Report: NGC Evaluation Final Submitted: 9/29/2011 Final Contract Report Evaluation of AHRQ’s National Guideline Clearinghouse™ (NGC) Prepared for: Agency for Healthcare Research and Quality U.S. Department of Health and Human Services 540 Gaither Road Rockville, MD 20850 www.ahrq.gov Prepared by: AFYA, Inc. Analytical and Technical Support Division 8101 Sandy Spring Road, Third Floor Laurel, MD 20707 Authors: Michelle Bieber-Tregear AFYA, Inc. Jenice James AFYA, Inc. Debra Dekker AFYA, Inc. Craig Dearfield AFYA, Inc. Jaclyn Marshall Lewin Group Jacob Epstein Lewin Group Carol Simon Lewin Group Financial Declarations None of the authors has any affiliations or financial involvement that conflict with the material presented in this report. Policy Statement This report was prepared by AFYA, Inc. and The Lewin Group under Contract No. 4203; Order No. 3, entitled “Evaluation of the National Guideline Clearinghouse™ (NGC),” with the Department of Health and Human Services’ Agency for Healthcare Research and Quality. Table of Contents List of Figures............................................................................................................................ 5 List of Tables ............................................................................................................................. 6 AHRQ Contract No. 4203; Order No. 3 2 Final Contract Report: NGC Evaluation Final Submitted: 9/29/2011 Acknowledgements .................................................................................................................... 7 Preface ....................................................................................................................................... 8 A. Project Scope .............................................................................................................................. 8 B. Organization of Report ................................................................................................................ 8 C. Disclaimer .................................................................................................................................. 8 Executive Summary .................................................................................................................. 9 A. Purpose of Evaluation ................................................................................................................. 9 B. Analytic Methods........................................................................................................................ 9 C. Key Findings and Potential Opportunities for AHRQ ................................................................ 10 I. Background .......................................................................................................................... 17 A. Development of NGC ............................................................................................................... 17 B. Purpose of Evaluation ............................................................................................................... 18 C. Questions of Interest ................................................................................................................. 19 D. NGC Stakeholder Groups .......................................................................................................... 21 II. Methodology .................................................................................................................... 22 A. Establishing a Participatory Evaluation Team ............................................................................ 23 B. Evaluation Approach ................................................................................................................ 23 C. Data Collection and Analyses.................................................................................................... 24 1. Survey .................................................................................................................................. 25 2. Focus Groups and Key Informant Interviews ........................................................................ 28 D. Evaluation Limitations .............................................................................................................. 29 III. General Findings ......................................................................................................... 30 A. Characteristics of Survey Participants ....................................................................................... 30 1. Occupation Characteristics .................................................................................................... 31 2. Geographical Distribution of Survey Respondents................................................................. 32 B. Characteristics of Focus Group and Key Informant Participants ................................................ 33 C. Awareness and Use of the NGC Web site .................................................................................. 34 1. 2001 Evaluation of the NGC Web Site .................................................................................. 34 2. Current (2010-11) Evaluation of the NGC Web Site .............................................................. 35 D. Use of Other Guideline Sources ................................................................................................ 36 1. Survey Findings .................................................................................................................... 36 2. Qualitative Findings .............................................................................................................. 38 E. Satisfaction with NGC Compared to other Guideline Sources.................................................... 40 1. Survey Findings .................................................................................................................... 40 2. Qualitative Findings .............................................................................................................. 41 F. NGC Trustworthiness and Appropriateness of the NGC Inclusion Criteria ................................ 43 1. Survey Findings .................................................................................................................... 43 2. Qualitative Findings .............................................................................................................. 47 IV. Stakeholder-Specific Findings—Impact, Influence, and Other Key Findings .......... 52 A. Guideline Developers ................................................................................................................ 53 1. NGC’s Influence and Impact for Guideline Developers ......................................................... 53 2. Use of Other Research Tools................................................................................................. 56 3. Guideline Developer Experience with NGC (Submitters Only) ............................................. 58 4. Comparison of Guideline Developers who Submit Guidelines to NGC and Those who Do NOT Submit Guidelines to NGC ................................................................................................... 63 B. Healthcare Providers (Physicians, Nurses, and other Healthcare Providers) ............................... 67 1. Healthcare Providers ........................................................... 67 2. Health Providers ............................................................. 68 C. Informaticians ........................................................................................................................... 68 AHRQ Contract No. 4203; Order No. 3 3 Final Contract Report: NGC Evaluation Final Submitted: 9/29/2011 1. Informaticians ................................................................................... 69 2. Informaticians ................................................................. 72 D. Medical Librarians .................................................................................................................... 73 1. Use of NGC – Medical Librarians ......................................................................................... 74 2. NGC’s Influence and Impact – Medical Librarians ................................................................ 74 3. Desired Enhancements to NGC – Medical Librarians ............................................................ 76 E. Researchers ............................................................................................................................... 76 1. NGC’s Influence Researchers ....................................................................... 77 2. Use of other Research Tools – Researchers ........................................................................... 78 F. Measure Developers.................................................................................................................. 79 1. Use of Guidelines and NGC in Measure Development .......................................................... 79 2. NGC’s Influence and Impact – Measure Developers ............................................................. 81 G. Policymakers, Purchasers, Quality Improvement Specialists, Other ..........................................