Evaluation Design Report for the National Cancer Institute's
Total Page:16
File Type:pdf, Size:1020Kb
February 2009 Evaluation Design Report for the National Cancer Institute’s Community Cancer Centers Program (NCCCP) Final Report Prepared for Steve Clauser, PhD Chief, Outcomes Research Branch Applied Research Program Division of Cancer Control and Population Sciences National Cancer Institute Executive Plaza North, Room 4086 6130 Executive Boulevard, (MSC 7344) Bethesda, MD 20892-7344 Prepared by Debra J. Holden, PhD Kelly J. Devers, PhD Lauren McCormack, PhD Kathleen Dalton, PhD Sonya Green, MPH Katherine Treiman, PhD RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709 RTI Project Number 0210903.000.005 RTI Project Number 0210903.0001.005 Evaluation Design Report for the National Cancer Institute’s Community Cancer Centers Program (NCCCP) Final Report February 2009 Prepared for Steve Clauser, PhD Chief, Outcomes Research Branch Applied Research Program Division of Cancer Control and Population Sciences National Cancer Institute Executive Plaza North, Room 4086 6130 Executive Boulevard, (MSC 7344) Bethesda, MD 20892-7344 Prepared by Debra J. Holden, PhD Kelly J. Devers, PhD Lauren McCormack, PhD Kathleen Dalton, PhD Sonya Green, MPH Katherine Treiman, PhD RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709 Contents Section Page Acronyms Error! Bookmark not defined. Executive Summary ES-Error! Bookmark not defined. 1. Introduction Error! Bookmark not defined. 1.1 Overview of the NCCCP Error! Bookmark not defined. 1.2 Summary of the NCCCP Sites Error! Bookmark not defined. 1.3 Process Completed for an Evaluability Assessment Error! Bookmark not defined. 1.3.1 Step 1: Pilot Site Evaluability Assessment (September 2007–June 2008) .................................................. Error! Bookmark not defined. 1.3.2 Step 2: Engage Stakeholders (September 2007–September 2008)Error! Bookmark not defined. 1.3.3 Step 3: Describe the NCCCP Pilot Program (October 2007–June 2008) .................................................. Error! Bookmark not defined. 1.3.4 Step 4: Focus and Finalize Evaluation Plan (March 2008–November 2008) .................................................. Error! Bookmark not defined. 1.4 Overview of this Report Error! Bookmark not defined. 2. Evaluation Overview Error! Bookmark not defined. 2.1 Overview of the NCCCP Evaluation Conceptual Framework Error! Bookmark not defined. 2.1.1 NCCCP Requires Organizational Learning .. Error! Bookmark not defined. 2.1.2 Key Domains of the Conceptual FrameworkError! Bookmark not defined. 2.2 Overarching Evaluation Questions Error! Bookmark not defined. 2.2.1 Evaluation Questions for the National NetworkError! Bookmark not defined. 2.2.2 Evaluation Questions for the Organizational LevelError! Bookmark not defined. 2.2.3 Evaluation Questions for the Program LevelError! Bookmark not defined. 2.2.4 Evaluation Questions for the Patient Level Error! Bookmark not defined. 2.2.5 Evaluation at the Program Component LevelError! Bookmark not defined. 2.2.6 Detailed Evaluation Planning Matrices ...... Error! Bookmark not defined. 2.3 Design Challenges Error! Bookmark not defined. 3. Case Study Error! Bookmark not defined. 3.1 Overview of Case Study Methodology and Related Evaluation Questions Error! Bookmark not defined. 3.2 Case Study Data Collection Error! Bookmark not defined. 3.2.1 Site Visits ............................................. Error! Bookmark not defined. 3.2.2 Document Review ................................. Error! Bookmark not defined. iii 3.2.3 Secondary Data Sources ........................ Error! Bookmark not defined. 3.3 Analysis Plan for Case Study Data Error! Bookmark not defined. 3.3.1 Preparing the Data for Analysis ............... Error! Bookmark not defined. 3.3.2 Cross-Site Analysis ................................ Error! Bookmark not defined. 3.4 Timeline for Case Study Reports Error! Bookmark not defined. 4. Patient Outcomes Error! Bookmark not defined. 4.1 Patient Survey Error! Bookmark not defined. 4.1.1 Survey Objectives ................................. Error! Bookmark not defined. 4.1.2 Data Collection ..................................... Error! Bookmark not defined. 4.1.3 Analysis Plan ........................................ Error! Bookmark not defined. 4.2 Patient (and Caregiver) Focus Groups Error! Bookmark not defined. 5. Economic Study Error! Bookmark not defined. 5.1 Overview Error! Bookmark not defined. 5.2 Micro-Cost Study Error! Bookmark not defined. 5.2.1 Types of Data Collected ......................... Error! Bookmark not defined. 5.2.2 Cost Assessment Tool ............................ Error! Bookmark not defined. 5.2.3 Data Collection Logistics......................... Error! Bookmark not defined. 5.2.4 Output Measures for Cost-Effectiveness ... Error! Bookmark not defined. 5.2.5 Analysis Plan ........................................ Error! Bookmark not defined. 5.2.6 Timing ................................................. Error! Bookmark not defined. 5.3 Strategic Case Study Error! Bookmark not defined. 6. Overall Analysis Plan Error! Bookmark not defined. 6.1 Statement of Hypotheses Error! Bookmark not defined. 6.1.1 Environmental-Level Hypotheses ............. Error! Bookmark not defined. 6.1.2 Organizational-Level Hypotheses ............. Error! Bookmark not defined. 6.1.3 Patient-Level Hypotheses ....................... Error! Bookmark not defined. 6.2 Measures Development Error! Bookmark not defined. 6.2.1 Dependent Measures Development .......... Error! Bookmark not defined. 6.2.2 Independent Measures Development ....... Error! Bookmark not defined. 6.3 Data Analysis Overview Error! Bookmark not defined. 6.3.1 Development of Composite Measures ....... Error! Bookmark not defined. 6.3.2 Qualitative Comparative Analysis ............ Error! Bookmark not defined. 6.4 Finalizing the Overall Analysis Plan Error! Bookmark not defined. 7. Deliverables and Timelines Error! Bookmark not defined. References R-Error! Bookmark not defined. Appendixes iv A NPAC and EOC Members A-1 B Site Visit Interview Discussion Guides B-1 C Evaluation Planning Matrix by Intervention Levels C-1 D Evaluation Planning Matrix by Program Components D-1 E Planning Call Discussion Guide for Case Study Site Visits E-1 F Qualitative Coding Dictionary for Site Visit Interview Data F-1 G Sample Tables for Site-Level Data Books G-1 H Concept Paper for Addressing the Strategic Case for Site Participation in NCCCP H-1 v vi Figures Number Page 2-1. NCCCP Pilot Program Model ................................... Error! Bookmark not defined. 2-2. NCCCP Evaluation Conceptual Framework ................ Error! Bookmark not defined. 2-3. Model of Team Effectiveness .................................. Error! Bookmark not defined. 3-1. NVivo Screenshot ................................................. Error! Bookmark not defined. 5-1. Schematic of the Cost Assessment Tool ................... Error! Bookmark not defined. 5-2. Sample Presentation: Shell Figure for Distribution of Core Component Expenditures, by Site and Funding Source ............... Error! Bookmark not defined. 6-1. NCCCP Evaluation Conceptual Framework ................ Error! Bookmark not defined. vii Tables Number Page 1-1. NCCCP Site Overview ............................................ Error! Bookmark not defined. 2-1. Prioritized Primary and Secondary Evaluation Questions across the First Four Levels of Program Intervention .............................. Error! Bookmark not defined. 2-2. Primary Evaluation Questions and Illustrative Methods and Outcomes for Each NCCCP Component ............................................... Error! Bookmark not defined. 3-1. Primary Purposes of the Case Study and Corresponding Levels of NCCCP Implementation and Primary Evaluation Questions ... Error! Bookmark not defined. 3-2. Data Sources and Sample Data Variables for the NCCCP Case StudyError! Bookmark not defined. 3-3. Key Words or Domains for Each Level of Evaluation to Cover for Each Year of Site Visits ............................................................ Error! Bookmark not defined. 3-4. Requested Interviewees for Year 1 Site Visits by Organizational Role and Suggested Interview Length .................................. Error! Bookmark not defined. 3-5. Detailed Timeline of Case Study Activities and ReportingError! Bookmark not defined. 4-1. Number of Completed Interviews Necessary to Achieve Acceptable Power for a Significance Level of 5% ........................................ Error! Bookmark not defined. 4-2. Summary of Dependent and Independent Variables .. Error! Bookmark not defined. 4-3. Scoring for Program Development Variables ............ Error! Bookmark not defined. 4-4. NCCCP Patient Survey: Hypotheses about Relationships between Independent and Outcome Variables at Baseline ......................... Error! Bookmark not defined. 4-5 Selected Independent Variables to Model Overall Rating of Care or to Predict the Probability of Being Informed about a Clinical Trial (Sample Table)Error! Bookmark not defined. 5-1. CAT Activity-Based Cost Allocations within Program ComponentsError! Bookmark not defined. 5-2. Summary of Year 1 Invoiced Costs, by Type of Expenditure and Core Component Activity .............................................. Error! Bookmark not defined. 5-3. Sample Presentation: Shell Table for Documented Costs Attributable to NCCCP Implementation, by Core Component and Funding SourceError! Bookmark not defined. 5-4. Examples of Cost-Effectiveness Ratios for ConsiderationError! Bookmark not defined. 6-1. NCCCP Deliverables and Metrics for Each