CENTER FOR STUDENT SUCCESS Ninth Grade Counts Evaluation Report Portland Schools Foundation Summer 2009 Acknowledgments This report has been made possible with the assistance of Nate Waas Shull representing Portland Schools Foundation, and Reese Lord representing the Office of the Mayor of the City of Portland (referred to throughout the report as project directors). They have worked diligently to ensure that all available data, materials, and records were expeditiously provided to the Center staff, and they were always available to provide additional explanation and insight into this vast collection of materials. In instances when surveys, interviews, and/or focus group data required additional clarification, the communication and organization for these was always thorough and timely. Without this level of support and follow through, this report could not have successfully been completed. The extensiveness of this report is an indication of their willingness to commit their time in support of providing precision and thoroughness to this evaluation process. Also, it should be noted that this Ninth Grade Counts project was successfully planned and implemented in an abbreviated timeline. As decisions were made at the highest levels, individuals charged with the responsibility for setting these plans in motion immediately proceeded with the design and implementation required for the successful launch of the Ninth Grade Counts initiative. Ultimately their work resulted in the successful implementation of this broad array of summer transition programs. Again, careful follow-through and clear communication was pervasive throughout implementation. Recognition of this careful, thoughtful, and thorough effort is requisite and fundamental to its continued successful implementation. Center for Student Success Continuing Education/Graduate School of Education Portland State University PO Box 751 Portland, Oregon 97207-0751 503-725-9519 [email protected] Report completed: February 2010 Table of Contents Section 1: Introduction 2-4 1.1 Introduction 2 1.2 Executive Summary 2-3 1.3 Methodology 4 1.4 Evaluation Team Biographies 4-5 Section 2: Program Background 6-15 2.1 Overview 6-9 2.2 Key Materials 10 2.3 Data Sources 11-13 2.4 “Connected by 25” Research Synopsis 13-15 Section 3: Impact of Program on Students 16-30 3.1 2009 Youth Corps Student Survey Results 16-26 3.2 Student Focus Group Results 27-30 Section 4: Impact of Program on Services and Resources 31-50 4.1 Debrief Session – Program Providers 31-34 4.2 2009 Ninth Grade Counts Partner Program Survey Results 35-46 4.3 Project Directors Focus Group Results 47-50 Section 5: Conclusions/Recommendations 51-53 Appendix A Table: Cohort Graduation Rate by District (Class of 2008) Appendix B Total number of completing students by program and district Appendix C Student Focus Group Protocols Appendix D Ninth Grade Counts and Summer Youth Corps Debriefing Session Agenda Appendix E Project Directors’ Focus Group Protocols Center for Student Success Portland State University 1 Section 1: Introduction 1.1 Introduction In October 2009, representatives from the Portland Schools Foundation and the City of Portland Mayor’s Office began meeting with representatives from Portland State University’s Center for Student Success (Graduate School of Education) to begin discussions leading to an agreement to provide third-party evaluation services related to the Ninth Grade Counts summer transition project. The scope of the work became more clearly defined to include a review and analysis of existing data, surveys, archived documents, and available qualitative and quantitative data. Further discussion of the requisite components of the Center’s evaluation process included data to be collected through student and program staff focus groups and interviews. From the compilation and analysis of this extensive information and substantial set of materials, a composite of conclusions, insights, and recommendations for programmatic modifications and thoughtful next steps was to be prepared. It should also be noted that the project directors retained the services of the Northwest Evaluation Association for quantitative analysis of participant demographics and performance data. That analysis is on-going and results will be reported separately from this report. What follows is the summary of findings related to program evaluation work completed primarily in 2009. The report is divided into the following sections for ease of accessing specific findings: overview of the project, impact of program on students, impact of program on services and resources, conclusions and recommendations, and appendices. 1.2 Executive Summary This report examines Ninth Grade Counts, a network of 22 distinct summer transition programs targeting at-risk incoming 9th graders from Multnomah County’s six major school districts during the summer of 2009. The initiative, coordinated by the Portland Schools Foundation (PSF), represents a broad-based effort to increase high school graduation rates by helping more students start high school prepared to succeed. The report also includes examination of Youth Corps – Exploration, an initiative of the Education Cabinet led by the Portland Mayor and the Multnomah County Chair, which provided college and career exposure opportunities for a subset of Ninth Grade Counts participants. Detailed information on findings, recommendations, and methodology is included in this report. Several key items from the report are highlighted below: • Nearly 800 students completed summer transition programs operated by 22 community-based organizations and school districts as part of the Ninth Grade Counts initiative; 420 of those students also took part in weekly career and college exploration trips through the Youth Corps – Exploration. Center for Student Success Portland State University 2 Section 1: Introduction • Approximately 500 participating students received high school credit through their participation in the program. Both students and partner program staff indicated that the opportunity to earn credit is a key motivator for students to participate in the program. • Leaders of the initiative worked with 6 local school districts to identify the students most at-risk of dropping out by applying a common set of academic indicators to the incoming 9th grade class; programs worked to focus recruitment on these students. • The initiative provided partner programs with significant in-kind resources to help increase their capacity. These included 36 full-time AmeriCorps staff, 28 Masters of Arts in Teaching students from Marylhurst University, transit passes, student gift card incentives, and coordination and transportation for weekly career site visits – all totaling more than $550,000 of in-kind support. • Students who participated in the career and college exploration experience reported overwhelmingly positive perceptions and impacts of the program. Students agreed or strongly agreed that the program made them think more about their futures (83%); made them feel better prepared for 9th grade (70%); motivated them to graduate high school (78%); helped them think about new career and postsecondary options (66%); and made them think college was possible for them (82%). • Survey and focus group findings show that students were consistently motivated to participate in the program due to personal factors including the desire to improve themselves, have fun, and gain new experiences; encouragement from a family member or school advisor was also crucial; other motivating factors included to possibility of earning credit, and the promise of a transit pass or gift card incentive • Representatives of the 22 partner programs that form the Ninth Grade Counts network reported very positive perceptions of their participation in the initiative. Staff from these partner programs agreed or strongly agreed that participating in the Ninth Grade Counts network had a positive impact on their programming (95%); increased their focus on incoming 9th graders (75%) and students at high risk of dropping out (60%); enhanced their services to youth (90%); and allowed them to serve more students (65%). 21 of the 22 programs indicated an intention to participate in the initiative again, and 100% of programs agreed that the effort was worthwhile and should be continued. These represent a selection of high-level findings from this evaluation. More detailed findings and recommendations for further development of the initiative are included in the following report. Center for Student Success Portland State University 3 Section 1: Introduction 1.3 Methodology For purposes of this evaluation, our methodology included a study of all evidence and associated materials that were provided; focus groups of students and project staff; in-depth analysis of student and program provider surveys; review of meeting notes and minutes; and the development of an understanding of the project’s historical background to include the work of key initiators and providers. With an emphasis on qualitative research, this study focuses on the accomplishments of the Ninth Grade Counts 2009 project, while endeavoring to answer questions related to program meaning and relevance, program effects and impact, and accomplishment of stated goals and objectives. Following the necessary protocols and design work associated with qualitative research has been important in this evaluation process, and the result is a valid and credible set of conclusions and recommendations