Ryman Arts: Program Evaluation An Overview of the Program and Assessment of Student Portfolio Work Kylie A. Peppler September 2006 University of California, Los Angeles City of Los Angeles Cultural Affairs Department This study was conducted as part of a project Expanding Opportunities for Talented Urban Youth, funded in part by the Jack Kent Cooke Foundation and the National Endowment for the Arts, a federal agency. Additional support for this project was provided by the Dwight Stuart Youth Foundation, the Ralph M. Parsons Foundation, the City of Los Angeles Cultural Affairs Department, and the Los Angeles County Arts Commission. Ryman Arts Evaluation Report, July 2006 Ryman Arts 2000-2006: An Overview of the Program and an Assessment of Student Portfolio Work Evaluation Report -- September 2006 Kylie A. Peppler UCLA Graduate School of Education and Information Sciences Graduate School of Education & Information Studies University of California, Los Angeles Los Angeles, CA 90095-1522 (310) 455-0785 Copyright © 2006 The Regents of the University of California The findings and opinions expressed in this report are those of the authors and do not necessarily reflect the positions or policies of Ryman Arts or its funders. R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 2 Ryman Arts Evaluation Report, July 2006 SUMMARY OF FINDINGS September 2006 Kylie A. Peppler, Principal Investigator UCLA Graduate School of Education & Information Sciences Introduction After a rigorous application procedure, students are selected from the Southern California region to participate in Saturday studio art classes, taught by professional working artists. Ryman Arts offers a series of three advanced level courses during every 12-week semester: (1) Beginning Classical Drawing; (2) Intermediate Drawing with an Introduction to Watercolors; and (3) Advanced Art: Acrylic Painting. Additionally, a summer course and other special programs are offered. All of the supplies and instruction are offered free of charge and class size is limited to 15-17 students. Students start in the Beginning class and, upon completion of the class, may reapply for the next level. This evaluation had two goals: to provide a longitudinal description of the student participants in the Ryman Arts program; and to assess aspects of student artistic development through portfolio analysis. This external investigation was planned and carried out as part of a grant-funded project, Expanding Opportunities for Talented Urban Youth, and is the first extensive evaluation since the start of the program in 1990. The initial design was worked out in meetings between the Principal Investigator and executives of Ryman Arts in early Fall 2005. During January through June of 2006, the Principal Investigator worked with Ryman Arts staff to gather program data and student work for use in the evaluation. This evaluation focused on providing an accurate description of the program’s participants from cohorts entering the Ryman Arts program from Fall 2000 to Spring 2006. We also took a closer look at a selection of student artwork to parcel out the potential contribution that the Ryman Arts program might have on students’ artistic development over and above the training in which they entered the program. Potentially, this could also point to areas of strength and weaknesses in the program’s instruction as well. We call this evaluation ‘exploratory’ for three main reasons: We wanted to learn both about the impact of Ryman Arts’ teaching on youth and also about the utility and feasibility of the portfolio assessment design. We also wanted to establish a basis for considering future evaluation work within R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 3 Ryman Arts Evaluation Report, July 2006 Ryman Arts. If a more general evaluation of Ryman Arts is carried out in the future, its design and instrumentation may benefit from the preliminary work presented here. Study Participants Ryman Arts has been collecting self-reported information from incoming students in a systematic way since Fall 2000. This data was then used in the following study (n = 740). (In the following passages, the reader should note that the numbers from each entering cohort should not be confused with the total number of students that were serviced in the Ryman Arts program each year – a substantially higher number.) Information included the city, state, zip code, gender, school, age/grade, graduation year, race/ethnicity, and the name of the recommending art teacher, among other information. The program has also documented the courses and instructors that each of the participants enrolled in since 2000 and, to a limited degree, the program has documented absences and dropouts from the program as well. The following table is a summary of this information by incoming cohort year. Overall, the program has tracked over 700 students over a period of six years. Cohort Total Number of Percent Percent of Percent of Known Average Year Ryman New Incoming Incoming Incoming Number of Number of Enrollment Incoming Female Attending Public Living in Mid- Ryman Students in Schools Los Angeles semester Semesters each Cohort County Dropouts 2000-2001 165 93 73.5% 80.6% 94.6% -- 2.1 2001-2002 222 113 64.1% 77.9% 86.7% -- 2.4 2002-2003 242 118 63.6% 84.7% 87.3% 6 2.1 2003-2004 307 116 63.5% 82.8% 92.2% 14 2.4 2004-2005 326 143 62.9% 82.5% 91.6% 4 3.0 2005-2006 290 1 142 58.5% 84.5% 90.1% 5 (1.4) 2 TOTAL/ 1552 740 64.3% 82.7% 90.42% 7.25 2.3 Average Table 1: This is a table of the number of incoming students participating in the Ryman Arts program and the breakdown by gender, the percent attending public school, the percent living in Los Angeles County and the average number of Ryman semesters taken by each cohort. 3 This table demonstrates that the total number of new students entering into the Ryman Arts program has grown steadily, up from 93 to 142 in the most recent cohort. This table also shows that size of the overall program and the number of students being serviced by the program has been steadily growing over the past six years (from 165 to 290 students). Two trends in these tables include the gradual move towards nearly equal numbers of females and males enrolling in the program and the relatively high percentage of public school students serviced in the program. While the vast majority of the students are coming from Los Angeles County, students have also been coming from San Bernardino (n = 10), Orange (n = 54), Ventura (n = 7) and Riverside Counties (n = 1). 1 These numbers are lower because Ryman Arts did not offer summer studio courses. 2 Students in this current cohort have not had an opportunity to take more than two courses. This is the average number of courses taken during the first year in Ryman Arts. 3 There was a high response rate for each of these fields, which resulted in very little (number missing < 5), if any, missing data. The only exception is the number of mid-semester dropouts. Based on the observations of the staff, these numbers seem low. Actual mid-semester dropout rates are probably slightly higher. R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 4 Ryman Arts Evaluation Report, July 2006 On average, students have taken slightly more than two classes, with no discernible longitudinal trend. Notably, the 2004-2005 cohort has taken more classes, on average, than the preceding cohorts with an average of 3 courses per student. Although it’s too early to determine anything, this is an interesting trend and worthy of continued tracking. Voluntary continued enrollment may be a possible indicator of the improved quality of the program over time. In the next graph we take a closer look at the geographic areas of the incoming cohorts of students. Questions that we were concerned with included: Where are the students coming from? Has this changed over time?4 Of concern was whether the Ryman Arts program was sufficiently expanding its recruitment pool and advertising to a wider and more diverse group of students around the Southern California area. Figure 1 demonstrates that the students entering the Ryman Arts program have been coming from a larger number (and arguably more diverse) representation of geographic areas (i.e. zip codes), cities, and schools over the past six years. All three of these indicators show positive upward trends. This is one indication that the Ryman Arts program has been steadily expanding to a more diverse group of communities and servicing a broader group of participants. Number of Zip Codes, Cities, and Schools Represented in I ncoming Class by Year 120 100 99 92 80 81 78 75 73 70 67 68 65 64 Zip Codes 60 Cities 57 56 56 54 Schools 51 49 42 40 20 0 2000 2001 2002 2003 2004 2005 Zip Codes 56 75 73 81 99 92 Cities 42 54 51 57 64 56 Schools 49 65 67 68 78 70 Figure 1: This graph demonstrates that the students entering the Ryman Arts program have been coming from a larger number (and arguably more diverse) representation of geographic areas (zip codes), cities, and schools over the past six years. All three of these indicators show positive upward trends. Additionally, we were interested in the average grade level or year in school when Ryman students entered and exited the program. This is a concern because students who enter the Ryman Arts program in the 12th grade may have too little time to finish the Ryman series before exiting high 4 See Appendix A for a list of all known art teachers that have sent students to the Ryman Arts program.