Ryman Arts: Program Evaluation

An Overview of the Program and Assessment of Student Portfolio Work

Kylie A. Peppler

September 2006 University of California, Los Angeles

City of Los Angeles Cultural Affairs Department

This study was conducted as part of a project Expanding Opportunities for Talented Urban Youth, funded in part by the Jack Kent Cooke Foundation and the National Endowment for the Arts, a federal agency. Additional support for this project was provided by the Dwight Stuart Youth Foundation, the Ralph M. Parsons Foundation, the City of Los Angeles Cultural Affairs Department, and the Los Angeles County Arts Commission.

Ryman Arts Evaluation Report, July 2006

Ryman Arts 2000-2006: An Overview of the Program and an Assessment of Student Portfolio Work

Evaluation Report -- September 2006

Kylie A. Peppler UCLA Graduate School of Education and Information Sciences

Graduate School of Education & Information Studies University of California, Los Angeles

Los Angeles, CA 90095-1522 (310) 455-0785

The findings and opinions expressed in this report are those of the authors and do not necessarily reflect the positions or policies of Ryman Arts or its funders.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 2

Ryman Arts Evaluation Report, July 2006

SUMMARY OF FINDINGS

September 2006

Kylie A. Peppler, Principal Investigator UCLA Graduate School of Education & Information Sciences

Introduction After a rigorous application procedure, students are selected from the Southern California region to participate in Saturday studio art classes, taught by professional working artists. Ryman Arts offers a series of three advanced level courses during every 12-week semester: (1) Beginning Classical Drawing; (2) Intermediate Drawing with an Introduction to Watercolors; and (3) Advanced Art: Acrylic Painting. Additionally, a summer course and other special programs are offered. All of the supplies and instruction are offered free of charge and class size is limited to 15-17 students. Students start in the Beginning class and, upon completion of the class, may reapply for the next level. This evaluation had two goals: to provide a longitudinal description of the student participants in the Ryman Arts program; and to assess aspects of student artistic development through portfolio analysis.

This external investigation was planned and carried out as part of a grant-funded project, Expanding Opportunities for Talented Urban Youth, and is the first extensive evaluation since the start of the program in 1990. The initial design was worked out in meetings between the Principal Investigator and executives of Ryman Arts in early Fall 2005. During January through June of 2006, the Principal Investigator worked with Ryman Arts staff to gather program data and student work for use in the evaluation. This evaluation focused on providing an accurate description of the program’s participants from cohorts entering the Ryman Arts program from Fall 2000 to Spring 2006. We also took a closer look at a selection of student artwork to parcel out the potential contribution that the Ryman Arts program might have on students’ artistic development over and above the training in which they entered the program. Potentially, this could also point to areas of strength and weaknesses in the program’s instruction as well.

We call this evaluation ‘exploratory’ for three main reasons: We wanted to learn both about the impact of Ryman Arts’ teaching on youth and also about the utility and feasibility of the portfolio assessment design. We also wanted to establish a basis for considering future evaluation work within

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 3

Ryman Arts Evaluation Report, July 2006

Ryman Arts. If a more general evaluation of Ryman Arts is carried out in the future, its design and instrumentation may benefit from the preliminary work presented here.

Study Participants Ryman Arts has been collecting self-reported information from incoming students in a systematic way since Fall 2000. This data was then used in the following study (n = 740). (In the following passages, the reader should note that the numbers from each entering cohort should not be confused with the total number of students that were serviced in the Ryman Arts program each year – a substantially higher number.) Information included the city, state, zip code, gender, school, age/grade, graduation year, race/ethnicity, and the name of the recommending art teacher, among other information. The program has also documented the courses and instructors that each of the participants enrolled in since 2000 and, to a limited degree, the program has documented absences and dropouts from the program as well. The following table is a summary of this information by incoming cohort year. Overall, the program has tracked over 700 students over a period of six years.

Cohort Total Number of Percent Percent of Percent of Known Average Year Ryman New Incoming Incoming Incoming Number of Number of Enrollment Incoming Female Attending Public Living in Mid- Ryman Students in Schools Los Angeles semester Semesters each Cohort County Dropouts 2000-2001 165 93 73.5% 80.6% 94.6% -- 2.1 2001-2002 222 113 64.1% 77.9% 86.7% -- 2.4 2002-2003 242 118 63.6% 84.7% 87.3% 6 2.1 2003-2004 307 116 63.5% 82.8% 92.2% 14 2.4 2004-2005 326 143 62.9% 82.5% 91.6% 4 3.0 2005-2006 290 1 142 58.5% 84.5% 90.1% 5 (1.4) 2

TOTAL/ 1552 740 64.3% 82.7% 90.42% 7.25 2.3 Average

Table 1: This is a table of the number of incoming students participating in the Ryman Arts program and the breakdown by gender, the percent attending public school, the percent living in Los Angeles County and the average number of Ryman semesters taken by each cohort. 3

This table demonstrates that the total number of new students entering into the Ryman Arts program has grown steadily, up from 93 to 142 in the most recent cohort. This table also shows that size of the overall program and the number of students being serviced by the program has been steadily growing over the past six years (from 165 to 290 students). Two trends in these tables include the gradual move towards nearly equal numbers of females and males enrolling in the program and the relatively high percentage of public school students serviced in the program. While the vast majority of the students are coming from Los Angeles County, students have also been coming from San Bernardino (n = 10), Orange (n = 54), Ventura (n = 7) and Riverside Counties (n = 1).

1 These numbers are lower because Ryman Arts did not offer summer studio courses. 2 Students in this current cohort have not had an opportunity to take more than two courses. This is the average number of courses taken during the first year in Ryman Arts. 3 There was a high response rate for each of these fields, which resulted in very little (number missing < 5), if any, missing data. The only exception is the number of mid-semester dropouts. Based on the observations of the staff, these numbers seem low. Actual mid-semester dropout rates are probably slightly higher.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 4

Ryman Arts Evaluation Report, July 2006

On average, students have taken slightly more than two classes, with no discernible longitudinal trend. Notably, the 2004-2005 cohort has taken more classes, on average, than the preceding cohorts with an average of 3 courses per student. Although it’s too early to determine anything, this is an interesting trend and worthy of continued tracking. Voluntary continued enrollment may be a possible indicator of the improved quality of the program over time.

In the next graph we take a closer look at the geographic areas of the incoming cohorts of students. Questions that we were concerned with included: Where are the students coming from? Has this changed over time?4 Of concern was whether the Ryman Arts program was sufficiently expanding its recruitment pool and advertising to a wider and more diverse group of students around the Southern California area. Figure 1 demonstrates that the students entering the Ryman Arts program have been coming from a larger number (and arguably more diverse) representation of geographic areas (i.e. zip codes), cities, and schools over the past six years. All three of these indicators show positive upward trends. This is one indication that the Ryman Arts program has been steadily expanding to a more diverse group of communities and servicing a broader group of participants.

Number of Zip Codes, Cities, and Schools Represented in I ncoming Class by Year

120

100 99 92

80 81 78 75 73 70 67 68 65 64 Zip Codes 60 Cities 57 56 56 54 Schools 51 49 42 40

0 2000 2001 2002 2003 2004 2005 Zip Codes 56 75 73 81 99 92 Cities 42 54 51 57 64 56 Schools 49 65 67 68 78 70

Figure 1: This graph demonstrates that the students entering the Ryman Arts program have been coming from a larger number (and arguably more diverse) representation of geographic areas (zip codes), cities, and schools over the past six years. All three of these indicators show positive upward trends.

Additionally, we were interested in the average grade level or year in school when Ryman students entered and exited the program. This is a concern because students who enter the Ryman Arts program in the 12th grade may have too little time to finish the Ryman series before exiting high

4 See Appendix A for a list of all known art teachers that have sent students to the Ryman Arts program. For privacy purposes, this section of the report will not be generally distributed. See Appendix B for a list of all known schools that have sent students to the Ryman Arts program. The tables show wide but not very deep recruitment. Very few teachers and/or school send more than 10 students. Although these trends hold up as we look at the school level, this trend is reflected even more strongly when we look at the individual teachers.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 5

Ryman Arts Evaluation Report, July 2006 school. However, many high school art teachers were known to nominate successful upper class, especially 12th graders, for the Ryman Arts program because they presumed that these students would have a better chance in the application pool. In recent years, there has been a targeted recruitment of younger students into the program. Therefore, it’s not surprising that Figure 2 demonstrates that the number of students at each grade level entering Ryman Arts at the start of the program has been changing. This graph (Figure 2) shows the percentage of students at each grade level (9-12) in the incoming cohorts. The data show that the percentage of incoming 9th and 10th graders has been on the rise, while the percentage of incoming 12th graders has been declining since 2000. This indicates positive results of the targeted recruitment of younger students into the program.

Percentage of I ncoming Students by Grade Level

60.0%

12th grade 50.0%

40.0%

9 10 30.0% 11 12

20.0%

th th Percentage of Students by Grade Level 9 and 10 grades 10.0%

0.0% 2000 2001 2002 2003 2004 2005

Figure 2: This graph shows the percentage of students at each grade level (9-12) in the incoming cohorts. The data shows that the percentage of incoming 9th and 10th graders has been rising, while the percentage of incoming 12th graders has been declining since 2000.

There has been an increasing concern about creating a racially, ethnically, and economically diverse incoming cohort and, consequently, students have been asked to self-report their racial and ethnic backgrounds for this purpose (see Figures 3A and 3C). Data collection began in 2001 (so there is no data for any prior cohorts) and has only been systematically collected since 2003 (hence, the relatively high percentage of missing data from 2001-2003). Interpretations regarding trends in the data should be made cautiously due to large amount of missing data in the early years and the difference in reporting methods that began in Fall 2003.

To better determine whether racial and ethnic groups are over- or under-represented within Ryman Arts, we compared our figures to the most recent data published by the California Department of Education on the ethnicity of Los Angeles County Public Schools5 (see Figure 3B). Figure 3C shows the percentage of students by ethnicity for Ryman Arts during the same year as the Los Angeles County Public Schools data was generated. Statistically significant differences were calculated comparing the two proportions. Strong evidence exists that three groups of students enrolled at

5 The source of this data is the California Department of Education, Educational Demographics Office (CBEDS, sifade04 7/26/05). This information can be found at: www.ed-data.k12.ca.us.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 6

Ryman Arts Evaluation Report, July 2006 significantly higher numbers in Ryman Arts than in Los Angeles County Public Schools in 2004-2005: Asian students (p < .01); students of ethnicities other than Asian, Latino, Caucasian, or African- American (p < .01); and students of multiple ethnicities (p < .01). The other differences in racial/ethnic make-up were not found to be statistically significant (p > .05 for all tests).

Racial/ Ethnic Composition of I ncoming Cohorts by Year

Percentage (%) 20

0 2001-2002 2002-2003 2003-2004 2004-2005 2005-2006 African-American 3.5 2.5 4 2.9 2.1 Asian 8.8 22 25 31.4 31.9 Caucasian 4.4 9.3 26 22.7 21.3 Latino 8.8 16.9 34 27.9 24.5 Eastern European 0 4.2 2.6 1.4 0 Mutiple Ethnicity 0 1.7 11 9.3 10.6 Other 0.9 0.8 8.4 13.7 4.3 Missing Data 73.5 42.4 0 0 5.3

Figure 3A: This graph illustrates the known racial and ethnic breakdown of the incoming cohorts by year. Longitudinal trends should be interpreted cautiously due to the large amount of missing data from 2001 and 2002. Starting in 2003, Ryman Arts has anonymously collected this information from all of its participants; hence lower percentages of missing data since this time.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 7

Ryman Arts Evaluation Report, July 2006

Students by Ethnicity for Students by Ethnicity for Ryman Arts, Los Angeles County, 2004- 2004-2005

1% 10% ** 9% 3% 3% African-American African-American 8% Asian 14% Asian ** 28%**

White/Caucasian White/Caucasian

16% Hispanic/Latino Hispanic/Latino

Other Other 62% 25% Multiple/No Multiple/No Response 21% Response

Figures 3B and 3C: Figure 3B is a pie chart of the percentage of students by ethnicity for Los Angeles County Public Schools during the 2004-2005 school year (n = 1,734,040). The source of this data is the California Department of Education, Educational Demographics Office (CBEDS, sifade04 7/26/05). This information can be found at: www.ed-data.k12.ca.us. For comparison, figure 3C shows the percentage of students by ethnicity for Ryman Arts during the same year (n = 326). Statistically significant differences were calculated comparing the two proportions and are marked with an asterisk (** = p < .001).

For some time now, Ryman Arts has been trying to estimate the average income of the families sending students to the program. However, asking students and families to self-report this data could be potentially problematic. Some families may misinterpret this effort as being part of the selection criteria, which may subsequently influence their decision to participate in the program. In addition, self-reports on the family’s yearly income may be inaccurate or incomplete. As a result, we have decided to use information based upon the students’ home address to give us a better picture of the Ryman students’ economic backgrounds. Using the students’ home zip codes, we were able to locate the average income for each zip code.6 Since zip codes also represent fairly small areas in Los Angeles and Orange County, this generated very specific information about the geographic and economic areas that the Ryman students were coming from. We also argue that average income by zip code may be a good indicator of the types and extent of social networks, neighborhood opportunities, and schools to which a given student might have access. Using the information generated from the zip codes, we have divided the range of average income into four income groups:

1) Average incomes that would qualify for a free lunch in the public school system and are below the Los Angeles ‘Basic Family Budget’ for a family of five7 (< $30,420);

6 The data used in this study was published in 2000. This information is available at American FactFinder.com, published by the U.S. Census Bureau: http://factfinder.census.gov/home/saff/main.html?_lang=en 7 This figure was generated using the basic family budget calculator created by the Economic Policy Institute (www.epinet.org). They recommend that, for a family of five in the Los-Angeles to Long Beach area, the income level would be $65,484. They argue, “The federal poverty line has traditionally been used to measure whether families have incomes high enough to enable them to meet basic needs. Yet most researchers now agree that a ‘poverty line’ income is not sufficient to

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 8

Ryman Arts Evaluation Report, July 2006

2) Average incomes that would qualify for a Reduced Price Lunch in the public school system and are below the Los Angeles ‘Basic Family Budget’ (between $30,421-$43, 290); 3) Average incomes that are below the Los Angeles ‘Basic Family Budget’ but would not qualify for a free or reduced price lunch (between $43,291-$65,484); and 4) Average incomes that are above the Los Angeles basic family budget (>$65,485).

In Figure 4, we see that the vast majority (83%) of Ryman students are coming from zip code areas where the average income is below the Los Angeles ‘Basic Family Budget’ for a family of 5 (< $65,484) and 40% are coming from areas that would qualify for either a free or reduced price lunch. (See Figure 4 for the exact break downs by average income category.) This is a good indication that Ryman Arts is servicing a high volume of low-income students.

Percent of Ryman Arts Students by Average Income* (2000-2006) Qualifies for Free Lunch/Below LA Basic Family Budget (< $30,420)

4% Qualifies for Reduced Price 18% 13% Lunch/Below LA Basic Family Budget ($30,421 - $43,290) Below Recommendation for LA Basic Family Budget ($43,291 - $65,484)

22% Middle Income/ Above Recommendation for LA Basic Family Budget (>$65,485) 43% Missing Data

Figure 4: This pie chart illustrates the percent of Ryman Arts students by average income. (*NOTE: The average income from the student’s home zip code was used to generate what we are abbreviating as “average income” for this analyses.)

Targeted Recruitment There has also been an effort to target a group of schools in the areas surrounding the University of Southern California (USC) campus where classes are held that predominantly serves low- income students of color. Ryman Arts has been targeting a set of six high-poverty, predominantly African-American and Latino high schools in the surrounding area through a special recruitment component of the project Expanding Opportunities for Talented Urban Youth. To better examine the results of this targeted recruitment program, we took a closer look at this group of students. Figure 5A demonstrates that this effort to recruit students from the target schools has been successful with a large number of students (n=17) entering in the fall of the most recent cohort, in 2005. However, as we know from the previous data, overall numbers of students entering the Ryman Arts program have also been on the rise. To determine whether or not the students coming from the target schools constitute support most working families. ‘Basic family budgets,’ individualized for communities nationwide and for type of family…offer a realistic measure of the income required to have a safe and decent, though basic, standard of living.”

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 9

Ryman Arts Evaluation Report, July 2006 a larger part of the incoming class, we examined the percentage of the incoming cohort that is made up of target school students. Figure 5B demonstrates an upward trend as well. This is further evidence of success of the targeted recruitment program.

When further examining the target group of students (n = 50), we found that target group members differed significantly from their remaining peers (n = 690) in several respects (see Table 2B). First, students from the target schools received significantly lower scores (p < .05) on the application drawings than their peers. Second, students accepted from the target schools into the Ryman Arts program were more likely to be males than females. This is significantly different from the remaining population (p < .01). And lastly, students from the target schools were significantly less likely to continue with the Ryman series after completing the first semester than their peers (p < .01). This could indicate that targeted students are not receiving enough support in the program, are unwilling, or unable to continue. This is worthy of further investigation.

Number of Students Recruited from Target Schools by Year

18 17

12 11

8 7 7 Number of Students 6

44 4

0 2000 2001 2002 2003 2004 2005

Figure 5A: This graph shows a statistically significant correlation (p < .05) between the number of students admitted from target schools and cohort year.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 10

Ryman Arts Evaluation Report, July 2006

Percentage of I ncoming Class from Target Schools

16.0%

14.0% 13.6%

12.0%

10.0%

8.1% 8.3% 8.0%

6.4% 6.0% Percentage of Students

4.0% 3.7% 3.5%

2.0%

0.0% 2000 2001 2002 2003 2004 2005

Figure 5B: This graph shows a significant positive correlation (p < .05) for the percentage of class coming from target schools and cohort year.

Total Percent Percent Percent Average Percent by Race/ Average Average Number of Female Attending Living in Number of Ethnicity Incoming Application Target Public Los Ryman Grade Level Drawing School Schools Angeles Semesters Score 8 Students County Mean Scaled Score (± sd.) Mean (± sd.) Mean (± sd.)

All n = 740 n = 474 n = 612 n = 669 n = 739 Unknown 50.8% n = 735 n = 270 n = 50 Ryman Asian 17.4% Students 100% 64.3% 82.7% 90.42% 2.07 (±1.05) Latino 16.8% 11.18 (±0.9) 19.7 (±2.23) 6.8% Caucasian 8.9% African-American 2.2% Mixed 2.2% Eastern European 1.4% Other 0.4%

Table 2A: This is a table of the general population of all Ryman Students, highlighting the total number of students in the study, the percent female, the percent attending public schools, the percent living in Los Angeles, the average number of Ryman semesters, the percent of students by race or ethnicity, the average incoming grade level, the average application drawing score, and the total number of target school students. This table is included here for comparison with Tables 2B-2C.

8 Ryman staff assessed these scores before the start of the evaluation based on an internally established rubric, separate from the Davis Assessment.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 11

Ryman Arts Evaluation Report, July 2006

Total Percent Percent Percent Average Percent by Race/ Average Average Number of Female Attending Living in Number of Ethnicity Incoming Application Target Public Los Ryman Grade Level Drawing School Schools Angeles Semesters Score Students County Mean Scaled Score (± sd.) Mean (± sd.) Mean (± sd.)

Target n = 50 n = 20 n = 50 n = 50 n = 30 Unknown 38% n = 50 n = 30 -- School Latino 46% Students 40% 100% 100% 1.7 (± .9) African-American 6% 11.22 (± .9) 18.75 (± 2.4) Asian 6% ** ** Mixed 4% *

Remaining n = 690 n = 454 n = 562 n = 619 n = 689 -- n = 685 n = 240 -- Ryman Students 65.8% 81.4% 89.7% 2.3 (± 1.0) 11.18 (± .9) 19.7 (± 2.2)

Table 2B: This is a comparative table showing the differences between Target School Students and the general population of all Ryman Students. Significant differences were tested using Students’ T-Tests or proportion tests and are marked with an “*” (p < .05) or “**” (p < .01) where significant differences occurred.

Who Persists in the Program? The students who successfully finished one semester but chose not to continue (or were not invited to come back) are referred to as “single semester students.” Significant differences were found between single semester students (n = 289) and the remaining population of Ryman students who took more than one semester (n = 451) (see Table 2C). First, we found that the entering grade level of the single semester group differed significantly from the continuing students (p < .01). Upon further analyses, it was found that grade level was positively (but non-significantly) correlated with the number of semesters that students complete. This indicates that earlier grade levels are less likely to take more than one class than their older peers entering in the same cohort. This correlation is weak, however. Second, we found that the single semester group differed significantly in terms of the gender composition than the continuing student group (p < .05). This indicates that males are more likely than females to stop participating in the Ryman series after taking only one semester of classes. It may be worthwhile meeting with a small focus group to determine some of the reasons for this trend. And finally, as aforementioned, students from target schools are significantly more likely to leave the Ryman series after only taking a single semester (p < .05).

Effect of Application Drawing Scores on Program Persistence Do the students that score lower on their application drawing continue on in the program? Do they persist at the same rates as the high scoring students? These are some of the questions posed by the Ryman staff at the start of the evaluation. From the results of the significance testing, the low scoring group (application drawing scores < 17.5 points, n = 48) did not differ significantly (p > .05) from the high scoring group (application drawing score > 20.5 point, n = 130) in any systematic way. The low-scoring group was equally likely to continue on in the program as the high scoring students. The only differences that were found were that target school students and males were more likely to receive lower scores on their application drawings than their peers in the program (p < .05).

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 12

Ryman Arts Evaluation Report, July 2006

Total Percent Percent Percent Average Percent by Race/ Average Average Number of Female Attending Living in Number of Ethnicity Incoming Application Target Public Los Ryman Grade Level Drawing School Schools Angeles Semesters Score Students County Mean Scaled Score (± sd.) Mean (± sd.) Mean (± sd.)

Single n = 289 n = 158 n = 247 n = 261 n = 289 Unknown 48.8% n = 285 n =135 n = 28 Semester Asian 18.7% Students 54.7% 85.5% 90.3% 1.0 (± 0.0) Latino 18.7% 11.01 (± .97) 19.7 (± 2.2) 9.7% Caucasian 8.0% * ** African-American 2.8% ** * Eastern European 1.4% Mixed 1.7%

Remaining n = 451 n = 316 n = 365 n = 408 n = 450 -- n = 450 n = 135 n = 22 Ryman Students 70.1% 80.9% 90.4% 2.8 (± .8) 11.29 (± .8) 19.8 (± 2.2) 3.0%

Table 2C: This is a comparative table showing the differences between Single Semester students and the general population of all remaining Ryman students. Significant differences were tested using Students’ T-Tests or proportion tests and are marked with an “*” (p < .05) or “**” (p < .01) where significant differences occurred.

Assessment of Student Artwork Portfolios: A Note on Methods At the start of the evaluation, we decided that it was necessary to examine the student work for evidence of growth in key areas of drawing. Together with the Ryman Arts program, we surveyed a number of potential instruments for use in the portfolio evaluation. Initially, we found a number of existing rubrics aimed at scoring early elementary-school aged work9 but there are surprisingly few rubrics that examine more advanced studio work especially in a systematic fashion. In the final stages of our search, we focused on the College Board Advanced Placement (AP) guidelines; the Ohio Department of Education’s assessment of children who are gifted in visual arts; the Davis Drawing Assessment Program (Cornia and Egan, 2005); and other internally established assessment rubrics. However, most of these rubrics lacked the level of detail necessary to reliably code different aspects of the work found in the large number of portfolios due to vague definitions for the different scales. The Davis Method was finally chosen as it provided the vocabulary and visuals to describe the key elements in basic drawing.

It is worthwhile taking a moment to discuss the strengths and limitations of the Davis Assessment Program. As the Davis manual states:

The Davis Drawing Assessment Program is designed for both student and teacher to assess performance related to basic drawing. All creative products require knowledge of the real world and its natural attributes. For this reason, and by intention, the charts are not designed to assess any degree of imagination, artistic expression, artiness, or creativity. Instead, the primary focus of the charts is to develop perception, knowledge, and realistic drawing skill (Cornia and Egan, 2005, p.4).

9 See for example the work of Howard Gardner (1980) and Arthur Tobias (2001).

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 13

Ryman Arts Evaluation Report, July 2006

From this excerpt we see that the focus of the Davis assessment is on high fidelity drawing skills or the ability to reproduce objects seen in the real world with a quality and accuracy with a sense of optical realism. The drawing assessment rubrics cover five key concepts seen as essential to basic drawing: (1) Visual Field; (2) Shape; (3) Value; (4) Dimension; and (5) Light. It is important to note that these five domains only address some core, and oftentimes, common problems encountered in basic drawing. However, there are arguably many attributes that are not included in this list, including the ones mentioned in the previous excerpt such as creativity, imagination, artistic expression, among others such as composition and sense of time and effort invested. Despite these limitations, it was agreed upon that the Davis Drawing Assessment Program was in line with one of the core goals of the Ryman Arts program: Dedication to teaching classical drawing skills to talented, motivated young people.

For the purposes of this evaluation, we chose to focus on three of these five domains (i.e., Shape, Value, and Dimension). The following is a brief description of the ways in which Cornia and Egan are defining each of these dimensions of drawing. For further explanation, please refer to the Davis Drawing Assessment Teacher’s Manual.

Shape. Shape is defined by Cornia and Egan as “an area completely surrounded by line. Shape is also the visible outline of a 3-D object called an outline or contour” (p. 22, 2005). Using this definition, the Davis Assessment identifies four subscales, including: (1) Basic Shapes; (2) Proportion; (3) Continuity and Regularity; and (4) Structure. Although it is outside the scope of this report to define and describe the elements important to these subscales, it is important to note that there are four scales that indicate improvement in one’s ability to draw shapes.

Value. Value is defined as “an element of art that refers to the relative lightness or darkness of an object or area” (p. 48). The subscales of value include: (1) Local Value; (2) Variations; (3) Irradiation; and (4) Aerial Perspective. The key aims of value are to have a range on tones from light to dark and to faithfully represent the perception of an object’s local color. This is not to be confused with the category, “Light,” which is the specific effect that light has on an object. This is seen as a separate area in the Davis Assessment.

Dimension. Dimension is defined as “a measure of height, width, and depth” and can also be seen as the portrayal of three-dimensions in a two-dimensional medium, such as drawing. Dimension also has four sub-scales, including: (1) Linear Perspective; (2) Three- Dimensionality; (3) Contour; and (4) Size and Placement. The key aims of this area of the assessment are to judge the artists’ ability to render three-dimensional space.

Training and Planning. At the start of the student portfolio evaluation, Ryman staff identified portfolios for inclusion in the evaluation. Portfolios that were selected had to meet a set of basic criteria for inclusion: (1) Application drawings must be present in the portfolios. These are the drawing that students completed and submitted prior to acceptance into the program, and were used as one criteria for admission; (2) Students must have enrolled in at least beginning and intermediate drawing; (3) There must be a sufficient number of drawings included in the portfolio to demonstrate the breadth and depth of the student’s ability from each of the beginning and intermediate classes (e.g., two or more from each class). Therefore, this selection of portfolios best represents those students that recently enrolled and actively produced work in both a beginning and intermediate Ryman course.

Once the individual portfolios were selected (n = 45), we assembled a team of evaluators, Ryman teaching artists, and Ryman staff to go through each of the portfolios and select three

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 14

Ryman Arts Evaluation Report, July 2006 drawings: (1) one application drawing; (2) one beginning drawing; and (3) one intermediate drawing. The drawings were chosen generally based on merit and some marked similarity or story line that was seen as developing in the work (such as the inclusion of all self-portraits, if applicable). Generally watercolor paintings, sketches, and unfinished work were excluded from inclusion in the assessment, although there were some exceptions to this. Notes were taken on these early discussions of the student work to try to triangulate the findings from the Davis Assessment with the general observations of the teaching artists since they were able to look over the entirety of each student’s work.

Coding the portfolios. A group of three research assistants worked together to code the entire set of portfolios. Together, the group of coders first discussed the Davis Drawing Assessment Method, verbal and visual descriptions, and illustrated charts. Once there was a solid understanding of each of the scales, the coders moved to coding each of the individual drawings based on the established scales and subscales using the Davis drawing assessment scoring sheets (see Appendix E). Periodically, the coders would meet to establish inter-rater reliability and discuss drawings that were difficult or problematic to code. Coders used the scoring sheet to record the student’s baseline, beginning, and intermediate skill level on each of the drawings in order to track improvement. Scores ranged from 1-20 (or Levels 1-4) for levels of proficiency for each of the four subscales of Shape, Value, and Dimension. Scores 1-5 (or Level 1) represent the “ultra-weak” while scores between 16-20 (or Level 4) represent the “ultra-strong” work. Raw scores were then entered into an SPSS database for further analysis.

It should also be noted that the Davis Assessment is only intended to examine graphite pencil drawings and is better suited for certain types of subject matter (i.e., landscape drawings). Because a variety of media were used in the Ryman portfolio drawings, (including colored pencil, charcoal, chalk, oil pastel, and watercolor) we had to freely adapt the Davis Assessment to these media when possible and/or refrain from scoring some of the drawings for certain dimensions of the assessment. We do not feel that this caused any inherent problems to the assessment as the Davis assessment materials encourage only scoring drawings for the relevant areas.

Selecting Case Studies. Acknowledging some of the limitations of the Davis Assessment technique, we thought that it would be worthwhile to include a collection of three case studies to illustrate the student work along with a brief description of what the teaching artists and Ryman staff had to say (oftentimes from insider or personal perspective) alongside of the findings from the Davis Assessment (an admittedly outsider perspective on the student work). Three case studies were chosen from among the 45 portfolios. Two were chosen based on being representative or typical of the quality of the student work and growth in artistic ability as a result of their participation in the Ryman Arts program. The third case was chosen from among the participants that came from the aforementioned targeted high schools.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 15

Ryman Arts Evaluation Report, July 2006

Key Findings from the Portfolios Students that were chosen to participate in the portfolio evaluation did not differ in any significant ways from the entire Ryman population except for gender. Within the portfolio evaluation, significantly more females than males were represented. This is an expected finding since a larger number of males discontinue the series during or shortly after the first semester and only students that have had consistent attendance in both the first two semesters were chosen to participate in the portfolio evaluation. This could also be an indicator that persisting females have better attendance than male peers.

Total Percent Percent Percent Average Percent by Race/ Average Average Number of Number (n) Female Attending Living in Number of Ethnicity Incoming Application Target Public Los Ryman Grade Drawing School Schools Angeles Semesters Level Score 10 Students County

Portfolio 45 n = 33 n = 45 n = 41 n = 45 Unknown 22.2% n = 45 n = 34 n = 2 Evaluation Latino 28.9% Students 73.3% 82.2% 91.1% LA 2.4 (±0.6) Asian 24.4% 11.2 (±0.8) 19.6 (±2.0) 4.4% (Remaining Caucasian 13.3% * from Mixed 6.7% Orange African-American 2.2% County) Eastern European 2.2%

Remaining 695 n = 441 n = 567 n = 628 n = 694 Unknown 48.3% n = 690 n = 235 n = 48 Ryman Asian 17.8% Students 63.5% 81.6% 90.4% 2.05 (±1.1) Latino 17.1% 11.2 (±0.9) 19.8 (±2.3) 6.9% Caucasian 10.5% African-American 2.3% Mixed 2.2% Eastern European 1.3% Other 0.4%

Table 3: This is a comparative table showing the differences between Portfolio Evaluation Students and the general population of all Ryman students. Significant differences were tested using Students’ T-Tests or Chi-Square Tests and are marked with an “*” (p < .05) or “**” (p < .01) where significant differences occurred.

On average, there were steady gains made in each of the three scales of the portfolio evaluation (see Figure 6). The average student moves from a low/mid level III entry score to a high level III/low level IV score by the end of the intermediate drawing course in all three areas. Statistical tests were run on each of these three domains and found that students demonstrated extremely significant growth for all areas, including shape (p < .01), value (p < .01), and dimension (p < .01) when the mean scaled scores for the intermediate drawings were compared to the application drawings (Drawings 1 and 3 being compared; see Table 3). These results demonstrate that Ryman students made significant gains in each of these areas over the course of the first two semesters in the program. We further analyzed the drawings comparing the application to the beginning drawing (Drawings 1 and 2) and the beginning to the intermediate drawings (Drawings 2 and 3). Results of the significance testing indicate that students make significant gains in all three areas (p < .01 for Shape and Dimension; p < .05 for Value) by the time they make their beginning drawing. At the time of the intermediate drawing course, students make additional significant gains in Value (p < .01) and marginally significant gains in Dimension. Students demonstrate positive although not significant gains in Shape during this same time. This is probably due to the strong initial understanding of

10 Ryman staff assessed these scores before the start of the evaluation based on an internally established rubric, separate from the Davis Assessment.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 16

Ryman Arts Evaluation Report, July 2006

Shape in the application drawings. These results indicate that there is not only educational merit to the first beginning course but there are also additional gains made in the intermediate drawing course. Additionally, one can see that students’ enter the Ryman Arts program with a strong understanding of shape but have a more difficult time with Value. Thus, the greatest gains are made in this area.

Application & Beginning Beginning & Intermediate Application & Intermediate Drawing (1 & 2) Drawing Drawing (1 & 3) (2 & 3) Mean Mean Mean n Scaled Sig. n Scaled Sig. n Scaled Sig. Scale Score Score Score (±sd.) (±sd.) (±sd.)

Shape Pre 45 12.8(±3.2) ** 45 14.3(±3.1) 45 12.8(±2.4) ** Post 45 14.3(±3.1) 45 15.3(±3.8) 45 15.3(±3.8)

Value Pre 44 9.3(±3.8) * 41 10.9(±4.0 ** 42 9.1(±3.8) ** Post 44 10.9(±3.8) 41 ) 42 13.5(±3.0) 13.5(±3.0)

Dimension Pre 44 11.7(±3.8) ** 44 13.6(±3.3) (m) 45 11.7(±3.8) ** Post 44 13.6(±3.3) 44 14.7(±2.7) 45 14.7(±2.7)

Table 3: This table shows the results of the paired samples T-Tests for significant differences by scale between (1) the application and beginning Ryman drawings; (2) the beginning and intermediated Ryman drawings; and (3) the application and intermediate Ryman drawings. Significant differences were tested using Paired Samples T-Tests and are marked with an “*” (p < .05) or “**” (p < .01), where significant differences occurred.

Average Scores for Shape, Value, and Dimension by Semester

20.00 LEVEL IV

15.00 15.00 14.00 13.60 13.00 13.30 m 12.40 * * m 10.00 10.50 * Application Drawing 10.00 8.97 Beginning Drawing * Intermediate Drawing

5.00 Davis Assessment Score LEVEL I 0.00 Shape Value Dimension

Figure 6: This graph reflects the average score for the three major Davis Assessment areas (i.e., Shape, Value, and Dimension). There is steady growth in all three areas, suggesting that students increased their understanding of shape, value, and dimension throughout the first two semesters in the Ryman Arts program. Note that students enter Ryman Arts with a very strong sense of shape at an average of 13 points (Level 3). *p < .05; m = marginally significant gains

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 17

Ryman Arts Evaluation Report, July 2006

In Appendix D of this report, we offer the Davis definitions of the subscales that make up the three primary scales, as well as provide graphs and tables of the average student scores for each subscale. The results indicate a positive upward trend for all but one of the sub-scales (i.e., the “Size and Location” subscale of Dimension). This is a strong indication that students at some point during the beginning and intermediate classes learn the vast majority of these concepts. Statistical tests indicate that ten of the subscales demonstrate significant gains (p < .05) between the application and beginning drawings (mostly the subscales of Shape and Dimension) and six of the subscales demonstrated significant gains (p < .05) between the beginning and intermediate drawings (mostly from the Value subscales). See the Table 4 in the Appendix for more detailed information.

It should be said at this point that these are average trends and not indicative of the progress made by every student at every time point. There exist several students that, for some reason or another, do not fit within this model. For example, some students enter the program with very strong (Level IV) core drawing skills. In these cases, there is a ceiling effect and you would not expect to see additional growth using the Davis Assessment. There are still other cases that demonstrate little or no growth in core drawing skills. Although there are many explanations for cases such as these, this is sometimes the case when students enter with an exceptionally strong application drawing (such as a self-portrait) but have difficulty branching out into other subject areas or media. At other times, some students have taken home their best work from their portfolios or have not gained as much in these areas. Regardless, the significantly positive overall trends are a good sign that the majority of students have increased their drawing skills over the course of their time in the Ryman Arts program.

A Closer Look In order to help illuminate central findings from the portfolio assessment, we have chosen to highlight three students. In the following narratives, the reader will find photographs of the three images that were chosen from each of the students’ portfolios for the purposes of the evaluation along with a brief description of the student. Then there are two perspectives on the students’ work. The first comes from the external evaluators who used the Davis Assessment. Here, they highlight some of the key findings and how the individual work falls within the spectrum (Levels I-IV) of the Davis Assessment. The second discussion comes from the conversations that were had among teaching artists and Ryman staff. These perspectives present somewhat of an insider’s view on the students’ work. For the most part, the teaching artists had these students in their classes at some point and knew the students personally. In addition, the teaching artists and staff were more familiar with the individual assignments and had a greater understanding of how these drawings fell within the scope of the work produced at Ryman. The central advantage to these discussions is that they were not bound by the Davis Assessment categories. They were free to talk about creativity, artistic potential, and visual expression as they saw fit. The teaching artists and Ryman staff also had the advantage of viewing the students’ entire portfolio when commenting on these students’ areas of strengths and weakness.

Alicia11– An Average Case Alicia is an 11th grade Latina, attending a public school in Los Angeles County. Her application drawing earned a score of 18 points during the application, which was slightly below the average. Since

11 All student names used in this report are fictional.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 18

Ryman Arts Evaluation Report, July 2006 the start, Alicia has completed three semesters in the Ryman Arts program. Here are her drawings that were submitted to be part of the portfolio evaluation:

Application Drawing Beginning Drawing Intermediate Drawing

Reflections based on the Davis Assessment Alicia’s Application hand drawing scored highly with regard to contour and shape. Lines showing the contour of her hand’s surface occurred at all levels of specificity, and a variety of line styles are used to define all structural parts of the hand’s surface. Alicia also appears to be experimenting with foreshortening as a means to portray volume without skewing the proportions of the object. As for the area of shape, the outline of her hand is continuous and graceful, with only slight errors in the upturned fingers. The strengths of her Application drawing were elaborated upon in her Intermediate charcoal portrait, with high marks scored for the accuracy of internal ratios and specificity of facial features, as well as rendering of structural lines on three-dimensional surfaces. However, the Intermediate drawing demonstrates a more realistic representation of local values. The wider range of shades are used in the drawing, giving a more realistic illusion of depth, and most dark lines surrounding objects have been dissolved by rendering the local value. The Application drawing scored on the border between Level II and Level III proficiency, and the Intermediate drawing scored on the border between Level III and Level IV proficiency.

Alicia’s Beginning pencil portrait, however, was not part of the trend towards improvement, which was the case with a handful of other portfolios in the evaluation. Unlike the other drawings, curved lines in these objects showed noticeable corners and edges along its path. The careful attention to smooth lines to render various surfaces has been replaced with an overall, stylized hatching pattern, and the appearance of depth in space is minimal due to the limited range of local values and value variations. The Beginning drawing scored at the borderline between Level I and Level II proficiency.

Reflections from Teaching Artists and Staff The application drawing is well drawn and the way that the thumb comes in with a three-quarters, foreshortened view is well handled. The whole feeling of it does not feel goofy or distorted and there is a sense of fleshiness in the palm of the hand. You can also tell that this was drawn from observation. Weaknesses of the application drawing include a lack of shading and a reliance on outlining and composition or placement on the paper is awkward or not considered. Her work is also “less stylized

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 19

Ryman Arts Evaluation Report, July 2006 than typical.” She seems to have a “fearless” approach to her drawing as she embraces the “muscularity” of her subjects and develops this over time. Her sense of composition demonstrates noticeable improvement. One of the artists recounts, “Oh, I remember, she had an epiphany in my class [i.e., beginning drawing]. When we were in the museum, she had a breakthrough drawing. It was unbelievable.” [This was said in reference to the drawing selected for the 2005-2006 gallery catalog of an elk with antlers well cropped within the composition.] In her other work that was not included in this evaluation, she has taken different perspectives and has consciously worked with the composition even in her work done outside of class. Overall, there is a clear development of depicting weight and physical form.

Tanya – A Case of Substantial Growth Tanya is an African-American female that began the Ryman Arts program as a 12th grader in a Los Angeles public school. Her application drawing, still life with pencil, earned her a 17.5 at the point of application. This is again slightly below the average. At the point of the evaluation, Tanya had taken two semesters of classes through the Ryman Arts. Here are her drawings that were submitted to be part of the portfolio evaluation:

Application Drawing Beginning Drawing Intermediate Drawing

Reflections based on the Davis Assessment Tanya showed very strong improvements in all areas of her drawings. A substantial difference in her facility with local values can be observed between her Application still life— which is composed of a limited range of two or three lighter values—and her Beginning self-portrait—which has darker, distinct values to distinguish her sweater, shirt, and skin tone from each other. While the value rendering of her skin is off-balanced in her Intermediate watercolor self-portrait, Tanya shows strength in her use of local value contrasts and is unafraid to include a variety of values within an object’s shape. Also a dramatic improvement in this area is her use of local value contrast in her later drawings to produce an illusion of depth, and not using dark outlines as the primary element used to determine an object’s identity. Tanya also showed immediate improvement with regard to drawing shapes with primary, secondary, and tertiary degrees of detail: the plant leaves are oversimplified, the bamboo legs do not appear to radiate from the same point, and the axial arrangement of the wall and the vase are misaligned horizontally in the Application drawing, whereas Tanya pays close attention to the folds of her sleeve in her Beginning portrait and to her facial details in her Intermediate portrait. As Tanya’s Intermediate drawing appeared to be unfinished, probably due to the laboriousness of the watercolor medium, we are unable to track her improvement in her use of size and location or linear perspective.

Reflections from Teaching Artists and Staff

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 20

Ryman Arts Evaluation Report, July 2006

In the words of the teaching artists, Tanya was “a student who did the work” and invested time in every assignment. Tanya “gets the assignment done and seems to be gaining from each piece.” This was evident to the teaching artists despite neither of them having had Tanya as a student. At the start, Tanya was good at drawing detail, particularly with the wrapping around the sticks of bamboo. The composition of the Application drawing is oddly cropped at the bottom of the vase with no shadow. Overall, her work demonstrated improved growth in her ability to create a composition and plan the drawing accordingly. One teaching artist said, “her jump in the compositional sense and how to plan from one drawing to another is almost unbelievable.” There is a certain precision to her drawing that develops over time. We can see her go from a centralized, privileged object to a more sophisticated composition as we move through her work seen only in her whole portfolio. The only exception to this is the composition of the final intermediate watercolor painting. The beginning drawing has a much stronger sense of composition but the new media introduced through watercolor seems to have “set her back.” Yet there is a stronger sense of value that develops in the work as she gradually starts to work larger areas of the composition at one time. This seems to be a common trend for most students when working in a new medium. There also seems to be a certain sense of trust in the viewer and an overall growth in confidence.

Diego – A Target School Student Diego is an English-Language Learner and a senior male at one of the targeted high schools in Los Angeles. His application drawing earned him an overall score of 22, which was well above average. At this point, Diego has taken three semesters of Ryman classes all with perfect attendance. Here are his drawings that were submitted to be part of the portfolio evaluation:

Application Drawing Beginning Drawing Intermediate Drawing

Reflections based on the Davis Assessment Diego’s Application drawing demonstrated a strong ability to render shape. Diego appeared to have some understanding of convexity in the interior basic shapes, and was able to draw simple shapes, for the most part, accurately and with some geometric precision. The weakest aspect of Diego’s Application drawing—and where he eventually showed the greatest improvement in his Intermediate drawing—was his grasp of dimension. Contour lines drawn on foreshortened objects in his Application still life (such as the magnifying glass) are not sufficiently curved, and a stylized shading treatment on the skull did not portray a convincing sense of depth. By contrast, contour lines on the ceiling and pillars of Diego’s far more ambitious Intermediate pastel landscape described the surface of the objects, large and small, and contributed to an illusion of three-dimensionality. Improvements in Diego’s portfolio were not as pronounced in the use of value. His use of variations and range of values in his Beginning drawing is particularly flat.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 21

Ryman Arts Evaluation Report, July 2006

Worth mentioning is Diego’s experimentation with scale. There is a steady improvement across all drawings with relation to how space is filled, and Diego appears more adventurous with each drawing when it comes to extending his objects to the periphery of his pages. While improvements between Beginning and Intermediate drawings were less pronounced, Diego’s portfolio shows a development from Level II to Level III drawing proficiency.

Reflections from Teaching Artists and Staff Diego didn’t have either of the teaching artists participating in the evaluation as instructors but they quickly remarked, “He’s going to be a real successful artist – no doubt about it.” At the moment, Diego doesn’t have any plans to attend college and this causes concern within the group12. Diego’s application drawing has nicely rendered cast shadows with harder and softer edges at various points but seems to struggle with some of the proportions and foreshortening of the skull. In the following drawings, everything seems to have been improved upon despite it being a more ambitious composition. Throughout the work there is a vivacity of line but the work tends to be flattened value- wise. They note that “he’s observing while he’s drawing” carefully noting the shadows and carefully rendering the objects in the drawing. In the Beginning drawing, one artist notes, “It’s already such an impressive drawing going on here with simple rhythms. And here [pointing to the cup], I’m buying the ellipse, which is pretty hard to pull off.” About the intermediate drawing, one teacher remarks, “He’s a very honest drawer.” The repeating elements of the pillars, for example, are well dealt with and well observed. The details haven’t been overly simplified and repetitive. Overall, there is a broadening of subject matter boundaries, a more ambitious use of composition, new techniques are played with, and there is a general movement from a relatively simple still life to an ambitious view of the rotunda.

Conclusions We identified a number of conclusions from the data and the discussion presented in this study, which we summarize as follows:

a) Since 2000, Ryman Arts has expanded the program by more than 50% to service more students in each successive year. The data also suggest that the program is now, in 2006, servicing a more geographically diverse group of students than ever before.

b) The Ryman Arts program has consistently serviced a high percentage of public school students (80-85%), predominantly from Los Angeles County.

c) On average since 2000, Ryman students tend to remain in the program for 2-3 semesters. Most recently in the 2004-2005 cohort, students took an average of 3 semesters.

d) Ryman Arts has been gradually servicing more underclassmen (grades 9-10) and more males (moving towards a male to female 50/50 ratio) in the program since 2000.

e) Males, more so than females, are at risk of being a single semester student.

f) The targeted high school recruitment program has been successful in recruiting youth from under-privileged schools into the program. However, targeted students seem to be at

12 Ryman Arts has recently contacted this student to check in with him and ascertain whether he has any new plans for his future.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 22

Ryman Arts Evaluation Report, July 2006

risk of dropping out of the program on average somewhere after one or two semesters. This is significantly earlier than their peers.

g) Scores on the application drawings are not predictive of persistence in the program.

h) On average, the Ryman Arts program seems to be contributing to substantial and significant development of students’ drawing skills, demonstrated in their understandings of shape, value, and dimension, which were the specific skill areas analyzed. This growth is seen both during the first semester of beginning drawing and again during the second semester of intermediate drawing.

i) The Davis Drawing Assessment has proven to be a very useful tool for evaluation within the Ryman Arts study. We would recommend the continuing use of this assessment tool for any future external or internal inquiry. This tool also looks promising for use within other organizations faced with the similar problem of looking at more advanced work to determine if there are any gains in core drawing skills.

Based on this exploratory longitudinal study and evaluation of student portfolio work, our observations and findings should prove encouraging. The twelve-week program brings high quality teaching to youth, encourages many to move toward active engagement, and provides them with new skills.

Future assessments within Ryman Arts? We were invited by Ryman Arts staff to give some thought to possibilities for future inquiry within the program. The program may wish to ask additional questions as it moves forward. The following possibilities emerged from our discussions, and we are sure others will have their own curiosities.

1) One of the limitations of this study is the lack of a comparison group. Although we know that the students gained certain skills over the course of their participation in the Ryman Arts program, it is impossible to tell whether these drawing skills were gained as a result of the Ryman Arts program, as a result of another outside or school program, through general developmental trends, or through a combination of any of these factors. In order to attribute more of the gains to Ryman, we would recommend following a small group of comparison students from the home institutions of the students enrolled in the Ryman Arts program. Ideally, students in the comparison group would have applied and been on the borderline for entry into the Ryman Arts program.

2) We recommend the creation of focus groups to study how to better target the recruitment and retention of under-represented student groups in the program. We also recommend the continued anonymous collection of data of the racial/ethnic make up from each incoming cohort to study how these trends change over time.

3) Follow-up focus groups with drop out students as well as with groups at-risk of dropping out of the program are needed to determine how better to support these students.

4) We also suggest the further studying of the entire applicant pool in order to better understand those students and/or teachers with unsuccessful applications. A longitudinal

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 23

Ryman Arts Evaluation Report, July 2006

approach to identifying unsuccessful applicants could suggest additional schools and/or programs to target with application assistance.

References Cornia, I. E., & Egan, A. S. (2005). Drawing Assessment Program. Worcester, MA: Davis Publications, Inc. Gardner, H. (1980). Artful Scribbles. New York: Basic Books. Tobias, A. (2001). A Test for the Visual Arts: The Harvard-Westlake School. Available at: http://harvart.home.mindspring.com/pdfs/ArtTest.pdf. Last accessed on July 10, 2006.

Kylie A. Peppler [email protected]

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 24

Ryman Arts Evaluation Report, July 2006

Appendix A:

Not included in public copies.

Appendix B: Summary table of all known schools sending students to Ryman Arts since 2000

Name of High School N Downey High School 2 Missing Data 1 Downtown Business Magnet 3 AGBU Manoogian 1 E.I.E Academy 1 Alex Pilibos Armenian School 1 El Camino Real High School 4 Alhambra 1 El Rancho High School 2 Aliso Niguel 2 Esperanza High School 3 Animo Inglewood 1 Fairfax High School 7 Arcadia High School 16 Fairfax Visual Arts Magnet 2 Arroyo 1 Fairmont High School 1 Azusa High School 2 Family Life Center Christian 1 Baldwin Park High School 5 Flintridge Preparatory School 3 Banning High School 10 Francis Polytechnic 1 Bassett High School 5 Franklin 4 Bell Gardens High School 1 Fremont 1 Bell High School 4 Gahr 2 Belmont High School 24 Garden Grove 5 Beverly Hills High School 2 Garey High School 7 Birmingham High School 3 Gilbert East High School 1 Bishop Amat High School 3 Gladstone High 1 Bishop Montgomery High School 3 Glen Wilson High School 2 Brentwood School 1 Glendale High School 7 Cabrillo 1 Glendora High School 3 Calabasas High School 3 Granada Hills Charter 2 California High School 2 Grant High School 2 Canyon 1 Hamilton High School 11 Carson High School 1 HART 1 Cerritos High School 1 Harvard-Westlake 2 Chaminade Prep. 1 Hawthorne High School 22 Charter Oak High School 2 Hearthstone Academy 1 Christian Liberty 1 Herbert Hoover High School 11 City of Angels High School 1 High Desert Alternative Education Center 1 Claremont High School 3 Hollywood High School 3 Clark Magnet High School 8 Huntington Park High School 5 Cleveland Magnet 2 Immaculate Heart High School 7 Crenshaw High School 4 Irvine High School 1 Crescenta Valley High School 36 Jefferson High School 6 Culver City High School 36 John Burroughs High School 10 Daniel Murphy Catholic High School 1 John F. Kennedy High School 7 Diamond Bar High School 2 John F. Kennedy High School -OC 1 Dominguez High School 1 John Marshall High School 18 Dorsey 1 La Canada High School 10

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 25

Ryman Arts Evaluation Report, July 2006

LA Center for Enriched Studies 1 Quartz Hill High School 2 LA County High School for the Arts 10 Ramona Convent Secondary 5 La Salle High School 1 Redondo Union High School 6 La Serna High School 1 Reseda Magnet 2 Laurel 2 Rio Mesa High School 1 Leuzinger High School 2 Robert A. Millikan 1 Lincoln 4 Rowland 1 Locke High School 6 Sacred Heart High School 1 Long Beach Polytechnic High School 7 San Dimas 2 Los Alamitos High School 1 San Gabriel 3 Los Angeles Center for Enriched Studies 3 San Marino High School 7 Los Angeles Lutheran High School 4 San Pedro 1 Louisville High School 6 Santa Ana 1 Loyola High School 1 Santa Fe 1 Lynwood 3 Santa Monica High School 9 Manual Arts High School 13 Saugus High School 3 Maranatha High School 5 School of Arts and Enterprise 1 Mark Keppel High School 2 Schurr High School 1 Marlborough High School 1 Simi Valley High School 3 Marshall Fundamental 1 Skyward Christian 1 Martin Luther King 1 Sonora High 1 Mater Dei High School 17 South El Monte High School 10 Maxine Waters 1 South Hills High School 3 Mayfield Senior School 5 South Pasadena High School 6 Milken Community High School 7 St. Anthony 1 Mira Costa High School 4 St. Francis High School 1 Monroe Magnet 2 St. Joseph High School 3 Montclair High School 1 St. Matthias High School 8 Montebello High School 3 St. Monica Catholic High School 3 Morningside 1 Sunny Hills High School 3 Mountain View High School 4 Sylmar 1 Narbonne 1 Taft High School 8 New Covenant 1 Temple City High School 23 Newport Harbor High School 6 Thousand Oaks 2 North Hollywood High School 6 Troy High School 1 North Hollywood Zoo Magnet 1 University High School 3 Northwood 1 Upland High School 3 Notre Dame 15 US Grant High School 1 Ontario High School 1 Van Nuys High School 6 Orange County School for the Arts 1 Venice High School 7 Pacifica 1 Verbum Dei 1 Palisades Charter High School 9 Verdugo Hills High School 3 Palos Verdes Peninsula High School 7 Village Christian 1 Paraclete 1 Vineyard Christian 1 Pasadena High School 5 West High School 3 Polytechnic-Pasadena 2 West Torrance 2 Pomona 2 Westchester High School 2

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 26

Ryman Arts Evaluation Report, July 2006

Westmark High School 1 Westridge School for Girls 6 Whitney High School 3 Woodbridge High School 2 Woodrow Wilson High School 8 Xavier Academy 1 Yucaipa High School 1 Total 740

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 27

Ryman Arts Evaluation Report, July 2006

Appendix C: Table of Results of Paired Samples Significance Testing by Subscale

Application & Beginning Beginning & Intermediate Application & Intermediate Drawing (1 & 2) Drawing (2 & 3) Drawing (1 & 3)

Mean Mean Mean

n Scaled Sig. n Scaled Sig. n Scaled Sig. Scale Subscale Score Score Score (±sd.) (±sd.) (±sd.)

Basic Shape Pre 45 12.7(±3.4) ** 45 14.2(±3.3) * 45 12.7(±3.4) ** Post 45 14.2(±3.3) 45 15.4(±2.5) 45 15.4(±2.5)

Proportion Pre 45 13.3(±3.1) * 45 14.5(±3.1) 45 13.3(±3.1) **

Post 45 14.5(±3.1) 45 15.4(±2.6) 45 15.4(±2.6)

APE

SH Continuity & Regularity Pre 45 12.3(±3.3) ** 45 14.0(±3.4) 45 12.3(±3.3) ** Post 45 14.0(±3.4) 45 14.8(±2.5) 45 14.8(±2.5)

Structure ** Pre 45 13.0(±3.5) * 45 14.4(±3.3) 45 13.0(±3.5) Post 45 14.4(±3.3) 45 15.3(±2.6) 45 15.3(±2.6)

Local Value Pre 44 9.7(±3.8) * 41 11.7(±4.0) ** 42 9.6(±3.9) ** Post 44 11.6(±3.9) 41 14.0(±3.5) 42 13.9(±3.5)

Variance & Clarity Pre 44 10.5(±4.0) * 41 12.0(±4.0 ** 42 10.3(±4.0) **

Post 44 12.0(±3.9) 41 ) 42 14.6(±3.2) E

U 14.7(±3.2)

VAL Irradiation Pre 44 9.4(±4.0) * 41 11.0(±4.2) ** 42 9.4(±4.0) ** Post 44 11.0(±4.1) 41 14.0(±3.4) 42 13.9(±3.4)

Aerial Perspective Pre 34 6.7(±3.8) (m) 33 8.5(±3.9) ** 33 6.8(±3.8) ** Post 34 8.4(±3.6) 33 10.7(±3.7) 33 10.9(±3.6)

Table 4: This table shows the results of the paired samples T-Tests for significant differences by subscale between (1) the application and beginning Ryman drawings; (2) the beginning and intermediate Ryman drawings; and (3) the application and intermediate Ryman drawings. Significant differences were tested using Paired Samples T-Tests and are marked with an “*” (p < .05), “**” (p < .01), or “(m)” (p = .05) where significant differences occurred. Dashed marks, “–“, indicate where there were less than 20 drawings that were able to be scored for the subscale (i.e., for Linear Perspective and Size & Location), indicating that significance tests were unable to be done and results were inconclusive.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 28

Ryman Arts Evaluation Report, July 2006

Appendix C (continued…)

Linear Perspective Pre < 20 3.3(±.6) -- < 8.5(±.7) -- < 20 5.5(±3.5) -- Post < 20 10.7(±2.5) 20 9.5(±.7) < 20 7.5(±3.5) < 20

Three- Dimensionality Pre 44 11.5(±4.0) ** 43 13.2(±3.6) 43 11.5(±4.0) ** Post 44 13.3(±3.6) 43 14.4(±3.2) 43 14.4(±3.2)

NSION

DIME Continuity Pre 43 11.8(±4.3) ** 43 13.8(±3.5) * 43 11.8(±4.3) ** Post 43 13.8(±3.5) 43 15.1(±3.1) 43 15.1(±3.1)

Size & Location Pre Post < 20 13.5(±3.1) -- < 12.6(±4.5) -- < 20 13.2(±2.2) -- < 20 13.3(±1.6) 20 15.7(±1.8) < 20 15.4(±1.5) < 20

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 29

Ryman Arts Evaluation Report, July 2006

Appendix D: Definitions and Analysis of the Individual Subscales for Shape, Value, and Dimension

Shape

1. Basic Shapes The shapes of all natural and manufactured objects are composed of simple shapes—circles, squares, and triangles—their combinations and variations. Shapes range from simple to complex. Realistic drawing skill includes the ability to accurately draw the overall shape of an object as well as its individual parts.

2. Proportion All shapes have a natural size and proportion relationship. Realistic drawing skill includes the ability to reproduce accurately the proportion of an object and its size relationship to other objects.

3. Continuity and Regularity The single component shapes of all objects possess continuous, regular, and symmetrical properties. Realistic drawing skill includes the ability to reproduce the continuous, regular, and symmetrical properties of an object’s shape.

4. Structure All objects have structure. There is external and internal structure. External structure includes features such as the lines that separate an object’s parts, corners and edges, wrinkles and folds, and surface detail. Internal structure includes the skeleton and anatomy of an object. Realistic drawing skill includes the ability to portray structural features, whether they are internal or external, or whether organic or inorganic.

Four Subscales for Shape

LEVEL IV

Basic Shapes Proportion 10 Continuity & Regularity Structure Davis Assessment Score 5

LEVEL I

0 Application Drawing Beginning Drawing Intermediate Drawing

Figure 7: This graph indicates relatively identical, moderate upward trends for all four of the subscales for Shape. The average student enters the program with a mid-Level III understanding and exits with a high level III to a low level IV understanding of shape and its individual components.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 30

Ryman Arts Evaluation Report, July 2006

Value

1. Local Value Every object has a specific value called local value. Local value is the degree of lightness or darkness of an object’s pigmentation, independent of any effect of light falling on it. Local values range from black to white.

2. Variations Variations of value can exist on an object’s surface. Local value contrasts, more than interior variations, provide visual separation between objects. Realistic drawing skill includes the ability to render variations in value on an object’s surface while maintaining contrasts in local value between multiple objects.

3. Irradiation This phenomenon refers to the brain’s perception of accents or increases in contrast/clarity at the point where values meet or overlap.

4. Aerial Perspective Local value, detail, and dimension are affected by distance and atmosphere. This effect is called aerial perspective. Realistic drawing skill includes the ability to render changes in value, detail, and dimension, depending on the object’s distance from the viewer.

Four Subscales for Value

LEVEL IV

Local Value Variations & Clarity 10 Irradiation Aerial Perspective Davis Assessment Score 5

LEVEL I

0 Application Drawing Beginning Drawing Intermediate Drawing

Figure 8: This graph indicates relatively strong upward trends for all four of the subscales for Value. The average student enters the program with a mid-Level II to low-Level III understanding and exits with a mid-high level III understanding of value and its individual components.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 31

Ryman Arts Evaluation Report, July 2006

Dimension

1. Linear Perspective Linear Perspective is the geometric system used to draw 3-D objects and their relationship to each other in space. Parallel lines appear to converge at a vanishing point or points in the distance, at the eye level of the observer. Realistic drawing skills include the ability to identify and draw objects with parallel lines converging at eye level. This may require one or more vanishing points.

2. 3-D Objects All natural and manufactured objects are composed of simple 3-D objects such as spheres, cylinders, cubes, cones, and pyramids, their combinations and variations. 3-D objects range from simple to complex. Realistic drawing skill includes the ability to draw 3-D objects ranging from simple to complex.

3. Contour Lines Structural lines that portray and enhance the surface volume and dimensionality of an object are called contour lines. Contour lines on a curved surface can be seen as straight or curved depending on the degree the object is foreshortened and the perspective of the viewer. Contour lines on a flat surface are straight except when they change direction at a corner.

Realistic drawing skill includes the ability to draw structural lines that portray or enhance the three- dimensionality of an object at any degree of foreshortening.

4. Size & Placement An object appears smaller in the visual field as it gets farther away from the viewer. Eye level affects the placement of an object in the visual field. The baseline of an object below eye level will appear higher on the picture plane as it gets farther away from a viewer. The baseline of objects above eye level will appear lower on the picture plane as it gets farther away from a viewer. Realistic drawing skill includes the ability to draw an object of any size and at any location in the distance so it maintains a natural size and proportion relationship with other objects.

Four Subscales for Dimension

LEVEL IV

Linear Perspective Three-Dimensions 10 Contour Size & Location Davis Assessment Score 5

LEVEL I

0 Application Drawing Beginning Drawing Intermediate Drawing

Figure 9: This graph indicates relatively moderate upward trends for most of the subscales for Dimension. The average student enters the program with a mid-Level III understanding of Three-Dimensionality, Contour, and Size & Location and exits with a high level III understanding of these components. The obvious outlier here is linear perspective. The average student enters with a low level II understanding and exits gaining an entire level. The majority of these gains seem to be made during the beginning class. However, this should be interpreted cautiously because very few drawings could be scored for linear perspective because of the subject matter of the drawings.

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 32

Ryman Arts Evaluation Report, July 2006

Appendix E: Copy of the Davis Drawing Assessment Scoring Rubric

R:\NEA Expanding Project\Packet\Peppler Overview of Ryman Program PUBLIC.doc 33