Year Two Evaluation of the LA's BEST After School Arts
Total Page：16
File Type：pdf, Size：1020Kb
Year Two Evaluation of the LA’s BEST After School Arts Program: Evaluating Student Learning in the Arts Final Report -- August 2006 Kylie A. Peppler, [email protected] James S. Catterall, [email protected] UCLA Graduate School of Education and Information Studies Graduate School of Education & Information Studies University of California, Los Angeles Los Angeles, CA 90095-1522 (310) 455-0785 With the assistance of: Ms. Kimberly Feilen, Ph.D. Student, Urban Schooling Copyright © 2006 The Regents of the University of California This work was supported by grant number 04063973 from LA’s BEST ASAP to the UCLA Graduate School of Education and Information Studies. The findings and opinions expressed in this report are those of the authors and do not necessarily reflect the positions or policies of LA’s BEST, the After-School Arts Program (ASAP), or the University of California. LA’s BEST After School Arts Program Evaluation Report, August 2006 TABLE OF CONTENTS I. Abstract……………………………………………………………………………… 3 II. Introduction……………………………….…………………………………………. 4 III. Methods and Participants……………………………………………………………. 5 IV. Key Findings Summary……………………………………………………………………. 7 On attendance……………………………………………………………….. 8 On prior experience in the arts……………………………………………… 8 On the ASAP Dance Program……………………………………………….10 On the ASAP Music Program…………………………………………...…..13 On the ASAP Visual Arts Program………………………………………….16 On the ASAP Theatre Program…………………………………...…………19 V. Conclusions and Recommendations………………………………………....….....…22 VI. References……………………………………………………………………….........25 VII. Appendices………………………………………………………………………...….26 Page 2 LA’s BEST After School Arts Program Evaluation Report, August 2006 SUMMARY OF FINDINGS August 2006 ABSTRACT As a follow-up to the first year evaluation, this study sought to assess student learning in the arts in each of four disciplines sponsored by the LA’s BEST After-School Arts Program (ASAP): music, drama, visual arts, and dance. The purpose of this evaluation was to investigate the nature and extent of learning in each of the respective art forms during the 10-week ASAP artists’ residencies. This reflects an important issue recognized in the arts and learning literature as well: few studies adequately (or at all) measure arts learning in efforts to connect learning in the arts to other child or adolescent developments (Deasy, 2002). The evaluation methods included regular videotaped observations of residency sessions, surveys of students’ prior experiences in the arts, analysis of attendance data, and in-depth interviews of the artists after completion of their residencies. The videotape recordings became sources of insight for expert appraisals of the students’ artistic development by members of the evaluation team as well as by an outside specialist in each discipline. The study found that the LA’s BEST After School Arts Program provided students with high quality learning experiences in the arts that led to significant achievements in standards-based learning in the arts. The study also found that students’ prior experiences in the arts had no significant effects on learning and that program attendance levels mattered significantly. Page 3 LA’s BEST After School Arts Program Evaluation Report, August 2006 Introduction Today, national and state funding agencies are frequently requiring evaluators to document arts learning in addition to learning in other academic areas.1 Traditionally, the arts have been little considered within research on evaluation and design. However, there is an increasing need for more research and evaluation in this area. While standardized test scores, surveys, and other types of measures may speak to the transfer of learning in other academic areas, the question remains: how do we document, describe, and analyze learning in the arts? As the arts become integrated into other academic areas, this becomes important to evaluators in other fields, as well. Arts education has been swept up in two movements during the past decade. One has been the increasing interest of cognitive researchers in the ways that learning in the arts may transfer to non-arts learning (Hetland, 2000; Deasy, 2002; Bransford & Schwartz, 2000). The effects of music learning on spatial reasoning skills comprise one example; classroom drama and reading comprehension provide another. The second movement has been the advent of formal standards for learning in the arts disciplines, standards often developed and codified at the state level. Alignment with prevailing arts standards is now a requirement of new public programming in the arts as well as a mandatory condition in applications for state and federal grants for elementary and secondary arts education. Both research on transfer and assessment in arts education policy stand to benefit from high quality measurement of learning in the arts, but for different reasons. In the case of standards based arts education, knowing whether or not standards have been met is central to the place of standards in the policy process. It’s one thing to claim that a program design addresses the visual arts skills spelled out at a given education level. It’s another to look back and be able to say that the children acquired those skills. The closer arts education comes to real participation in education accountability systems, the more developed must be the associated measures of learning in the arts. Cognitive research related to the arts would also benefit from incorporating measures of artistic learning in research designs. This is because of a little recognized shortcoming of treatment-control group experiments: namely their traditional reliance on differences between group average scores to gauge whether or not the treatment mattered. On the one hand, finding that the arts group gained significantly more than the control group on a criterion learning measure argues for a true effect of the arts experience on learning. On the other hand, adding a measure of individual arts learning to the research design creates a powerful opportunity to test the hypothesis that learning in the arts actually produces the effect. This reflects an important issue recognized in the arts and learning literature, as well: Few studies adequately (or at all) measure arts learning in efforts to connect learning in the arts to other developments (AEP, 2004). The basic design is simple: test for a relationship between individual learning in the arts and individual learning in the target skill(s). Unraveling the Impact of the Arts Documenting learning in and through the arts can be a difficult challenge for evaluators for several reasons. Oftentimes evaluators are asked to assess programs that serve youth in a variety of disciplines, all of which may have differing instructional goals. Even within a single discipline such as music, there may not be a standardized curriculum with similar instructional and learning goals. While this is a tremendous challenge to evaluators, this is often seen as the strength of such arts programs. How 1 As examples, both the Los Angeles and Santa Monica (CA) Departments of Cultural Affairs, community arts education sponsors, were attuned and insistent upon drawing the California Visual and Performing Arts Standards into the design and goals of recent evaluations. Page 4 LA’s BEST After School Arts Program Evaluation Report, August 2006 then does one begin to assess the benefits of participation in each of these art forms while still saying something about the program as a whole? Making the task even more difficult is the overwhelming challenge of accurately documenting student learning and expression in art forms that don’t easily translate to standardized measures or test scores. One example of this type of study design is shown in a visual arts education research project we conducted for the Ford Foundation in 2003 (Catterall & Peppler, 2003, 2007). In this study of third- graders in a five-month program of high quality visual arts instruction, we found significant effects on individual self-efficacy beliefs and internal attributions for success when comparing program and control group students. We took this work a step further by measuring art-skill development of program children by assessing drawings produced at the start of the program and drawings produced at the program’s end, about 20 weeks later. The children who showed more gains in drawing skills, i.e., those who showed more arts learning in our experiment, also tended to show greater gains in personal efficacy and attribution beliefs. This tightened our argument that learning in the arts was, in fact, tied to gains in these motivation-related developments. In this evaluation we augment this design by assessing student learning in the arts in each of four disciplines sponsored by the LA’s BEST After-School Arts Program: music, drama, visual arts, and dance. The purpose of this evaluation was to investigate the nature and extent of learning in each of the respective art forms during the 10-week time frame of residencies. The design of the new assessment was propelled by the program’s need to demonstrate alignment with the California Visual and Performing Arts (VAPA) Standards. The evaluation methods included videotaped observation of residency sessions during weeks one and ten to use as pre- and post-test measures, surveys of students’ prior experiences in the arts, attendance data, and in-depth interviews. On the whole, this evaluation sought to better understand the following four questions: . What was accomplished