The Arthur Interactive Media Buddy Project Summary of Research Findings

The Institute for Applied Research in Youth Development Tufts University June 2017

The Arthur Interactive Media (AIM) Buddy Project is an innovative, research-based, supplemental, social, emotional, and character development program that leverages the power of the beloved PBS children’s series ARTHUR, interactive technology, and cross-age buddy relationships in elementary school classrooms. The AIM Buddy Project, which has as its primary focus first- and second-graders, is composed of five units, each focused on one of the five AIM topics: empathy, honesty, forgiveness, generosity, and learning from others. Each unit features an online comic or game, based on an ARTHUR episode, along with related non-digital activities. A comprehensive educator’s guide embedded with companion videos offers step-by- step instruction and illustrated strategies and tips for preparing and implementing a successful AIM Buddy Project.

WGBH, the leading producer of children’s media content for PBS, including ARTHUR, and the Institute for Applied Research in Youth Development at Tufts University worked together to iteratively develop and test program components, culminating in a year-long research study during the 2015-2016 school year, with delayed testing in fall 2016. Study participants included 90 teachers and 1,155 students in first and second grades (known as Little Buddies) and fourth and fifth grades (known as Big Buddies), across two Massachusetts school districts.

In the research study, we sought to answer the following questions regarding program impact, implementation, and both teacher and student perceptions of the program:

1. Does participation in the AIM program change student self-ratings of social, emotional, and character-related attributes, perceptions of their classroom climate, and values in school? 2. Were the AIM materials implemented with fidelity to the curriculum? What were AIM teachers’ perceptions of the program and its impact on their students? 3. What were students’ perceptions of the AIM comics and games, their buddy relationships, and the program overall?

AIM Research Report July 2017 2

Study Design In total, nine schools, 90 classrooms, 1,155 students, and 540 parents participated across three study conditions: the AIM program condition, the Martha Speaks Reading Buddies program comparison condition (a buddy program designed to increase oral vocabulary and which has a similar implementation schedule as AIM), and an “As-Is” comparison condition (in which teachers did not alter their planned instruction and participated in survey administration only). Participants were relatively evenly distributed across AIM and other schools, with the AIM program administered to four schools, 41 classrooms, and 588 students (270 Little Buddies and 318 Big Buddies) in comparison to the five Martha Speaks and As-Is schools which reached a total of 49 classrooms and 567 students.

Research Instruments To examine the relation between the AIM program and student outcomes, several research instruments were designed specifically for AIM and used across all three program conditions.

Impact evaluation tools were administered to students, teachers, and parents from all study conditions. These included: • Student self-report measures assessing five key character attributes (empathy, generosity, honesty, forgiveness, and learning from others), values in school (e.g., getting good grades and being a good person), social and emotional attributes (e.g., self-regulation, empathy), school and classroom climate, and bullying (for older students only); • Teacher-report measures assessing classroom and school climate, media-related attitudes, bullying perceptions and attitudes, and ratings of a subsample of students on character-related, social, and emotional attributes; and • Parent-report measures assessing school climate, media-related attitudes, and evaluations of their child’s character-related, social, and emotional attributes.

In addition, we collected process evaluation, or program feedback, data, which included: • Evaluation booklets assessing ongoing feedback from students and teachers about specific features of the program, reflections of the buddy experiences, and their overall experiences at the completion of the program; AIM Research Report July 2017 3

• Student interviews with a subsample of students (n= 60) asking them about how they think about character, social, and emotional issues. AIM and Martha Speaks program students were asked to provide feedback about their experiences and perceptions of the program, the program material, and working with their buddy; • Teacher interviews with a subsample of teachers (n = 36) asking them about their experiences and perceptions about character education and social and emotional learning; AIM and Martha Speaks program teachers were asked to provide feedback about their experiences and perceptions of the program material; and • Buddy videos of a subsample of buddies (n = 366) in the AIM program across the five topic interactive sessions. These videos were recorded to capture the quality of buddy relationships, responses to the comics and games, and depth of conversations about the key topics within the buddy pairs.

Assessing Program Impact The goal of the impact evaluation was to examine the links between AIM program participation and outcomes of interest, which include students’ social, emotional, and character attributes, students’ prioritized values in school, their understanding of the program topics, and quality of the buddy relationships. Many of our analyses involved assessment of student outcomes in relation to the fidelity of the program implementation. That is, we contrasted the outcomes linked to experiencing the AIM program in high versus low fidelity schools.

Note: “Fidelity of program implementation” means that schools enacted the AIM program fully. Of the four schools studied, two had markedly high program fidelity. This distinction was based on researcher observations and teacher surveys, showing that these schools had distinct characteristics, such as a whole-school adaptation (e.g., posters relating to the AIM program on the school walls) and high teacher responsiveness and program adoption (e.g., covering more program sessions, making connections to other subject matters and extending the learning opportunities beyond the program content). Of the four schools in the AIM program condition, one of the high fidelity schools was located in an affluent suburban school district; the other was located in a middle- to low-income urban school district. Given this distribution, program fidelity and socio-economic status were not found to be related.

Character and Other Related Attributes Before and after the program implementation (at the start and end of the school year), we asked students to use a paper-and-pencil survey to rate themselves on the five core character attributes we studied in the AIM program: empathy, honesty, forgiveness, generosity, and learning from others.

Students also rated themselves on other attributes important to character, including future- mindedness, tolerance, and school-related indicators, such as perceptions of the school climate. In analyses assessing pre- and post-program, the results showed that: • Little Buddies who participated in the AIM program in high fidelity programs had statistically significant increases in their empathy, future-mindedness, and positive perceptions of their classroom climate. AIM Research Report July 2017 4

• Both Little and Big Buddies who participated in the AIM program in high fidelity classrooms also had statistically significant increases in their levels empathy and tolerance. • There were no significant overall increases in students’ self-rated honesty, forgiveness, or generosity. Students’ levels of these characteristics did not change in relation to the AIM program participation.

Values in School In the student survey, students were asked to select what they value most in school. They could only choose one from a menu of six options which included “being liked by the teacher,” “getting good grades,” and “to be a good person.” • After participating in the program, 23% of Little and Big Buddies in the AIM program condition selected “being a good person” as what they value most in school, compared to only 14% of students from the non-AIM schools. This difference between AIM and all other students was statistically significant and not found in the pre-test data (prior to program participation). •

Students in AIM Program Schools Students in Other Condition Schools

Defining Empathy in Big Buddies We studied how mentoring Little Buddies might affect how Big Buddies understand core constructs. We asked students to give definitions of empathy immediately before and after buddy pairs engaged with the online comic, “So Funny I Forgot to Laugh.” Students responded to the prompt, “What does it mean to show or have empathy?” and their responses (both before and after the comic) were categorized based on the empathy literature. There were increases in major components of empathy, as shown below:

• 20% defined empathy as involving perspective taking, an increase from 6% prior to the comic. Post-comic definitions included examples like: “It means to put yourself in someone else’s shoes” or “to really try and understand what they’re going through” • 12% defined empathy as emotional sharing (e.g., “to feel someone else’s feelings”), an increase from 9% prior to the comic. • 12% also provided a definition after the comic, whereas they had provided an incorrect or no definition prior to the comic. AIM Research Report July 2017 5

Delayed Effects To assess delayed or longer-term program effects, we surveyed students for a third time when students returned to school in the fall of the 2016 school year. We asked students who participated in the AIM program and those who did not participate about their understandings of the concepts behind the core character-related attributes. We gave students four possible answers describing each attribute, and students were asked to pick the best option.

Issues related to empathy, forgiveness, generosity, honesty, and learning from others are central to navigating relationships and contributing as a caring citizen from childhood to adulthood. A central goal to the AIM program was to use creative and interactive media through buddy scaffolding to help children understand the complexities of such topics. The delayed assessments helped us discern whether children had grown in their understanding.

In analyses assessing the delayed program effects, findings showed that both Little and Big Buddies who participated in the AIM program were: • Four times more likely to show a better understanding of honesty than those who did not participate in AIM. • Twice as likely to show a better understanding of empathy and generosity than students who did not participate in the AIM program. • Two and a half times more likely to show a better understanding of forgiveness and learning from others than those who did not participate in AIM.

Thus, Little and Big Buddies who participated in the AIM program showed a greater under- standing of the concepts involved in the character attributes (and empathy), compared to students who did not participate in the program.

Further analyses on the delayed program effects were assessed with effects differed in relation to the level of fidelity with which schools implemented the AIM, program. Findings showed that: • Little and Big Buddies from high fidelity classrooms showed more confidence in choosing the best options compared to all other students. That is, students from the high fidelity AIM schools were significantly less likely to pick “I don’t know” as an option to selecting their answer, compared to all other students. Notably, students who did not participate in the program were significantly more likely to say, “I don’t know” what empathy, generosity, and making assumptions about other people mean. • Little and Big Buddies from low fidelity classrooms still did better at grasping the concepts behind the attributes, compared to students who did not participate in the program at all. That is, students in the low fidelity AIM schools chose the best options for the virtues about as we would expect (by statistical chance), but the students who did not participate in the program were significantly less likely to provide the best options. They performed lower than at chance levels. Thus, even when students were in low fidelity schools, the AIM program seems to have made an impact on understanding the complexities of challenging topics.

AIM Research Report July 2017 6

Assessing the Quality of Buddy Relationships Because an essential component of the AIM program is to foster positive relationships between Little Buddies and Big Buddies, we expected that the AIM program would promote positive buddy relationships.

Prior to starting each unit of the AIM program, students were asked, “How excited are you to work with your buddy?” and after each unit, students were also asked, “How much do you like working with your buddy?” (on a scale of 1 = Not at all to 4 = A lot). Finding showed that: • Students in the high fidelity schools reported significantly higher levels of excitement and liking in working with their buddies than students in the low fidelity schools. • The differences in excitement and liking were evident after the first unit was completed, and the gap widened as the schools progressed through the units.

3.90 3.79 3.80 3.79 3.77 3.73 3.75 3.70 3.68 3.63 3.60 3.54

3.50 3.43 3.42 3.40 3.30 1 2 3 4 5

AIM High Fidelity AIM Low Fidelity

Fig 1. Excitement for buddy interaction before each program session

3.90 3.80 3.79 3.77 3.80 3.79 3.75 3.70 3.62 3.60 3.54 3.50 3.43 3.40 3.40 3.37

3.30 1 2 3 4 5

AIM High Fidelity AIM Low Fidelity

Fig 2. Quality of buddy interaction after each program session

These buddy relationship indicators also predicted outcomes related to intentional self- regulation (e.g., confidence in seeking help when needed). For students in the high fidelity program schools, positive changes in excitement to work with your buddy predicted statistically significant increases in self-regulation from pre- to post-test. AIM Research Report July 2017 7

The “Arthur Effect” Arthur continues to be both popular and ubiquitous among young elementary school students. Analysis of student surveys reveal that: • 91% of students reported current or past viewing of the ARTHUR television show or reading of Arthur books (“Arthur exposure” group). • 86% of students reported current or past viewing of the ARTHUR television show (“ARTHUR TV show exposure” group). • 68% of students reported current or past reading of Arthur books (“Arthur book exposure” group).

Students’ familiarity with and exposure to Arthur (through the television program and/or books) may have affected students’ experience with the AIM program. Analyses show significantly robust differences in their liking and interests between students with Arthur exposure and students with no exposure–whether it was past or present exposure.

Specifically, students with any Arthur exposure reported significantly higher liking of the comics/games, compared to students with no exposure. Similarly, students with any Arthur exposure reported significantly greater willingness to play more Arthur comics and games, compared to students who had no such exposure.

Assessing Program Implementation and Response Teacher Feedback Analysis of teacher feedback booklets and interviews conducted after program participation showed that, despite competing demands for their time in the classroom, teachers: • Valued the AIM program goals. “First of all, in my class, I am a primary teacher. We live and die by the social-emotional. You can’t teach a child reading unless you’re teaching the child. You have to teach the child first. This fits in with that because it helps a child to see the situations that are happening and the way a character is responding to them.” –Little Buddy Teacher • Reported a high capacity to integrate the program into other subjects. “What I’ve used often is to say, ‘Let’s imagine a speech bubble. What do you think this character is thinking? Remember Arthur.’ It’s really helped me in my literacy education. It’s also helped me when I have a problem that comes up on the playground. I can say, ‘I know AIM Research Report July 2017 8

you probably don’t remember, but we’re trying to imagine what might you have been thinking. Boys and girls, why do you think this happened? Can you have a speech bubble about that person’s head? What could you possibly think?’” –Little Buddy Teacher • Described the interactive comics and games as an innovative approach to navigating difficult topics. “I think, one, it's a very well thought out program that clearly took a tremendous amount of time to develop and come up with a format and also bringing in the interactive aspects of the game, or the video that they watched, so that being really successful, I think, made it better than some of the older, traditional ways of teaching social kinds of ... ‘Okay, we're going to look at the workbook and you're going to fill out this paper or we're going to look at this chart.’ ” –Big Buddy Teacher • Recognized the importance of buddy relationships. “I think we had a great relation- ship with the buddies... one child had said that I feel like now I know how to connect with people.” –Big Buddy Teacher • Found both the content of the comics and games and the curriculum to be highly relevant to the classroom and student experiences. “[Due to the program] They really have gotten open about what's been going on in their lives. Connecting with their emotions.” –Little Buddy Teacher

Analysis of teacher program implementation data revealed: • 81% of teachers completed all 20 sessions (3 prep sessions, 3 topic sessions each unit, 1 celebration, 1 conclusion session). • 82% reported that they were “very” to “mostly” satisfied with the program. • 80% of teachers reported “very” or “mostly” high student engagement across the program activities. • 86% said that the content was “excellent” or “good” across all units.

Student Feedback Students also responded very positively to the program. Analysis of student feedback booklets and interviews conducted after program participation showed that students: • Enjoyed the comics and games and working with their buddy. • Reported that the comics and games helped them talk with their buddy and learn about the topics. • Found the program to help them learn how to ask for help (in the case of “Little Buddies”) or mentor (in the case of “Big Buddies”)

Analysis of student program implementation data revealed (on a 1-4 scale ranging from “Not at all” to “A lot”): • Students gave a 3.39 average rating of liking all the interactive comics and games • Students gave a 3.42 average rating of interest in playing more interactive comics and games like in the AIM program AIM Research Report July 2017 9

• Students showed a high average rating of how much each interactive comic or game helped the buddies talk about each topic: o Empathy: 3.50 o Honesty: 3.40 o Forgiveness: 3.29 o Generosity: 3.32 o Learning from Others: 3.20

• Both Little (86%) and Big (70%) Buddies liked working with their buddies ‘a lot’ • 63% of Big Buddies said they helped their buddy when they needed help with something, and similarly, 56% of Little Buddies said they could ask their buddy for help • 79% felt happy working with their buddy, followed by 72% excited and 53% proud (these percentages were similar for Little and Big Buddies) • 58% of Big Buddies said what they learned most was how to be a good role model and in open-responses, they added: o “how to take care of someone and treat them right” o “how to be a better friend” o “how to think before acting” • 41% of Little Buddies said what they learned most was how to understand what others are thinking and feeling, and in open-responses, they added: o “how to be a good citizen” o “how to make a new friend” o “how to work and listen to others” • Big Buddies also said some of the following things, in response to what they would tell another student about the AIM program: o “I would say it’s very fun, and it helps you learn things, as well as bonding with younger students” o “I would say it’s [an] awesome program that helps you learn about different emotions” o “I would say that you will learn how to deal with other people in a kind and respectful way” o “I would tell them that the comics are fun and you learn a lot about how to be a better person. I before didn’t [sic] think as much as I should have about how others would feel. Now I am more aware of that, and I have this program to thank for it.” o “I would say it is an interesting way to tell important lessons”

Summary The AIM program was successfully implemented and very well received. Evaluation data demonstrated that the majority of teachers were highly satisfied with the program, and likewise felt that their students were very or mostly engaged throughout the program. The program’s content resonated with both Little and Big Buddies. Impact findings showed program effects, especially for Little Buddies and students in high program fidelity schools. Delayed AIM Research Report July 2017 10 findings also showed that the content of the program was long-lasting and resonated with the students; students who participated in the AIM program were more likely to understand the complexities of difficult topics than students who did not participate in the program, even 6 months following the program intervention. We are encouraged by the findings presented in the report. Our work shows the value of embedding interactive media into classrooms, as well as the complementing role of buddy relationships and accompanying curriculum, in promoting positive attributes among elementary school children. The research especially highlights the importance of high quality program implementation, or what we deem “high fidelity.” Schools that implemented the program with high fidelity show the greatest impact, including in the quality of buddy relationships. Based on our findings to date, we believe that the AIM program will be an important contribution to both research and educational practice and in highlighting the impact of interactive media on positive youth development.

AIM Research Report July 2017 11

Principal Investigators: Lacey J. Hilliard, Ph.D. Richard M. Lerner, Ph.D.

Co-Investigator Research Advisor: Milena Batanova, Ph.D. Edmond P. Bowers, Ph.D., Clemson University Graduate Research Assistants Elise M. Harris, M.A. Senior Research Coordinators: Kathleen M. Robinson, M.Ed. Akira Gutierrez, M.A. Danielle C. Stacey, M.A. AnneMarie McClain, M.Ed. Jonathan M. Tirrell, M.A. Research Staff: Undergraduate Research Assistants Heidi Johnson Sylvie Grenier Kyle Paul Christine Rattigan Claire Hoffman Alyssa Rivas Liam McCarthy Safiya Subegdjo

Funding by the John Templeton Foundation