® Research Design, Program Evaluation, Learning Assessment, Statistical Analysis, and Instruction in Education and Technology $#%%$# Coral Reef Adventure An IMAX® Dome Film !#! $ ! & ! !#$! " #! "! !# P.O. Box 3124 • Leesburg, VA 20177 Fax: (703) 443-9697 Telephone: (703) 443-9997 art_ [email protected] Table of Contents EXECUTIVE SUMMARY PAGE Project Description ............................................................................................................. i Evaluation Goals ................................................................................................................. i Evaluation Design ............................................................................................................... i Research Procedure ............................................................................................................. i Evaluation Findings ........................................................................................................... i EVALUATION REPORT Introduction ........................................................................................................................... 1 Project Description ..................................................................................................... 1 Research Issues and Goals ...................................................................................... 1 General Evaluation Design .................................................................................... 2 Research Procedures and Measures .................................................................. 3 Demographics ............................................................................................................... 5 Evaluation Findings ................................................................................................... 6 Prior Interest in Coral Reefs ............................................................................... 6 Prior Self-Reported Knowledge About Coral Reefs .................................... 7 Learning Outcomes ............................................................................................... 7 Overall Appeal of Coral Reef Adventure ...................................................... 10 The Film’s Level of Visual Interest ................................................................ 11 The Film’s Level of Entertainment ................................................................. 11 The Film’s Pace ..................................................................................................... 12 Amount of Information Contained in Film ................................................. 12 Meeting Viewer’s Expectations ....................................................................... 12 What Viewers Like Most About the Film .................................................... 15 What Viewers Like Least About the Film .................................................... 22 What Surprises Film Viewers .......................................................................... 26 What Disappoints Film Viewers ..................................................................... 30 What Confuses Viewers ..................................................................................... 33 The Lobby Exhibit’s Impact .............................................................................. 34 References .............................................................................................................................. 37 Acknowledgements .......................................................................................................... 37 Summative Evaluation of Coral Reef Adventure ® An IMAX Dome Film Executive Summary September 18, 2003 PROJECT DESCRIPTION – The summative evaluation reported here focuses on two project components: (1) an IMAX® Dome film, titled Coral Reef Adventure, and (2) an associated interactive exhibit, titled Weird, Wild and Underwater, that is intended to be display in museum and science center lobbies. EVALUATION GOALS – The general goals for this summative evaluation study are to assess the appeal of Coral Reef Adventure and the lobby exhibit; acquisition of scientific knowledge and understanding as related to the project’s learning goals and understanding of film topics; and the impact that the exhibit has on viewers of the film. EVALUATION DESIGN – A separate-sample pretest-posttest design was used to evaluate Coral Reef Adventure in a natural theater setting with a random sampling of film viewers. This research design was found effective in evaluating the IMAX® Dome films Stormchasers (Flagg & Johnson, 1996), Special Effects (Flagg & Johnson, 1997), Everest (Johnson, 1998), and Dolphins (Johnson, 2000) and was applied to Coral Reef Adventure to add to the baseline knowledge MacGillivary Freeman Films has acquired about the effects of large-format films. RESEARCH PROCEDURE – Surveys were administered to randomly selected audience members either prior to or immediately following showings of Coral Reef Adventure at theaters located in St. Paul, Minnesota and San Jose, California. All of the survey respondents were 18 years of age or older. Single adults accompanied by children below the age of five and adults who were part of a group of five or more were excluded. Data collection efforts were performed at each of the two locations on both weekdays and a weekend day. Questions on the pre- and post-viewing surveys include content questions to measure learning outcomes and questions to assess viewers’ reactions to the film. To gain insight into the impact that the interactive coral reef lobby exhibit has on viewers of Coral Reef Adventure, a comparison of learning outcomes was performed between San Jose, which had the exhibit displayed adjacent to the theater’s lobby, and St. Paul, which did not have the exhibit. Data was collected in St. Paul over the course of 21 showings of the film on July 17-19 (Thursday thru Saturday). The total number of usable surveys collected in St. Paul (N=407) included 202 pre-viewing surveys and 205 post-viewing surveys. Similarly, data was collected in San Jose over the course of 17 showings of the film on July 20-24 (Sunday thru Thursday). The total number of usable surveys collected in San Jose (N=386) included 189 pre-viewing surveys and 197 post-viewing surveys. Demographics regarding gender, age group, education, occupation (related or not related to science), and number of IMAX® and/or OMNIMAX films seen are presented in the report. Analyses revealed that the samples did not differ significantly with respect to these classification variables. EVALUATION FINDINGS – The results summarized below reflect a portion of the findings contained in the body of the report, which contains additional finer-grained findings along with survey respondents’ quoted remarks. i Impact Evaluation Prior Interest in Coral Reefs. Prior to viewing Coral Reef Adventure, respondents were asked to rate their interest level in learning about coral reefs. Of the 388 responses received, 73.8% were either “very interested” or “moderately interested.” About 18.8% of responding viewers reported being “a little interested,” and 7.5% of the respondents to this inquiry reported “not interested at all.” On average, respondents in St. Paul gave their interest rating a 2.91 on a four-point Likert scale ranging from 1 (not interested at all) to 4 (very interested). San Jose respondents generally rated their interest as 3.00. On average, the entire pre-viewing sample (St. Paul and San Jose combined) gave their interest in coral reefs a 2.95 rating, with 73.7% describing their interest level as either “very interested” or “moderately interested.” Prior Self-Reported Knowledge About Coral Reefs. Participants in the pre-viewing sample did not report high levels of knowledge about coral reefs prior to viewing the film. Of the 389 members of the pre-viewing sample, 5.4% felt they knew “a lot,” and about one-fourth (23.1%) reported knowing “a moderate amount.” A larger percentage of the pre-viewing sample(62.0%) indicated that they know “a little,” and about 9.5% reported having no knowledge about coral reefs. On average, respondents in St. Paul gave their prior knowledge of coral reefs a 2.33 rating on a four-point scale ranging from 1 (I know nothing) to 4 (I know a lot). San Jose respondents generally rated their prior knowledge as 2.16. On average, the entire pre-viewing sample gave their knowledge about coral reefs a 2.24 rating, with 62.0% indicating they only know “a little.” Learning Outcomes. Both the pre- and post-viewing surveys included a knowledge test to assess understanding of content associated with Coral Reef Adventure learning goals. Toward this end, film viewers in St. Paul and San Jose were asked to provide responses to 8 true-or-false and 2 multiple-choice content questions. Correct answers received 1 point. Thus, there is a total of 10 points possible for each survey. Criterion for question selection is based upon a correspondence with film content. The post-viewing mean achievement score for the St. Paul sample is 8.55, significantly higher than the pre-viewing mean score of 5.39. Thus, the learning outcomes resulting from viewing Coral Reef Adventure is statistically significant at this site. More specifically, there was significant improvement on each of the content questions when considered independently. Similar to St. Paul, the post-viewing mean achievement