Art Museum Digital Impact Evaluation Toolkit
Total Page:16
File Type:pdf, Size:1020Kb
Art Museum Digital Impact Evaluation Toolkit Developed by the Cleveland Museum of Art’s Office of Research & Evaluation in collaboration with Rockman et al thanks to a generous grant from the National Endowment for the Arts. 2018 Content INTRODUCTION ..........................................................2 01 VISITOR CONTEXT .....................................................3 Demographics ............................................................4 Motivations ..............................................................5 Visitation Frequency .......................................................6 02 CONTEXT OF THE DIGITAL EXPERIENCE ...................................7 Prior Digital Engagement ................................................... 8 Timing of Interactive Experience ..........................................9 03 VISITORS’ RELATIONSHIP WITH ART .....................................10 04 ATTITUDES AND IDEAS ABOUT ART MUSEUMS .............................12 Perceptions of the Art Museum ..............................................13 Perceptions of Digital Interactives ............................................14 05 OVERALL EXPERIENCE AND IMPACT ....................................15 06 AREAS FOR FURTHER EXPLORATION .....................................17 ABOUT THE CLEVELAND MUSEUM OF ART ...................................19 Introduction The Art Museum Digital Impact Evaluation Toolkit in which the experience was studied. The final (AMDIET) is intended to provide artmuseums and version of data collection instruments took the arts organizations with the ability to evaluate form of paired pre-visit and post-visit surveys, an the roles and impacts of digital interactives on essential process to understanding the impact the visitor experience. Overall, this toolkit is of any variable in a single museum visit. In the meant to empower organizations not just to case of the CMA, ARTLENS Gallery served as the evaluate the level of success of specific digital interactive factor in the study. From the results, interactive experiences but also to evaluate researchers determined which metrics and the overall effects these newer museum methods can show impact and change, resulting elements have on overall visitor experiences and in this toolkit that examines evaluation of visitor relationships with art. context, the context of the digital experience, visitors’ relationship with art, attitudes and ideas This toolkit was developed by the Cleveland about art museums, and overall experience and in Museum of Art (CMA) in collaboration with impact. Rockman et al as part of a study aimed at better understanding the outcomes of digital engagement, including: • How do these interactives extend and expand the relationship visitors have with the collection and the overall museum visit? • Are new visitors and/or millennials more likely to visit due to these types of technologies? • What is the value of this interactive engagement, and what metrics best measure this value? This toolkit is not intended to be final or finite. Rather, AMDIET should serve as a reference The evaluation of digital interactives at the CMA point for adaptation and expansion for individual involved multiple iterations of data collection, projects to meet the evaluation capacities and evaluating not just the experience but the way needs of various organizations. 2 01 Visitor Context What background information is important to know about your visitors when considering their engagement with digital interactives at the museum? Demographics Motivations Visitation Frequency 3 01 Visitor Context: Demographics USE AT CMA IMPLEMENTING AND ADAPTING Collecting visitor demographics enabled the With a pre-visit/post-visit survey methodology CMA to measure to what extent ARTLENS within a short time frame, demographics will not Gallery was serving anticipated audiences. Not change and therefore only need to be asked on only that, it allowed researchers to look closely one instrument. The post-visit survey is likely at how museum expectations, perceptions, and to have more questions, so demographics can experiences may differ among groups both provide a good balance to the length of the pre- dependent and independent of engagement visit survey. Questions and answer categories with digital components. Demographic data should be adjusted to align with an organization’s also allowed for comparison between survey established categories for age, group types, etc. participants and the overall Cleveland Museum Examples of Art audience. Pre-Visit: What is your age? 18–24 55–64 25–34 65–74 35–44 75+ 45–54 Pre-Visit: Including yourself, how many people in your group are within the following age ranges: Children 6 and under: VALUE OF METRICS Children 7 to 12: Children 13 to 17: Demographic questions should only be asked Adults 18 to 34: Adults 35 to 54: when essential to the goals, process, and validity Adults 55 to 74: Adults 75 and over: of the study. The CMA instruments asked about age, group composition, membership status, and zip code in order to align with the intended audiences but did not ask about race/ethnicity or gender, as these were not necessary toward measuring success. 4 01 Visitor Context: Motivations USE AT CMA IMPLEMENTING AND ADAPTING Motivators toward CMA visitation were The CMA study instructed visitors to select a included on the pre-visit survey to explore, maximum of three motivational categories, a through a comparison with identified satisfying prioritization which can only be ensured through experiences on the post-visit survey, the extent a digital survey collection method. This method to which motivations were met by the museum. also enables choices to appear in random order Researchers also explored whether certain to reduce respondent bias toward higher-listed motivations, or a combination of motivations, options. Other institutions may choose to ask indicate higher rates of ARTLENS Gallery about motivations via a Likert scale or similar visitation and related behavior among various rating. We advise against asking visitors to rank audiences. motivations by importance, as this was tested in the early part of data collection and proved VALUE OF METRICS confusing for many participants. The motivations of visitors have long been known Example to direct many elements of museum experience, as established by the works of, among others, Pre-Visit: Which THREE factors were most important in motivating you to visit the John Falk and Lynn Dierking. This study aimed museum today? to look at how these established motivational To have fun and be entertained categories and subcategories influenced digital To learn something new or feel interaction and potential impact. intellectually stimulated To socialize or be with friends and family To relax or slow down To feel inspired or admire things that are beautiful or interesting To have a new experience (by visiting a museum or exhibition I haven’t seen) To give friends or family an enjoyable experience None of the above 5 01 Visitor Context: Visitation Frequency USE AT CMA IMPLEMENTING AND ADAPTING Knowing which study participants were new or Being a free-admission museum in a midwestern infrequent visitors was necessary in the CMA city, the CMA has a high visitation rate of study to provide researchers the opportunities to frequent visitors. If your institution sees higher 1) study the impact of ARTLENS Gallery on this numbers of infrequent or first-time visitors, particular target audience; 2) compare visitation having many categories for frequent visitation rates between the study sample and overall CMA may not apply. As with demographic questions, attendees; and 3) determine whether new or categories concerning visitation should reflect infrequent visitors’ perceptions and motivations institutional norms. aligned with museum goals for the interactive Examples space. Additionally, researchers hoped to learn more about how visitation to non-art museums Pre-Visit: How often do you visit the Cleveland Museum of Art? impacted expectations and perceptions of art More than 6 times a year museums. 4–6 times a year VALUE OF METRICS 2–3 times a year Visitation frequency of your museum and Once a year comparable cultural areas can shed light on Once every few years whether a visitor is new to just your institution Have not visited in at least 5 years or is less familiar with the broader idea of art Today is my first visit to the CMA museums and other places of informal learning. Research findings suggest that willingness Pre-Visit: In the last year, how often have you visited… to use digital tools may More than 4 to 6 2 to 3 Once Not at all 6 times times times be connected to these ...any OTHER art museum? areas of familiarity ...a non-art museum (including and comfort. zoos, aquariums, science centers, and historic sites)? 6 02 Context of the Digital Experience What do you need to know about when/where/how visitors engaged with digital interactives in order to interpret their impact on the visitor experience? Prior Digital Engagement Timing of Interactive Experience 7 02 Context of the Digital Experience: Prior Digital Engagement USE AT CMA IMPLEMENTING AND ADAPTING Since data collection began nearly six months While the CMA chose to ask about prior after the completion of the newest iteration of engagement in the interactive space, other ARTLENS Gallery, it was possible many of the institutions may instead want to ask about museum’s frequent visitors had visited the gallery familiarity