Toward Data-Rich Models of Visitor Engagement with Multimodal Learning Analytics

Toward Data-Rich Models of Visitor Engagement with Multimodal Learning Analytics Jonathan Rowe, Wookhee Min, Seung Lee, Bradford Mott, and James Lester North Carolina State University, Raleigh, NC 27695, USA {jprowe, wmin, sylee, bwmott, lester}@ncsu.edu Abstract. Visitor engagement is critical to the effectiveness of informal learning environments. However, measuring visitor engagement raises significant challenges. Recent advances in multimodal learning analytics show significant promise for addressing these challenges by combining multi-channel data streams from fully-instrumented exhibit spaces with multimodal machine learning techniques to model patterns in visitor experience data. We describe initial work on the creation of a multimodal learning analytics framework for investigating visitor engagement with a game-based interactive surface exhibit for science museums called FUTURE WORLDS. The multimodal visitor analytics framework involves the collection of multichannel data streams, including facial expression, eye gaze, posture, gesture, speech, dwell time, and interaction trace log data, combined with traditional visitor study measures, such as surveys and field observations, to triangulate expressions of cognitive, affective, behavioral, and social engagement during museum-based learning. These data streams will be analyzed using machine learning techniques, with a focus on deep recurrent neural networks, to train and evaluate computational models of engagement using non-intrusive data sources as input (e.g., interaction logs, non-identifying motion tracking data). We describe distinctive opportunities and challenges inherent in using multimodal analytics within informal settings, as well as directions for utilizing multimodal visitor analytics to inform work by exhibit designers and museum educators. Keywords: Multimodal Learning Analytics, Informal Learning Environments, Interactive Tabletop Exhibits, Game-Based Learning. 1 Introduction Engagement is the cornerstone of learning in informal environments [1]. During free- choice learning, such as in museums and science centers, visitor engagement shapes how learners interact with exhibits, move around the exhibit space, and form attitudes, interests, and understanding of science. In recent years, the space of possibilities for visitor engagement has been enriched by the growing presence of advanced learning technologies within museums, including digital games, tangible devices, and augmented reality. Engagement with these technologies can be operationalized in many different ways, and research in the area spans a broad range of fields and theoretical perspectives [2]. Disentangling visitor engagement from related constructs, such as motivation, flow, interest, and self-regulation, is a common challenge. Engagement has also proven difficult to measure [2]. Much of the research on learner engagement has depended upon the use of subjective measures, such as self-reports, questionnaires, and interviews. These measures provide a snapshot view of visitor engagement, but they provide limited data for modeling visitor engagement at the process level. Observational methods have also been utilized, but they raise issues of scalability, as well as potential disruptive effects inherent in video recording, audio recording, and even written consent itself [3]. Recent developments in multimodal learning analytics show particular promise for addressing these challenges. Learning analytic techniques enable the creation of computational models for inferring complex relationships between variables, which can be utilized to detect the presence of engagement-related phenomena from non- identifying data, such as trace logs of learner behavior [4]. Multimodal learning analytics expands upon these methods by using multiple physical hardware sensors to concurrently capture multi-channel data on learner behavior and modeling salient patterns of learner experience using machine learning [5, 6]. Multimodal learning analytics has shown significant promise in laboratory and classroom environments [7, 8], but there has been comparatively little work investigating multimodal learning analytics in informal contexts, such as science museums. In this paper, we describe work on the design and development of a data-driven framework for investigating visitor engagement in science museums using multimodal learning analytics (Figure 1). We focus on visitor interactions with a game-based interactive surface exhibit about environmental sustainability called FUTURE WORLDS. By instrumenting FUTURE WORLDS with multiple hardware sensors, it is possible to capture fine-grained data on visitors’ facial expression, eye gaze, posture, gesture, conversation, dwell time, and learning interactions to triangulate key components of visitors’ cognitive, affective, behavioral, and social engagement during free-choice learning. We use these data streams to train and evaluate multimodal machine learning models to infer visitors’ engagement levels with non-intrusive data sources as input (e.g., interaction logs, non-identifying motion tracking data). We describe recent progress on the development of the multimodal visitor analytics framework, and discuss distinctive opportunities and challenges inherent in using this approach to devise computational models of visitor engagement. 2 Background and Related Work Learner Engagement. We adopt a conceptualization of engagement that is organized in terms of several core components: cognitive engagement, emotional engagement, behavioral engagement, and social engagement [9, 10]. Cognitive engagement describes individuals’ psychological investment in learning, which has close ties to motivation and interest as well as self-regulated learning [9]. Emotional engagement refers to individuals’ affective responses to learning, including attitudes, mood, and moment-to-moment emotional expressions, such as engaged concentration, delight, confusion, and surprise. Behavioral engagement refers to learners’ positive, on-task, and productive learning behaviors. In a museum context, low levels of engagement may appear as passive or shallow interactions with an interactive exhibit, whereas high-level behavioral engagement can manifest as productive exploration behaviors, as well as Figure 1. Multimodal visitor analytics framework. expressions of interest that extend outside of the exhibit (e.g., prompting a friend to try the interactive tabletop display). Social engagement acknowledges the key role of social interactions during learning in small groups, a common context in museums and other informal environments [10]. Adopting this conceptualization, we seek to utilize rich multi-channel data streams to identify salient patterns of meaningful visitor engagement that integrate cognitive, affective, behavioral, and social measures. Multimodal learning analytics. Advances in multimodal learning analytics have been enabled by the increased availability of low-cost physical sensors (e.g., motion-tracking cameras, eye trackers) combined with significant progress in machine learning tools and techniques. By taking advantage of information across concurrent sensor-based data channels, multimodal learning analytic techniques have been found, in many cases, to yield improved models compared to unimodal techniques [11]. This extends to a range of tasks within educational technologies, including automated detection of affective states [7, 12], computational models of assessment [13], and models of learner metacognition [14]. Although these applications have shown significant promise, the preponderance of work on multimodal learning analytics has been conducted in laboratory and classroom settings. Using multimodal learning analytics to investigate visitor engagement in informal environments is a natural next step for the field. 3 FUTURE WORLDS Testbed Exhibit To conduct data-rich investigations of visitor engagement in science museums, we utilize a game-based museum exhibit called FUTURE WORLDS. Developed with the Unity game engine, FUTURE WORLDS integrates game-based learning technologies and interactive surface displays to enable collaborative explorations of environmental sustainability in science museums [15]. In FUTURE WORLDS, visitors solve sustainability problems by investigating the impacts of alternate environmental decisions on a 3D simulated environment (Figure 2). The virtual environment is rendered from a top-down perspective on a 28” interactive surface display, a Microsoft Surface Studio 2. Learners tap and swipe to test hypotheses about how different micro- Figure 2. Screenshots from FUTURE WORLDS game-based museum exhibit. and macro-scale environmental decisions—such as modifying a region’s electricity portfolio or augmenting a farm’s waste management practices—impact the environment’s sustainability and future health. The effects of visitors’ environmental decisions are realized in real-time with 3D game engine technologies. FUTURE WORLDS’ focus on environmental sustainability targets three major themes—water, energy (both renewable and non-renewable), and food—and it facilitates exploration of the interrelatedness of these themes. Initial pilot testing with both school and summer-camp groups at our partner museum, the North Carolina Museum of Natural Sciences, have indicated that learners’ interactions with FUTURE WORLDS enhance sustainability content knowledge, as well as yield promising levels of collaboration and engagement as indicated by observations of learner behavior [15]. 4

Load more