Designing an Adaptive Assisting Interface for Learning Virtual Filmmaking Qiu-Jie Wu, Chih-Hsuan Kuo, Hui-Yin Wu, Tsai-Yen Li To cite this version: Qiu-Jie Wu, Chih-Hsuan Kuo, Hui-Yin Wu, Tsai-Yen Li. Designing an Adaptive Assisting Interface for Learning Virtual Filmmaking. WICED 2020 - 9th Workshop on Intelligent Cinematrography and Editing, May 2020, Norrköping, Sweden. 10.2312/wiced.20201126. hal-02960409 HAL Id: hal-02960409 https://hal.inria.fr/hal-02960409 Submitted on 7 Oct 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. EUROGRAPHICS 9th Workshop on Cinematography and Editing (2020) M. Christie, H.-Y. Wu, T.-Y. Li and V. Gandhi (Editors) Designing an Adaptive Assisting Interface for Learning Virtual Filmmaking Qiu-Jie Wu1, Chih-Hsuan Kuo1, Hui-Yin Wu2 and Tsai-Yen Li1 1National Chengchi University, Taiwan 2Université Côte d’Azur, Inria, France Abstract In this paper, we present an adaptive assisting interface for learning virtual filmmaking. The design of the system is based on the scaffolding theory, to provide timely guidance to the user in the form of visual and audio messages that are adapted to each person’s skill level and performance. The system was developed on an existing virtual filmmaking setup. We conducted a study with 24 participants, who were asked to operate the film set with or without our adaptive assisting interface. Results suggest that our system can provide users with a better learning experience and positive knowledge harvest. CCS Concepts • Human-centered computing ! Interaction design process and methods; User studies; • Computing methodologies ! Virtual reality; • Applied computing ! Media arts; 1. Introduction helpful interactions that enable the students to do something be- yond his or her independent efforts [MS08]. Based on this concept, Film cinematography has the function of eliciting emotions from we designed an adaptive assisting learning interface that builds on the audience, which influences not only how scripts are written, an existing virtual film set, which tutors the users through visual but also how the director chooses to direct the movie. The tech- and audio messages. The user is assigned an initial skill level at the niques applied by the cinematographer can lead to a wide variety beginning of the tutorial. As they advance through the hands-on tu- of interpretations of the same story. All of these show the role of torial, the system automatically adjusts the level of help according cinematography in telling the story of the film. It is important for to the user’s performance. We carried out a user study to evaluate the users studying film production to be able to experiment with the assisting interface and scaffolding mechanism that we have de- different ways of filming and observe the effects on the audience. signed, the results of which are reported in this paper. To improve one’s skills, we believe that practice makes perfect. The paper is organized as follows. First, the state of the art on However, typically, building a film set with real props and actors the scaffolding theory and virtual camera control is presented. We is costly and not readily available to most novice filmmakers. Re- then show how the adaptive assistant is constructed, and our system cently, virtual reality has been deployed to develop a virtual film architecture. This is followed by the description and results of a set [LLGC18] that allows users to take on the role of the cine- study in which we invited users to evaluate our system. Finally, we matographer, director, or editor, thus allowing trainees and students summarise the findings and limitations and offer perspectives on to experience the process of filmmaking at a lower cost, and with future work. minimal professional equipment and personnel. Unfortunately, vir- tual reality technologies introduce a high technical barrier for those with little to no experience in 3D virtual environments. In this work, 2. Related Work we design an innovative adapting assisting interface to help the user In this section, we provide an overview of research on the scaffold- learn to operate a virtual film set. Our primary goal is to assist the ing theory and the camera control methods for 3D environments users in familiarizing themselves with the virtual set without the that concern the design and operation of the virtual film set. need for in-person mentoring. A secondary goal is to offer non- expert users the opportunities to make a head start on cinematogra- phy. 2.1. Intelligent tutoring systems and virtual training To this end, we refer to the scaffolding theory, introduced in the For many years, virtual reality (VR) has been expected to bring late 1950s by Jerome Bruner. The term “scaffolding” represents the on the next evolution in learning technologies, introducing sev- © 2020 The Author(s) Eurographics Proceedings © 2020 The Eurographics Association. Qiu-Jie Wu, Chih-Hsuan Kuo, Hui-Yin Wu, Tsai-Yen Li / Designing an Adaptive Assisting Interface for Learning Virtual Filmmaking eral advantages, such as low cost, high comfort, and unlimited distance [PIB∗16, RBB∗00, You98]. In a recent work on adaptive technologies in virtual reality training [VGD16], data from trainee assessment was used as an input to an autonomous system for cus- tomized training and automated difficulty level adjustment to meet individual needs. 2.2. Scaffolding Theory Scaffolding theory was first introduced in the late 1950s by Jerome Bruner, a cognitive psychologist. Scaffolding represents the help- ful interactions that enable students to do something beyond his or her independent efforts [MS08]. Instructional scaffolding is the support given to a student by an instructor throughout the learn- ing process. This support is specifically tailored to each student such that the ”scaffolds” are incrementally removed when the stu- dents acquire the knowledge or skills under the learning target. This instructional approach allows students to experience student- centered learning, which tends to facilitate more efficient learning than teacher-centered learning. In the learning process, the educators may question a student’s Figure 1: The architecture of our virtual film set is composed approach to a difficult problem and provide constructive feedback. of three main parts: (1) a navigation component to move around The type and amount of support needed are dependent on the needs cameras and the monitor, (2) shot management components (right of the students during the time of instruction [VL14]. Scaffolds are of the figure) including a database, recommendation, and editing developed to assist students with a difficult task [SB02]. The key is functions, (3) the workspace allowing users to take on various that the assistance needs to be planned. filmmkaing roles, and export the output as a video. Scaffolding is also used to support problem-based learning (PBL), an instructional approach in which students are presented with an ill-structured problem [Ali19]. After being presented with constraints. Christie et al. [CO09] have summarized comprehen- the problem, defining it, and generating learning issues, students sive solution techniques from different approaches. While photog- proceed to address their learning issues, and then develop a poten- raphy and cinematography practices have been documented for the tial solution, backed with evidence. There is some evidence that operation of real-world cameras, the cinematographers and pho- PBL can be effective when students work individually, leading to tographers have little to no control over the staging of day-to-day strong learning outcomes. Many educators incorporate PBL in their content. Whereas in 3D environments, cameras can be intuitively classrooms to engage students and help them become better prob- placed at given coordinates and rotated. Solutions to generate cam- lem solvers. However, educators must first identify the content that era paths given constraints have also been proposed [GCLR15]. needs scaffolding support, the appropriate time to implement the support, and determine when the scaffold can be removed [Laj05]. For virtual cinematography [HCS96], Jhala and Young created the Darshak system that generates shot sequences to fulfill goals of The design of our interface was also inspired by game-based showing specific actions and events in a 3D environment [JY11]. learning, which can help learners acquire real-world knowledge O’Neill et al. [ORN09] proposed a system of blockings that would through a game-like environment. A study by Barzilai and Blau choreograph the relative positions of actors and cameras in an en- [BB14] suggests that presenting scaffolds may have “problema- vironment. In these systems, context – the placement of characters tized” learners’ understandings of the game by connecting them and cameras in accordance with the story and the environment – to disciplinary knowledge. Implications for the design of scaffolds is crucial, making camera control in 3D environments a complex for game-based learning are discussed. Zook et al. [DZO∗13] also problem. demonstrate how a rule-based intelligent system can reduce the fre- quency of errors that novices make by providing information about The adoption of cinematic idioms for virtual camera control was first demonstrated through rule- and constraint-based systems rule violations without prescribing solutions, and discuss the role ∗ of error reduction in creativity support tools. [CAH 96, BGL98], and through the representation of state tran- sitions between various shot types [HCS96]. Ronfard et al. then proposed the Prose Storyboard language (PSL) [RGB13] as a for- mal representation for describing shots visually. PSL illustrates 2.3.