The Texas Tech Community Has Made This Publication Openly Available
Total Page:16
File Type:pdf, Size:1020Kb
BLOCKLYXR: AN INTERACTIVE EXTENDED REALITY TOOLKIT FOR DIGITAL STORYTELLING The Texas Tech community has made this publication openly available. Please share how this access benefits you. Your story matters to us. Citation Jung, K., Nguyen, V.T., & Lee, J. (2021). BlocklyXR: An interactive extended reality toolkit for digital storytelling. Applied Sciences, 11(3), 1073. https://doi.org/10.3390/app11031073 Citable Link https://hdl.handle.net/2346/86889 Terms of Use CC BY 4.0 Title page template design credit to Harvard DASH. applied sciences Article BlocklyXR: An Interactive Extended Reality Toolkit for Digital Storytelling Kwanghee Jung 1 , Vinh T. Nguyen 2,* and Jaehoon Lee 1 1 Department of Educational Psychology and Leadership, Texas Tech University, Lubbock, TX 79409, USA; [email protected] (K.J.); [email protected] (J.L.) 2 Department of Information Technology, University of Information and Communication Technology, Thai Nguyen 24000, Vietnam * Correspondence: [email protected] or [email protected] Abstract: Traditional in-app virtual reality (VR)/augmented reality (AR) applications pose a chal- lenge of reaching users due to their dependency on operating systems (Android, iOS). Besides, it is difficult for general users to create their own VR/AR applications and foster their creative ideas without advanced programming skills. This paper addresses these issues by proposing an interac- tive extended reality toolkit, named BlocklyXR. The objective of this research is to provide general users with a visual programming environment to build an extended reality application for digital storytelling. The contextual design was generated from real-world map data retrieved from Mapbox GL. ThreeJS was used for setting up, rendering 3D environments, and controlling animations. A block-based programming approach was adapted to let users design their own story. The capability of BlocklyXR was illustrated with a use case where users were able to replicate the existing Palmi- toAR utilizing the block-based authoring toolkit with fewer efforts in programming. The technology acceptance model was used to evaluate the adoption and use of the interactive extended reality toolkit. The findings showed that visual design and task technology fit had significantly positive effects on user motivation factors (perceived ease of use and perceived usefulness). In turn, perceived usefulness had statistically significant and positive effects on intention to use, while there was no significant impact of perceived ease of use on intention to use. Study implications and future research Citation: Jung, K.; Nguyen, V.T.; Lee, directions are discussed. J. BlocklyXR: An Interactive Extended Reality Toolkit for Digital Storytelling. Keywords: blockly; digital storytelling; extended reality; GIS, remote sensing; visual programming; Appl. Sci. 2021, 11, 1073. https:// computational thinking; technology acceptance model doi.org/10.3390/app11031073 Academic Editor: Chang-Hun Kim Received: 11 January 2021 1. Introduction Accepted: 22 January 2021 Published: 25 January 2021 Virtual reality (VR) and augmented reality (AR) are emerging technologies that have attracted researchers from a wide variety of fields in recent years [1–3]. VR enables Publisher’s Note: MDPI stays neu- users to immerse themselves in a virtual environment that is isolated from the physical tral with regard to jurisdictional clai- world [4], while AR supplements the real world with augmented information [5]. They have ms in published maps and institutio- distinctive roles in helping users carry out certain tasks more efficiently. VR is often used nal affiliations. for simulation and training when the actual content/scenario in the real world is expensive to design (e.g., visiting the inside of a volcano [6], practicing surgery and simulations [7], examining a new house design [8], engaging in disaster response training [9]) or impossible to recreate (e.g., ancient artifacts [10] or a military battle [11]). On the other hand, AR is more Copyright: © 2021 by the authors. Li- censee MDPI, Basel, Switzerland. focused on providing contextual information (e.g., annotations [12]), presentations [13], This article is an open access article mapping virtual objects with the physical world [14] (i.e., automatically situating an object distributed under the terms and con- on the targeted floor). ditions of the Creative Commons At- Integrated development environments (IDEs) and software are typically used to create tribution (CC BY) license (https:// VR/AR applications; these include Android Studio, XCode, and Unity3D [15]. Android creativecommons.org/licenses/by/ Studio is dedicated to producing applications for Android devices, while XCode is used for 4.0/). Apple products. Unity3D can be considered as a general IDE that supports the proposed Appl. Sci. 2021, 11, 1073. https://doi.org/10.3390/app11031073 https://www.mdpi.com/journal/applsci Appl. Sci. 2021, 11, 1073 2 of 19 applications for both Android and iOS. These tools have many rich capabilities (or APIs) to utilize functionalities of the targeted operating system, such as camera, accelerator, motion, and device orientation, which provides huge opportunities in developing an AR/VR application. However, using these tools requires a high level of expertise from developers [15–17]. Part of the problem is due to the unstable nature of the operating system and the incompatible API version [18], meaning that the application will crash, stop working, or show black screen if the developed application is not built for the target device and the operating system version. As such, developers should pay attention to the features of the operating system for the device of interest. This kind of dependency on a specific target device poses big challenges for users who want to create VR/AR applications, especially when the applications are not compatible with their devices. To alleviate the aforementioned issues, researchers have brought VR/AR to the web with programming tools such as ARtoolkit [19], ThreeJS [20], A-Frame [21], or Baby- lonJS [22]. These enable users to experience AR/VR regardless of the operating systems they are using. Web-based VR/AR have rapidly emerged over the years and their capabili- ties and applicability have been demonstrated in many studies [18,23–26]. The studies have focused on providing a web environment for VR/AR programming, as well as enhancing the usability with high fidelity and speeding up the rendering performance. For exam- ple, the Khronos Group aimed to speed up the web’s processing time by defining a new lightweight 3D model scheme (i.e., GLTF [27]) that is intended for the web environment. ThreeJS provides a library to set up and manipulate the virtual environment and A-Frame wraps up ThreeJS into components for users to shorten programming time in VR/AR creation. However, interactions in web-based VR/AR are still limited, which is partly due to the lack of input controls and accessibility to mobile device sensors from the browser. To handle this issue, web extended reality (or WebXR) has been introduced as a new mech- anism that enables developers to gain access to mobile sensors, thus providing users with a huge opportunity to have the same experience as in-app applications. Although WebXR is still under development and experimental progress, developers can incorporate its testing features with the existing methods to develop an interactive extended reality application. Unlike in-app VR/AR development, in which the IDE provides rich APIs available for use, creating a web-based VR/AR application is a challenging task for developers, espe- cially for general users who have no formal training in computer science and computing- related fields. As such, there is a need to introduce a visual programming platform that can help general users (enthusiasts, instructors, artists, or kids) to develop a VR/AR applica- tion without advanced programming knowledge. To address this issue, some frameworks and tools have been provided (e.g., AR Scratch [28], CoSpace [29], and VEDILS [30]) that shift VR/AR creators from advanced programming skills to logical and computational thinking skills by utilizing visual programming tools. Using these tools, general users can create their own VR/AR applications by “dragging and dropping” the visual cues to the appropriate place to form a logical flow for the program. These tools are gaining in popularity, especially in the education sector, to inspire young learners in science, technol- ogy, engineering, and mathematics (STEM). However, these tools still suffer from creating dynamic spatial situated contents or imposing animations of 3D models in a given context. That is, a series of 3D models’ actions or animations are often defined in another software (e.g., Blender), which causes difficulties for users to use and manipulate/customize the animations. As such, it is warranted to find a way to manipulate/customize animations more efficiently in the browser to convey users’ ideas and stories. This paper introduces a novel web-based visual programming interface for creating an extended reality application, which enables general users to create a digital storytelling application with a block-based programming toolkit. 2. Motivation and Research Aim Motivated by the new presence of WebXR and inspired by the existing work of Scratch and CoSpace, our aim with this research is to close the gap by providing developers, young Appl. Sci. 2021, 11, 1073 3 of 19 learners, and enthusiasts with an interactive web environment