Poster: Studying the Role of Kinect as a Multi-Sensory Learning Platform for Children Harsh Sinha Abstract Birla Institute of Technology and According to the theory of Embodied Cognition, our be- Science, Pilani havior is a result of real-time interaction with surroundings, Pilani, India our cognitive skills, and the nervous system. From this per- [email protected] spective, researchers are considering a learning environ- ment which promotes physical activities to achieve cogni- Shivin Srivastava tive tasks. Such Natural User Interfaces (NUI) make use Birla Institute of Technology and of gesture-based sensors like the Microsoft Kinect. Yet we Science, Pilani lack in-depth studies of how they improve the learning pro- Pilani, India cess. In this paper, we present observations of two deploy- [email protected] ment studies which focus on different roles that NUI can play as a part of learning activities. We deploy the Kinect Yash Sinha based applications:- Yoga Soft: A Digital Yoga Instructor Birla Institute of Technology and and Mudra: A Kinect based Learning System in real life Science, Pilani scenarios. The first study is conducted at residences of pre- Pilani, India adolescent children in Gurgaon, India. The second study is [email protected] conducted at an education center specializing in the care of kindergarten children in Pilani, India. Author Keywords Permission to make digital or hard copies of part or all of this work for personal or Education; Interactive Applications; Natural User Interface classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. ACM Classification Keywords For all other uses, contact the owner/author(s). K.3.1 [Computers and Education]: Computer Uses in Ed- Copyright held by the owner/author(s). UbiComp/ISWC’18 Adjunct, October 8–12, 2018, Singapore, Singapore ucation - Computer Assisted Instruction; K.8.0 [Personal ACM 978-1-4503-5966-5/18/10. Computing]: General - Games https://doi.org/10.1145/3267305.3267574 251 Introduction development. We make use of Microsoft RxPY to handle Today, the society is driven by innovations in technology Kinect’s data as event streams. Further, custom APIs as and with rapid developments in technology, the approach to web apps are deployed on Microsoft Azure to monitor the teaching is constantly changing. Everyday digital comput- development of children. ing is transforming education by integrating smart-boards and tablet technology in classrooms [5]. There is no doubt To observe and study the impact of NUI with Kinect on that in today’s digitally enhanced society, there is a need learning, and cognitive skills, the proposed platform is an- for children to get equipped with technology from the very alyzed across different environments and context. The fol- beginning. lowing section discusses the different contextual studies undertaken to study gesture-based interactions. The adoption of new technologies, especially the Microsoft Kinect has received an enthusiastic reception across schools. The Digital Yoga Instructor: Yoga Soft The Microsoft Kinect sensor is a high resolution (low cost) Yoga Soft is meant to guide a practitioner through all the depth camera, which is able to track hand and body ges- steps of an asana (yoga posture). The system has a database Figure 1: A participant learn- tures. This type of Natural User Interaction (NUI) is ap- of many asanas or postures. It is explicitly programmed to ing an asana by following propriate for children as it promotes improvement in motor recognize a particular asana by tracking the motion and audio-visual instructions planning, execution skills and hand-eye However, most of orientation of the torso, legs, and arms. It guides the user the existing applications for Kinect don’t focus on the goal towards the particular posture in multiple stages. The dif- of enhancing learning in children. Although the integration ferent postures are encoded in the application in terms of of NUI technologies have already begun, questions regard- the relative positioning of different body parts. Yoga Soft ing the suitability of software systems that claim to meet the provides real-time feedback (Figure 2) in the form of voice needs of children and teachers are still unexplored. instructions so that the person can focus on his asana in- stead of looking at the screen for subsequent instructions. In this work, the role of NUI as an illustration to Microsoft Kinect is investigated. We study its impact on children’s Empowering children with Kinect learning games: creativity, motivation to learn and easy of usability. Mudra Children’s learning ability is hindered if they are in anxiety. Using Kinect as a Learning Platform The amygdala which processes emotions [1] takes a de- The KinectV2 sensor can track various joints and their dis- fensive stand by obstructing information flow. Therefore it tance from the horizontal plane [4]. The joint locations can is very important to make sure that children are motivated be used to track the hands, legs, torso and angles between and enthusiastic to learn. Traditionally, it is supposed that Figure 2: The Application In- them. physical activities can provide the break to rejuvenate the terface shows the user’s spark to study. Mudra is developed to provide children a The proposed applications are developed using Kinectv2 real-time posture (left) and fun-based learning platform by coupling physical activities sensor for Microsoft Windows 8 and above. The hardware training video (right). and education, a methodology known as head-fake learning API is accessed using PyKinect2. The applications are based on Pygame engine which allows real-time game 252 [6]. We develop touch-less games which require human- According to the observation and findings in both the con- computer interaction mediated by Kinect. textual studies, Kinect was perceived as an enjoyable expe- rience by most of the participants. In general, children didn’t The application is based on head-fake learning [6] which find any difficulty in interacting with the application. connects learning with fun. We employ it to help children to improve their motor planning, concentration, and memory We consider study participants belonging to age groups 5-8 skills [3]. Educational games focusing on arithmetic and lin- for several reasons. Firstly, at this age children have much guistics are developed using adaptive learning to monitor better learning capabilities [2]. Secondly, if a child can learn a child’s development and accordingly adjust the difficulty by following voice commands of the digital assistant, then it level. Moreover, a score is provided to give instant feed- is indicative that the system is competent in communicating back. with the user in general. The platform included 3 games for the study. 1) Tell the Context 1: Yoga Soft Time (a participant matches the displayed time using his We explore Yoga Soft with pre-adolescent children in a res- hands as that of an analog clock, Figure 3), Aftermath idential setting to simulate a controlled environment. All (given an incomplete equation, the user need to fill the ap- participants were requested to practice all the three dif- propriate operator (+; −) using his arms to make the sign) ferent yoga asanas:- Tree Pose, Warrior Pose and Trian- and Catch the vowel (the user learns to differentiate be- gle Pose. Yoga Soft supports two languages:- English and tween vowels and consonants by catching vowels, Figure Hindi (mother tongue for children). 4). Context 2: Mudra Contextual Studies: Result and Discussions The research on Mudra was conducted with kindergarten To prepare for the study visits, we requested some children children. Children were instructed to match the time us- (after taking permission from their parents) in a residential ing their hands as that in an analog clock (Figure 3). We society in Gurgaon, India to volunteer to learn 3 different also tested their ability to differentiate between vowels and Yoga asanas (Tree Pose, Warrior Pose and Triangle Pose) consonants (Figure 4). Some of the relatively elder chil- with Yoga Soft. Similarly, the study visit for Mudra was con- dren were given basic arithmetic problems. All the games ducted at Pilani, India in education centers specialized in involved gestures. Teachers were allowed to choose appro- the care of kindergarten children. priate games for different children. Throughout each study visit, study data are collected in the Results Figure 3: A participant form of observational notes, video recordings, and pho- We find a number of similarities and differences between (above) using his hands to tographs of learning activities. The application’s API on the interaction experience with Yoga Soft and that of Mu- match the time shown on his Microsoft Azure automatically saved quantitative data re- dra. In general, children were excited to see the new format screen (below). garding the performance of students. In addition, we inter- of learning. The voice recognition and feedback system of viewed the teacher or guardian about their perspective and Yoga Soft especially amused the participants. With regard experiences with the application. to games such as Aftermath, it was observed that children’s 253 performance was enhanced and the general fear against skills, motivation and free-play. But at present Kinect is not math problems was reduced. Similarly, with ’Catch the suitable for providing guidance for physical recreational ac- vowel’, children improved their hand eye coordination and tivities like Yoga. The future work may address the design could successfully recognize vowels. Use of natural interac- of Natural User Interfaces (NUI) so that it can serve as an tions and gestures helped the children direct their impulsive valuable assisted tool to children as well as teachers.