NIGHTMARE RUNNER: A PERSONALIZED AVATAR AND PHYSICAL ACTIVITY GAME A Thesis Presented to the Faculty of the Department of Computer Science University of Houston In Partial Fulfillment of the Requirements for the Degree Master of Science By Madhur Thangadurai Rajendran May 2016 NIGHTMARE RUNNER: A PERSONALIZED AVATAR AND PHYSICAL ACTIVITY GAME Madhur Thangadurai Rajendran APPROVED: Dr. Zhigang Deng, Chairman Dept. of Computer Science Dr. Weidong Shi Dept. of Computer Science Dr. Daniel O'Connor Dept. of Health and Human Performance Dean, College of Natural Sciences and Mathematics ii Acknowledgements I would like express my thanks to my parents, Mr. Rajendran Thangadurai & Mrs. Bagya Latha Rajendran, and my sister, Ms. Mahitha Rajendran for motivating me throughout the entire process. I am truly grateful for their unwavering support and patience. I would like to thank my twin, Mayur, for his contribution and support. With- out him I would be incomplete. I would also like to thank my brothers, Aditya, Vishwanath, and Shivaram, for always being there for me. I would also like to extend my heartfelt gratitude to Dr Zhigang Deng, who served as my advisor & committee chair and guided me throughout the project. I am also grateful to Dr. Weidong Shi, and Dr. Daniel O'Connor for being on my committee, and for their valuable insights and suggestions for this project. I am grateful to the lecturers and professors of the Department of Computer Science, especially Dr. Victoria Hilford, who enlightened me about the software development process. I would also like to thank my friends and colleagues at the University of Houston for their encouragement and support which made my studies here a truly enjoyable experience. \Be well, do good work, and keep in touch." iii NIGHTMARE RUNNER: A PERSONALIZED AVATAR AND PHYSICAL ACTIVITY GAME An Abstract of a Thesis Presented to the Faculty of the Department of Computer Science University of Houston In Partial Fulfillment of the Requirements for the Degree Master of Science By Madhur Thangadurai Rajendran May 2016 iv Abstract In this thesis, a novel method to calculate energy expenditure of subjects using the Microsoft Kinect is presented. There are two main types of methods used in calorime- try, direct and indirect. Direct calorimetry involves the measurement of actual heat transfer of the subject, which is infeasible in most situations. Indirect calorimetry, however, involves measuring secondary variables, like oxygen consumption, and using those measurements to calculate energy expenditure. The method presented is an indirect calorimetry method that uses basic physics laws of work and energy to cal- culate the energy expenditure of a body by continuously tracking the body through the Microsoft Kinect. A process of 3D scanning a person in order to use them as avatars in a game was developed and the feasibility and benefits of using such personalized avatars in game was tested. Lastly, a physical activity game that serves as a wrapper for both the proposed energy expenditure method, and the personalized avatar was developed. v Contents 1 Introduction 1 1.1 Problem Statement . 1 1.2 Contribution . 2 1.3 Outline . 3 2 Background 4 2.1 Calorimetry . 4 2.1.1 Direct Calorimetry . 4 2.1.2 Indirect Calorimetry . 5 2.2 3D Scanning . 6 2.3 Mesh Processing . 7 2.4 Exercise Game . 7 2.4.1 Unity Game Engine . 8 3 Energy Expenditure Algorithm 10 3.1 Assumptions . 13 4 Scanning & Mesh Processing 14 5 Exercise Game 19 5.1 Development . 21 vi 5.1.1 Hardware Requirements . 21 5.1.2 Software Requirements . 22 5.1.3 Algorithm Integration . 23 5.2 Game . 24 5.2.1 Personalized Avatar . 25 5.2.2 Environment . 25 5.2.3 Story . 27 5.2.4 Cutscenes . 28 5.2.5 Goals . 33 5.2.6 Game Menus . 34 5.3 Gameplay . 38 5.3.1 Dream Energy . 38 5.3.2 Pick Ups . 39 5.3.3 Controls . 39 6 Results 43 6.1 Energy Expenditure Algorithm . 43 6.1.1 Preliminary Validation . 44 6.1.2 Subject Testing . 47 6.2 Game . 53 7 Conclusion 55 7.1 Future Work . 56 7.1.1 Algorithm . 56 7.1.2 Exercise Game . 57 7.2 Contribution . 57 Appendices 58 vii A Calorific Data of Child Testing 59 A.1 Average Calories Burned Per Minute . 59 A.2 Calories Burned per Minute . 61 B Calorific Data of Adult Testing 103 B.1 Calories Burned Per Interval . 103 B.2 Cumulative Calories Burned . 109 Bibliography 115 viii List of Figures 2.1 Occipital's Structure Sensor . 6 2.2 Exercise Game: Nightmare Runner Menu . 8 2.3 Unity Game Engine . 9 3.1 A sample 3D model used for Algorithm . 11 4.1 An image of the structure sensor . 14 4.2 A subject being scanned by the structure sensor and ItSeez3D app . 15 4.3 A sample model after rigging and animating . 16 4.4 A sample subject that has been split in Maya 2016 . 18 5.1 An Endless Runner Game: Temple Run 2 [3] . 20 5.2 The Microsoft Kinect v2 . 22 5.3 The Mixamo Website used for Rigging . 23 5.4 The Floating Island Environment . 26 5.5 The monster in the game . 27 5.6 The game story screenshot . 28 5.7 Introduction: The player's avatar going to sleep . 29 5.8 Introduction: The player's avatar exploring the dream . 30 5.9 Introduction: The player's avatar discovering the cave . 30 5.10 Victory: The player running away from the Nightmare Monster . 31 ix 5.11 Victory: The player facing their Nightmare . 32 5.12 Victory: The player defeating the Nightmare Monster . 32 5.13 Victory: The player escaping the nightmare world . 33 5.14 The final screen when the player beats the game . 34 5.15 The score menu . 35 5.16 The Difficulty Select screen . 36 5.17 The Game Over screen . 37 5.18 The Dream Energy Bar . 38 5.19 The Pick Ups used in the game . 39 5.20 Gameplay: Run . 40 5.21 Gameplay: Limp . 41 5.22 Gameplay: Slide . 41 5.23 Gameplay: Jump . 42 6.1 Subject 1: Calories burned per interval . 45 6.2 Subject 2: Calories burned per interval . 45 6.3 Subject 2: Calories burned over time . 46 6.4 Subject 2: Calories burned over time . 47 6.5 Average calories burned per minute by subject . 48 6.6 Subject 1: Comparison with Just Dance 4 . 51 6.7 Subject 2: Comparison with Just Dance 4 . 53 x List of Tables 6.1 Subject 1 Energy Expenditure Estimation Comparison . 50 6.2 Subject 2 Energy Expenditure Estimation Comparison . 52 xi Chapter 1 Introduction 1.1 Problem Statement The standard procedure for calculating and estimating the energy expenditure of the human body is to use VO2 machines while the subject is performing the activity. However, this is bulky, requiring the subject to wear a mask that is connected to the machine, in order to be able to measure oxygen consumption. Other sensors that use accelerometers to measure activity were also used, but they're fixed at one location on the body. While they do give a generalized estimate of the energy expenditure, the VO2 method is generally regarded as the most accurate method for calculating energy expenditure. Therefore the major drawbacks to the current methods are that they either require bulky external devices, or they aren't as accurate due to mostly being local to a specific area of the body. 1 With current available technology, 3D graphics have become very popular. The focus of graphics technology has been the ability to create, manipulate, and animate high definition 3D models, as can be seen by recent movies and games. Many 3D scanners have been developed, allowing users to capture objects, or even people around them as a 3D file that can be printed out using a 3D printer. These models can even be skinned and rigged with a skeleton, allowing animation, and put into games. Using a person's personalized model in a game has improved playability by improving immersion and enjoyment [7]. 1.2 Contribution In this thesis, three main contributions were presented. First, a new method of calculating the energy expenditure of a human body using the Microsoft Kinect was presented. Secondly, a workflow was created using existing software and certain hardware systems to scan a human subject, and to create a 3D mesh that modeled the subject. This pipeline significantly decreased the time required for scanning, mesh creation, mesh processing, and game avatar creation from the scanned model. Lastly, an exercise game was created that uses the Microsoft Kinect as an input controller. The avatar created from the scanned model was used as the main game character, so subjects could take control of an avatar that not only represented them, but was also visually similar to them. We kept track of the energy expenditure of the subject using this method while the subject was playing the game. 2 1.3 Outline The thesis is organized according to the following outline. In Chapter 2, some back- ground of the systems and domains used in the thesis is presented. The concepts and techniques used in the later chapters are described. In Chapter 3, the new Energy Expenditure Algorithm. The hardware required is discussed, and the background behind the development of the algorithm is explained. In Chapter 4, the hardware and software requirements of the 3D scanning and Mesh Processing method are de- scribed. In Chapter 5, the exercise game that was created to serve as a wrapper for the algorithm, the scanned mesh and avatar is described.