Digital Dojo: A Serious Gaming Approach for Supplementing Martial Arts Education Using Kinect Motion Capture Analysis A Thesis Submitted to the Faculty of Drexel University by Michael John Dixon in partial fulfillment of the requirements for the degree of Master of Science in Digital Media June 2012 ii © Copyright 2012 Michael J. Dixon. All Rights Reserved. iii Dedications This thesis work is dedicated to my mother and father, Sue and Mark Dixon, who always taught me to finish what I start. iv Acknowledgements I extend a great deal of gratitude and credit to my thesis advisor, Jervis Thompson. His continuous advice kept this thesis work precise and focused. Through his council, I was able to scope my time and workload effectively and reach goals consistently. I would also like to express my thanks to my committee members, Troy Finamore and Dr. Will Dampier. Their suggestions, critiques, and demands pushed me to produce a high quality project that surpassed my own expectations. My fellow students, Glenn Winters and Matt Haas, have been incredibly supportive in developing a reliable motion capture workflow at Drexel and deserve my sincere thanks. Without their continued interest and eagerness to experiment, I would surely not have been able to realize my vision to its fullest potential. I thank Dr. Will Dampier once again, for allowing me to utilize his talent and record his motions for the Digital Dojo application. I give my dearest thanks to Christine Adams for her love, support, and patience during the past year of intense production and late hours. Finally, an enormous debt of gratitude and respect goes to my Kempo Karate instructor of nearly 10 years, Frank Manzelli. Frank pushed me mentally and physically to become the best athlete, friend, brother, son, and person I could be. He was instrumental in instilling me with a tremendous work ethic, self-discipline, and an indomitable spirit. He taught me about the importance of teaching others, and is ultimately the inspiration for this thesis work. v Table of Contents LIST OF FIGURES ............................................................................................................................. vii ABSTRACT ...................................................................................................................................... viii I. INTRODUCTION & THESIS OVERVIEW ........................................................................................... 1 1. Definition of Terms ...................................................................................................................... 4 II. REVIEW OF RELEVANT RESEARCH & THEORY .............................................................................. 7 1. History of Judo Education ............................................................................................................ 7 2. Education Theory ......................................................................................................................... 7 3. Case Studies: E-Learning used for Physical Tasks ........................................................................ 9 3.1 E-Learning Martial Arts .............................................................................................................. 9 3.2 Interactive Video Mirrors ......................................................................................................... 10 3.3 Training for Physical Tasks in Virtual Environments ................................................................ 12 3.4 A Serious Computer Game to Assist Tai Chi Training for the Elderly ....................................... 14 4. Case Studies: Mainstream Kinect Martial Arts Games .............................................................. 14 4.1 Ubisoft’s Self Defense Training Camp ...................................................................................... 14 4.2 UFC Personal Trainer ............................................................................................................... 17 III. APPROACH ................................................................................................................................ 18 1. Research: Selecting the ideal Kinect platform ........................................................................... 19 1.1 Research Overview .................................................................................................................. 19 1.2 Brekel Kinect ............................................................................................................................ 19 1.3 Processing ................................................................................................................................ 20 1.4 Microsoft SDK .......................................................................................................................... 21 1.5 Unity 3D with Zigfu wrapper .................................................................................................... 21 vi 2. Development of an intuitive workflow for capturing and importing usable, 3D motion data into Unity 3D. ................................................................................................................................. 22 2.1 Motion Capture Pipeline Overview.......................................................................................... 22 2.2 Capturing the Live Motion Performance ................................................................................. 22 2.3 Cleaning and Stitching the Data ............................................................................................... 24 2.4 Extracting the Rotational Values of Primary Joints .................................................................. 27 2.5 Importing the Data into Unity .................................................................................................. 28 3. Build an external program, capable of motion analysis and feedback ...................................... 28 3.1 External Program Feature Overview ........................................................................................ 28 3.2 Body Trackers ........................................................................................................................... 29 3.3 Displaying and Navigating the Tutorial Menus ........................................................................ 30 3.4 Creating and Navigating the Main Menu ................................................................................. 32 3.5 The In-Game Navigation Menu ................................................................................................ 32 3.6 Navigating the 3D Scene in Unity 3D ....................................................................................... 33 3.7 Real-Time Feedback ................................................................................................................. 34 3.8 Level Progression ..................................................................................................................... 34 3.9 Special Consideration for Hands-Free Operation .................................................................... 34 4. Design the lessons ...................................................................................................................... 36 4.1 Overview .................................................................................................................................. 36 4.2 The Avatar ................................................................................................................................ 36 4.3 Getting Input from the Kinect into Unity 3D ........................................................................... 37 4.4 The Belt Menu .......................................................................................................................... 38 4.5 Demonstration Phase .............................................................................................................. 39 4.6 Replication Phase ..................................................................................................................... 40 4.7 Assessment Phase .................................................................................................................... 40 IV. CONCLUSION ............................................................................................................................ 42 vii IV. RECOMMENDATIONS & FUTURE RESEARCH ............................................................................ 44 LIST OF REFERENCES ...................................................................................................................... 45 LIST OF FIGURES Figure 1. Hamalainen’s setup for capturing and analyzing video for martial arts training. .......... 11 Figure 2. The user must wear a head mounted display and motion capture suit. ........................ 12 Figure 3. Different configurations for representing the student and the instructor in a virtual space. ............................................................................................................................................. 13 Figure 4. ‘Swipe and punch’ based navigation menu. [Screen shot] ............................................. 15 Figure 5. The default player avatar is a female, pulling male participates out of the game world. ......................................................................................................................................................