A VERSATILE HUMAN MACHINE INTERFACE FOR GESTURE RECOGNITION WITH APPLICATIONS IN VIRTUAL REALITY A Thesis submitted to the faculty of San Francisco State University In partial fulfillment of the requirements for the Degree - Master of Science In Engineering: Embedded Electrical and Computer Systems by Kartik Bholla San Francisco, California May 2018 Copyright by Kartik Bholla 2018 CERTIFICATION OF APPROVAL I certify that I have read A Versatile Human Machine Interface for Gesture Recognition with Applications in Virtual Reality by Kartik Bholla, and that in my opinion this work meets the criteria for approving a thesis submitted in partial fulfillment of the requirement for the degree Master of Science in Engineering: Embedded Electrical and Computer Systems at San Francisco State University. Xiaorong Zhang, Ph.D. Assistant Professor Hao Jiang, Ph.D. Associate Professor A VERSATILE HUMAN MACHINE INTERFACE FOR GESTURE RECOGNITION WITH APPLICATIONS IN VIRTUAL REALITY Kartik Bholla San Francisco, California 2018 In recent years, the technological advancements had led to the development of ultra-high definition immersive video technology and virtual reality (VR) systems from companies like Microsoft, Nintendo, Facebook etc. VR has applications in many fields such as gaming, surgery training, robotics, and rehabilitation. To enable intuitive control of VR systems, this research aimed to evaluate a user-oriented and versatile gesture control interface for VR systems. Specifically, a first-person shooter (FPS) VR game was developed using Unity 3D and interfaced with an electromyogram (EMG)-based gesture control interface developed by the SFSU Intelligent Computing and Embedded Systems Laboratory (ICE Lab). A comparable study has been done on able-bodied human subjects to compare the gesture-based control scheme and the keyboard and mouse-based control scheme. I certify that the Abstract is a correct representation of the content of this thesis. Date ACKNOWLEDGEMENTS I would first like to thank my thesis advisor Dr. Xiaorong Zhang. The door to Dr. Zhang’s office was always open whenever I ran into a trouble spot or had a question about my research or writing. She consistently allowed this paper to be my own work but steered me in the right the direction whenever she thought I needed it. Besides my advisor, I would like to thank my friend and colleague Ian Donovan for his wonderful collaboration. You supported me greatly and were always willing to help me. Finally, I must express my very profound gratitude to my parents for providing me with unfailing support and continuous encouragement throughout my years of study and supporting me spiritually throughout my life. This accomplishment would not have been possible without them. Thank you. v TABLE OF CONTENTS List of Table............................................................................................................................... viii List of Figures...............................................................................................................................ix List of Appendices........................................................................................................................ x 1. Introduction................................................................................................................................ 1 1.1 Current Approach of EMG Gesture Recognition................................................. 2 1.2 ICE HMI: Introduction............................................................................................. 3 2. Virtual Reality: Introduction...................................................................................................7 2.1 Background of Virtual Reality................................................................................. 8 2.2 Current State of Virtual Reality............................................................................. 10 2.3 VR Branching Out of Gaming............................................................................... 13 2.4 Uses for Virtual-Reality Tech ............................................................................... 15 2.5 Virtual Reality Game Accessories ......................................................................24 3. Virtual Reality Zombie Game............................... 29 3.1 Human Machine Interface for VR Gam e............................................................ 32 4. Experiment & R esults........................................................................................................... 37 4.1 Experiment Protocol .............................................................................................. 37 4.2 Evaluation & Results ............................................................................................. 39 4.3 Results Summary.....................................................................................................43 4.4 Drawbacks of Current Approach........................................................................ 44 4.4.1 Drawbacks of Virtual Reality G am e....................................................44 4.4.2 Drawbacks of Custom Interface Pipe ................................................. 45 5. Features Added to Interface...................................................................................................46 5.1 Implementing Third Byte in the HMI...................................................................46 5.2 Designing an HMI Simulator................................................................................. 48 6. Conclusion & Discussion.......................................................................................................51 6.1 Results Concluded...................................................................................................51 6.2 Future Work............................................................................................................. 52 Reference..................................................................................................................................... 54 Appendices.................. 56 vii LIST OF TABLES Table Page 1. Average Age of VR Tests Subjects...................................................................... 37 2. Average Experience in FPS Games of VR Tests Subjects................................38 3. Average Experience with VR of VR Tests Subjects..........................................39 4. Average Results with Standard Deviation Results of VR Tests.......................40 5. Post Survey Results for Ease of play with different control schema................41 6. Post Survey Results for feel of game with different control schema................42 7. Post Survey Results depicting engagement with different control schema......42 8. Post Survey Results depicting associated level of fatigue with different control schema............................................................................................................43 LIST OF FIGURES Figures Page 1. MYO Armband........................................................................................................... 4 2. HMI Structure..............................................................................................................5 3. ICE HMI Console....................................................................................................... 6 4. Nintendo Virtual Boy..................................................................................................8 5. Nintendo Virtual Boy 3D Graphics..........................................................................9 6. Oculus Rift CV1......................................................................................................... 11 7. HTCVIVE.................................................................................................................. 12 8. VR in Military Training...........................................................................................17 9. VR Training for Dental Students............................................................................19 10. VR CAD Simulation for Automotive Industry.................................................... 20 11. VR Travel Experience for the Disabled................................................................ 21 12. Physical Therapy Through VR................................................................................ 22 13. VR helping to fight Phobias in a safe controlled environment............................23 14. PRIO VR.....................................................................................................................25 15. Cyberith Virtualizer....................................................................................................26 16. Dexmo Exoskeleton Gove.........................................................................................27 17. VR Zombie FPS Gamepay......................................................................................30 18. VR Zombie Game Control Schema....................................................................... 32 19. Duplex NamePipe Server Implement on Local Machine......................................34 20. HMI Interface Flow.................................................................................................... 35 21. Intensity Bar Implementation.................................................................................. 47 22. HMI Simulator Flow...................................................................................................49