A Multiple Optical Tracking Based Approach for Enhancing Hand- Based Interaction in Virtual Reality Simulations Adam Grant Worrallo BSc (Hons), MSc A thesis submitted in partial fulfilment of the requirements of the University of Wolverhampton for the degree of Doctor of Philosophy September 2020 This work or any part thereof has not previously been presented in any form to the University or to any other body whether for the purposes of assessment, publication or for any other purpose (unless otherwise indicated). Save for any express acknowledgments, references and/or bibliographies cited in the work, I confirm that the intellectual content of the work is the result of my own efforts and of no other person. The right of Adam Grant Worrallo to be identified as author of this work is asserted in accordance with ss.77 and 78 of the Copyright, Designs and Patents Act 1988. At this date copyright is owned by the author. Signature……A G Worrallo……………………………. Date…………26/09/2020…………………………… II Acknowledgements Throughout my PhD I have received a great deal of support and advice. I would like to first thank my supervisor, Dr Tom Hartley whose expertise, advice and endless support was invaluable throughout my studies. In addition, I would like to thank my PhD administrator, friends and colleagues at the University of Wolverhampton for their support. I would also like to thank my entire family for their endless support and encouragement throughout my studies. You were always there for me with a sympathetic ear and made the hardest parts seem that much easier. I would like to say a special thank you to my uncles; Dave and Tim, you lit and nurtured my passion for computer science in my early years. I would not be where I am today without your input. Finally, I need to say a special thank you to my father, Mark, for always being there for me no matter what. From my first day of school, you have always supported me with my studies, going to the library with me on a Saturday morning and helping with homework in the afternoon. It’s been many years since we did homework together, but your support has never wavered. Your endless support during this endeavour has been invaluable. Whether it was offering advice or just a friendly face at the end of a long day, I could not have done this without you. III Abstract Research exploring natural virtual reality interaction has seen significant success in optical tracker-based approaches, enabling users to freely interact using their hands. Optical based trackers can provide users with real-time, high-fidelity virtual hand representations for natural interaction and an immersive experience. However, work in this area has identified four issues: occlusion, field-of-view, stability and accuracy. To overcome the four key issues, researchers have investigated approaches such as using multiple sensors. Research has shown multi-sensor-based approaches to be effective in improving recognition accuracy. However, such approaches typically use statically positioned sensors, which introduce body occlusion issues that make tracking hands challenging. Machine learning approaches have also been explored to improve gesture recognition. However, such approaches typically require a pre-set gesture vocabulary limiting user actions with larger vocabularies hindering real-time performance. This thesis presents an optical hand-based interaction system that comprises two Leap Motion sensors mounted onto a VR headset at different orientations. Novel approaches to the aggregation and validation of sensor data are presented. A machine learning sub-system is developed to validate hand data received by the sensors. Occlusion detection, stability detection, inferred hands and a hand interpolation sub-system are also developed to ensure that valid hand representations are always shown to the user. In addition, a mesh conformation sub-system ensures 3D objects are appropriately held in a user’s IV virtual hand. The presented system addresses the four key issues of optical sessions to provide a smooth and consistent user experience. The MOT system is evaluated against traditional interaction approaches; gloves, motion controllers and a single front-facing sensor configuration. The comparative sensor evaluation analysed the validity and availability of tracking data, along with each sensors effect on the MOT system. The results show the MOT provides a more stable experience than the front-facing configuration and produces significantly more valid tracking data. The results also demonstrated the effectiveness of a 45-degree sensor configuration in comparison to a front-facing. Furthermore, the results demonstrated the effectiveness of the MOT systems solutions at handling the four key issues with optical trackers. V Contents Acknowledgements ...................................................................................... III Abstract ....................................................................................................... IV List of Figures ............................................................................................. XII List of Equations ........................................................................................ XXI List of Algorithms ...................................................................................... XXII List of Tables ........................................................................................... XXIII List of Abbreviations ............................................................................... XXVI Chapter 1 – Introduction ............................................................................... 1 1.1 – Virtual Reality ................................................................................... 1 1.2 – Serious Games ................................................................................. 5 1.3 – Hand Interaction ............................................................................... 6 1.4 – Optical Sensor Issues ....................................................................... 8 1.5 – Academic Questions, Aims and Objectives of the Research ........... 14 1.6 – Thesis Contributions ....................................................................... 16 1.7 – Research Methodology ................................................................... 17 1.8 – Thesis Organization ........................................................................ 21 Chapter 2 – Literature Review .................................................................... 22 2.1 – Overview of Interaction Techniques/Methods used in Virtual Reality ................................................................................................................ 24 2.2 – Hand-Based Input Mechanisms ...................................................... 30 VI 2.2.1 – Indirect/Contact Based Interaction Approaches in VR .............. 31 2.2.2 – Vision/Optical Based Interaction Approaches in VR ................. 48 2.3 – Machine Learning in Optical Tracking Systems .............................. 62 2.4 – Immersion & Presence in VR .......................................................... 68 2.5 – Interactive Virtual Reality in Education ............................................ 73 2.6 – Summary ........................................................................................ 80 Chapter 3 – Design Phase One .................................................................. 84 3.1 – Custom Bracket & External Sensor Configuration ........................... 87 3.2 – External Data Packet Handler ......................................................... 91 3.3 – Data Processor ............................................................................... 95 3.4 – Data Aggregation System ............................................................... 96 3.5 – Hand Validation System ............................................................... 103 3.6 – Post Aggregation Processing System ........................................... 109 3.7 – Vive Arm Tracking System............................................................ 110 3.8 – Inferred Hand Pose System .......................................................... 112 3.9 – Mesh Conformation System .......................................................... 118 3.10 – Summary .................................................................................... 128 Chapter 4 – Experimentation Phase One .................................................. 129 4.1 – Experimental Setup & Procedure .................................................. 130 4.2 – Semi-structured Interview Analysis ............................................... 138 4.2.1 – Likert Questionnaire Analysis ................................................. 138 VII 4.2.2 - Thematic Analysis ................................................................... 141 4.3 – Tracking Data Validity ................................................................... 146 4.3.1 – Data Validation Process ......................................................... 147 4.3.2 – Sensor Tracking Data Validity ................................................ 149 4.3.3 – Chi-Square Test ..................................................................... 153 4.3.4 – Linear Regression Test .......................................................... 155 4.4 – User Performance ........................................................................ 157 4.4.1 - Total Completion Time Condition Comparison ........................ 161 4.4.2