Vision-Based Human Interaction Devices in a 3D environment Using Nintendo Wiimote, Webcam and DirectX Thomas Luel Florent Mazzone Vision, Graphics and Interactive Systems Aalborg University June 1, 2009 AALBORG UNIVERSITY DEPARTMENT OF ELECTRONIC SYSTEMS Niels Jernes Vej 12 – DK-9220 Aalborg East Phone 99 40 86 86 TITLE : Vision-Based Human Interaction Devices in a 3D Environment th PROJECT PERIOD : 10 Semester, February 2009 to June 2009 PROJECT GROUP : 1021 ABSTRACT This report documents the development of new Human Interface Devices (HID) and their use in a 3D environment to enhance the immersion factor of a modeling tool such as an architecture PARTICIPANTS: designer software. The study focuses on studying and comparing two common devices used in vision-based motion tracking: the wiimote THOMAS LUEL developed by Nintendo and a webcam. New means of interaction between the user and software, going beyond the mouse and keyboard bounds, are explored and described to interact FLORENT MAZZONE efficiently and intuitively with desktop objects. In this system, the user’s fingers and head are infrared lighted by adapted gloves and frames fit SUPERVISOR : with infrared diodes. The wiimote, developed by Nintendo, provides a high accuracy camera sensor ZHENG -HUA TAN whereas the webcam has a good detection range. Both are adapted answers to the application requirements. A 3D DirectX architect software, PUBLICATIONS : 3 Master Builder, has been implemented, allowing NUMBER OF PAGES : 99 the user to create his own buildings using his fingers and to view the result in 3D by moving his FINISHED : JUNE 1, 2009 head. The resulting projects and the users tests revealed that infrared vision-based tracking fits perfectly to the tracking requirements and that handling objects by using both hands is much more interactive for the user than using common human device interfaces such as the mouse. This report can be published or reproduced without permission provided that the source is mentioned. Copyright ©2009, project group 1021, Aalborg University Preface Report purpose This report is part of the 10 th semester project of the “Vision, Graphics and Interactive Systems” Master program. It has been written in 2009 at the Aalborg University Department of Electronic Systems and presents the Master thesis of the group VGIS 09gr1021. The project aim is to design and implement new means of interaction such as fingers and head tracking between the user and a 3D architecture modeling software, in order to improve the ease of use of this type of application. The overall project has been documented in this report, which describes the design choices, implementation, testing and improvements of the solution. Report overview Seven main sections compose this report. The Pre-Analysis describes common hardware and software technologies used to improve the user’s immersion in an application and to track his motions. The Analysis part defines more precisely the technologies used in the project and presents the specification of requirements for the ideal system. The Design section describes the different modules developed for this project. A first sub-section presents the hardware components developed for the fingers and head tracking. The tracking software is then described in a second part. Finally, the development of the three dimensions game and of the user interface using DirectX9 is detailed. The Implementation part describes the development of the different modules designed, the problems encountered during this phase and some modifications of the initial design. The Tests part describes how the usability tests have been made to validate the application design choices and the results obtained. The Tests section then presents all the modifications of the application made to enhance the interaction between the user and the software and to correct some bugs discovered during the test period. Finally, the Conclusion will explain the project achievement and the improvement perspectives. A list of the figures included is available; each figure caption is preceded by his chapter number separated by a dot from its order number in the section. The appendix contains the references and technical documents. A number in parenthesis marks each reference quoted. Technology notice The CD-Rom includes the binaries with documented source codes of MasterBuilder.exe, the 3D environment, and of MotionTracking.dll, the motion-tracking library. The CD-ROM also contains a MotionTracking.dll sample application, the webcam driver, a tool to handle the camera properties, a documentation of the MotionTracking.dll external methods (.HTM and Microsoft Compressed HTML) detailed installation instructions and the electronic components datasheets. The source code has been developed in C# using .Net framework 2 using Microsoft Visual Studio 2005 development environment. The report has been written with Microsoft Office 2007, the schematic and diagrams have been drawn with Microsoft Visio 2003 and the quality of the screen or hardware pictures has been improved using Paint.NET. Acknowledgements We would like to thank all the people who helped us to achieve our project, and more particularly: • Ben Krøyer and Peter Boie Jensen, Electronic Systems Department assistant engineers, for their help and good advice during the electronic components development. • Lars Bo Larsen, our semester coordinator for his support. • Mette Billeskov, semester secretary, for her support regarding the administrative issues. • Our supervisor Zeng-Hua Tan, our project supervisor, for his help, management and advice. • All the testers who helped to check, validate and improve the design choices. • The computer sciences workshop for their help regarding the computers and network issues. • The components workshop that helped us to order all the needed components. Table of contents Part I Introduction ............................................................................................................................. 1 Presentation ........................................................................................................................................ 2 Problem statement.............................................................................................................................. 3 Fundamental requirements................................................................................................................. 3 Limitations ........................................................................................................................................... 3 Part II Pre-Analysis ............................................................................................................................. 5 1 “Immersion” factor in software and games ................................................................................ 6 2 Motion tracking ......................................................................................................................... 10 Part III Analysis .................................................................................................................................. 13 1 Head Tracking ............................................................................................................................ 14 2 What is Fingers tracking? .......................................................................................................... 17 3 Installation layout ...................................................................................................................... 21 4 Choice of the Application .......................................................................................................... 24 5 Choice of the programming language ....................................................................................... 27 6 Specification of requirements ................................................................................................... 32 Part IV Design .................................................................................................................................... 33 1 General Structure ...................................................................................................................... 34 2 Tracking ..................................................................................................................................... 36 3 3D Game Design ........................................................................................................................ 55 4 User Interface ............................................................................................................................ 68 Part V Implementation ..................................................................................................................... 81 1 Infrared Frames ......................................................................................................................... 82 2 Infrared Gloves .......................................................................................................................... 83 3 MotionTracking DLL ................................................................................................................... 85 4 Implementation of the 3D Environment ................................................................................... 87 Part VI Tests......................................................................................................................................