NATURAL HUMAN-COMPUTER INTERACTION APPLIED ON DESKTOP APPLICATIONS USING MULTIMODAL INPUT DEVICES by ANASTASIOS VLASOPOULOS B, TEI of Crete, 2014 A THESIS submitted in partial fulfillment of the requirements for the degree MASTER OF SCIENCE DEPARTMENT OF INFORMATICS ENGINEERING SCHOOL OF APPLIED TECHNOLOGY TECHNOLOGICAL EDUCATIONAL INSTITUTE OF CRETE 2016 Approved by: Major Professor Nikolaos Vidakis i Copyright ANASTASIOS VLASOPOULOS 2017 ii Abstract Human-Computer Interaction as a term that exists from the early 80s. It is an evolutional term which has its root from the term Man-Machine Interaction. Man-Machine Interaction as an idea came out of studies in the 1940s. Researchers from that period tried to understand how human interacts with any kind of machine and they wanted to realize how this interaction could be better and more efficient from the user experience perspective. A lot of years have passed since the user was interacting with computers through difficult and not user-friendly interfaces like a command console for example. Today HCI is accomplished by as simple and user friendly interfaces as possible. User is in the center of this process and combination of knowledge in computer science, behavioral sciences, design, media studies, and several other fields is considered in order to make HCI a better user experience. If we take a look at today’s HCI we can say that the way that human interacts with computer has become more natural. Computer is treated like a human and the means of interaction are more physical (speech, gestures or touch). This thesis will be a proof of concept to claim that it is actually feasible to create an interface through which the user can interact with computer by a more natural and multimodal way. The implementation part will be a game that the user will interact with though natural means. This game will be an open source project which will be adapted in the terms of Natural HCI and multimodality. The way in which this will be accomplished is through the use of multimodal input devices. The first MID that will be used is the Microsoft Kinect and the second is the Leap. These two devices will be the main controllers of the game. Of course each of these devices will be analyzed how they work and it will be a guide of how they can be configured for further usage. Finally the purpose of this thesis is to demonstrate how HCI is changed through years and how is being experienced today through an open source game. Machines plays a key role in human’s life so it is very important for all of us to understand how humans interact with them in order to make our lives more easy and delectable. This thesis is a serious effort to accomplish this goal. iii Table of Contents Copyright ........................................................................................................................................ ii Abstract .......................................................................................................................................... iii List of Figures ................................................................................ Error! Bookmark not defined. List of Tables ................................................................................. Error! Bookmark not defined. Chapter 1 - Introduction .................................................................................................................. 1 Summary ..................................................................................................................................... 2 Research questions ...................................................................................................................... 3 Structure ...................................................................................................................................... 3 Chapter 2 - User Interfaces History Review ................................................................................... 4 Computer Interfaces (CI) ............................................................................................................ 4 Command Line Interfaces (CLI) ................................................................................................. 5 Graphical User Interfaces (GUI) ............................................................................................... 6 Natural User Interfaces (NUI) .................................................................................................... 7 Chapter 3 - NUI Review - State Of The Art ................................................................................... 9 Introduction ................................................................................................................................. 9 Nintendo Wii ............................................................................................................................... 9 PlayStation Move ...................................................................................................................... 10 Kinect ........................................................................................................................................ 11 Leap Motion .............................................................................................................................. 14 Xtion Pro LIVE by Asus ........................................................................................................... 17 MYO by Thalmic ...................................................................................................................... 18 Oculus Rift by Oculus VR ........................................................................................................ 19 Google Glass by Google X ....................................................................................................... 20 Blended Learning ...................................................................................................................... 24 Chapter 4 - Implementation’s Technologies & Environment ....................................................... 27 Programming Language ............................................................................................................ 27 Build System ............................................................................................................................. 28 Integrated Development Environment ...................................................................................... 36 iv Chapter 5 - Analysis & Architectural Design ............................................................................... 37 Use Case Diagrams ................................................................................................................... 37 Class Diagram ........................................................................................................................... 41 Chapter 6 - Implementation & Scenarios of Use .......................................................................... 42 Code Analysis ........................................................................................................................... 42 Main Class ............................................................................................................................ 42 ConfiguredListener Class ...................................................................................................... 43 Leap Java API Extension ...................................................................................................... 47 Scenarios of Use ....................................................................................................................... 48 Media Player Manipulation ................................................................................................... 49 Browser Manipulation .......................................................................................................... 52 Microsoft Word Manipulation .............................................................................................. 54 Microsoft Power Point Manipulation .................................................................................... 56 Windows Photo Viewer Manipulation .................................................................................. 57 Chapter 7 - Conclusions & Future Work ...................................................................................... 59 References ..................................................................................................................................... 60 Appendix A: Environment Configuration & Integration Instructions .......................................... 64 Java ........................................................................................................................................... 64 GitHub ...................................................................................................................................... 66 Git ............................................................................................................................................. 66 IntelliJ ....................................................................................................................................... 75 v Chapter 1 - Introduction Evolution of technology over the last years has changed the relationship between man and computer in a way that human mind could not have been imagined a few decades ago. Multimodality and