Masaryk University Faculty of Informatics Multimodal Control for Augmented Reality Applications Bachelor’s Thesis Michal Rychetník Brno, Spring 2019 Masaryk University Faculty of Informatics Multimodal Control for Augmented Reality Applications Bachelor’s Thesis Michal Rychetník Brno, Spring 2019 This is where a copy of the official signed thesis assignment and a copy ofthe Statement of an Author is located in the printed version of the document. Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Michal Rychetník Advisor: doc. Ing. RNDr. Barbora Bühnová, Ph.D. Consultant: Ing. Michal Košík, M.Sc. i Acknowledgements I would like to thank tremendously to my advisor doc. Ing. RNDr. Barbora Bühnová, Ph.D., and consultant Ing. Michal Košík, M.Sc. that had massive patience with me, both always pushed me in the right direction and helped me immensely to make this thesis see the light of the day. iii Abstract As the augmented reality and smart glasses appear more prominently on the market and even start to influence everyday work in many industries, questions about the use of those technologies arise. More traditional touch control methods may be insufficient when it comes to wearable technology and mainly smart glasses or headsets. On the other hand, touchless methods could be more comfortable to use and be even more efficient. Alternatively, touchless methods like eye track- ing can improve the precision of other inputs. The thesis shows the possibilities of different means of control of augmented reality appli- cation with a focus on touchless methods, which are introduced in the thesis and implemented by the prototype. This prototype was further subjected to end-user testing to discover the current possibilities and how users accept those control methods. iv Keywords Multimodal control, smart glasses, augmented reality, AR, input meth- ods, voice control, hand gestures, eye tracing v Contents Introduction 1 1 Smart glasses and augmented reality 3 1.1 Sensors and input methods on smart glasses .........3 1.2 Smart glasses currently on the market ............4 1.3 Examples of smart glasses and augmented reality utilization 6 2 Control methods for AR applications 9 2.1 Touch methods ........................9 2.1.1 On-device interaction . 10 2.1.2 On-body interaction . 10 2.2 Touchless methods ....................... 11 2.2.1 Voice input . 11 2.2.2 Hand gesture . 12 2.2.3 Head movement . 12 2.2.4 Eye tracking . 13 3 Prototype design and development 15 3.1 Available toolkits ....................... 15 3.1.1 Available toolkits for voice control . 15 3.1.2 Available toolkits for hand gesture . 21 3.1.3 Available toolkits for eye tracking . 22 3.2 Prototype design ....................... 23 3.3 Prototype development .................... 24 4 Prototype testing 27 4.1 Testing of individual control methods ............ 27 4.1.1 Voice input testing . 27 4.1.2 Hand gesture testing . 28 4.1.3 Eye tracking testing . 28 4.2 Comparison test ........................ 28 5 Results from prototype testing 31 5.1 Individual tests results .................... 31 5.1.1 Voice input . 31 5.1.2 Hand gesture . 33 vii 5.1.3 Eye tracking . 34 5.2 Comparison test ........................ 34 5.3 Possible future follow-up work ................ 37 Conclusion 39 Bibliography 41 A Comparison test questionnaire 51 B Electronic appendix 55 viii List of Tables 1.1 Sensors and features on smart glasses 5 3.1 Browser support of voice input libraries 16 3.2 Languages supported by voice input libraries 17 3.3 Available functions of voice input libraries 18 3.4 Browser support of hand gesture and eye tracking toolkits 22 ix List of Figures 4.1 The testing grid for eye tracking 29 5.1 Unsuccessful recognition rate of English commands 32 5.2 Unsuccessful recognition rate of Czech commands 33 5.3 Result distribution of comfortability of control methods usage 35 5.4 Result distribution of willingness to use control methods in public or at work 36 xi Introduction Augmented reality (AR) and smart glasses are not particularly new on the market, but they are neither widespread nor commonly used. Not yet, at least. Augmented reality, virtual reality (VR) and even mixed reality (MR) products are on the rise, and some predict their continuous expanding growth [1]. Utilization can be from simple applications showing a manual to a worker to more complex ones highlighting the specific object and projecting the whole work process [2]. Whatis more, some companies like DHL [3] or GE [4] already started testing ways to make use of augmented reality. There is a significant potential for these somewhat new technolo- gies and various fields from logistics and manufacturing [5, 3]to healthcare [6] or entertainment [7, 8] and military[9] can benefit from AR applications. Moreover, the influence of augmented reality is al- ready more noticeable as Apple introduced AR Measure app [10] and AR game Pokémon GO made over 2 billion dollars with the end of 2018 [11]. Nevertheless, there are still things that need sorting out, given this field is somewhat new. Since smart glasses are wearable technology and just a keyboard or touchscreen on the device could not be an option for the user, there is a need for a different way of controlling the device. Thesis aim The main objective of the thesis is to introduce control methods for augmented reality applications in addition to presenting a prototype that could be used for end-user testing to examine the potential of each control method. Structure of the thesis Chapter 1 introduces smart glasses and their presence on the market today. Chapter 2 takes a closer look at control methods of AR applications at the whole and presents some of the control gadgets in development that might become relevant very soon. The core part of the thesis follows, which presents the development of a prototype with currently available options to test some of the possible control methods – voice input, hand gestures, and eye tracking. Firstly, 1 Chapter 3 describes the development and design of a prototype while showing some toolkits that can be used for different inputs as well as describing their usage. Chapter 4 follows up with actual prototype testing and how exactly were those methods tested. In continuation, Chapter 5 showcases the results of those tests and their comparison with other similar studies in this area. 2 1 Smart glasses and augmented reality Today, there is plenty of different types of smart glasses. This chapter covers the features and sensors of some currently available smart glasses. Additionally, it presents some of the smart glasses to illustrate typical features as well as to introduce some of the different available input approaches. Furthermore, this chapter also shows some of the possible utilization of smart glasses and augmented reality. 1.1 Sensors and input methods on smart glasses Given the increasing amount of smart glasses models, there is a slight diversity in their features and what sensors and other technical spec- ification they offer. Not all of them use or can use the same input methods. Nevertheless, practically every model possesses a camera, microphone, accelerometer, magnetometer, and gyroscope. Smart glasses can use different types of camera, which can be positioned in different ways. The primary camera (or possibly more cameras) is positioned to follow the user’s field of vision, where it can capture hand gestures. The most common is a regular RGB camera, although some have a depth camera, that provides depth perception of images and makes it easier to work in 3D space to, for instance, position some elements. Alternatively, smart glasses may have an infrared camera for thermal vision. Cameras can also be positioned for other purposes such as to scan eye and act as an eye-tracking device in some capacity. The microphone is a standard feature on smart glasses, as can be seen in Table 1.1. The wearer may use microphone either for voice commands or for recordings. Speakers, on the other hand, are not necessarily a built-in feature. However, if included, they may be either regular audio speakers or speakers using bone conduction. The gyroscope, accelerometer, magnetometer, and GPS all detect the orientation of the device in some way. Gyroscope measures ori- entation with rotation around three primary axes. Accelerometer de- termines the acceleration of the device, which can then let the device detect if the wearer is in motion or being still. This detection can further aid in reducing noise input or reducing wrong input recognition (for 3 1. Smart glasses and augmented reality example, unwanted gesture inputs). A magnetometer is essentially a compass, as it evaluates magnetic fields, and can be also helpful with the calculation of the wearer’s position and orientation. Some smart glasses can be equipped with GPS so the application may tell directions, position on the map, or react to certain places like showing information about some building or distance from the destination. Another standard equipment is the light sensor for light detection, which allows the application to react to different environments and conditions, mainly to adjust the brightness of the display. Additionally, proximity sensor allows the device to detect obstacles or detect if the smart glasses are currently worn. The eye tracker is lately becoming more frequently embraced by smart glasses and headsets. Tracker reads eye movement and gaze, both of which could improve control over the device or help adjust elements on display. Lastly, smart glasses might have built-in touch sensors like a track- pad or some mechanical interface like buttons.