WeST People and Knowledge Networks Faculty 4: Computer Science Institute for Web Science and Technologies Multi-model optimization of gaze and touch-based PIN Entry Master’s Thesis in partial fulfillment of the requirements for the degree of Master of Science (M.Sc.) in Web Science submitted by Daniyal Akbari First supervisor: Prof. Dr. Steffen Staab Institute for Web Science and Technologies Second supervisor: Dr. Chandan Kumar Institute for Web Science and Technologies Koblenz, July 2018 Statement I hereby certify that this thesis has been composed by me and is based on my own work, that I did not use any further resources than specified – in particular no references unmentioned in the reference section – and that I did not submit this thesis to another examination before. The paper submission is identical to the submitted electronic version. Yes No I agree to have this thesis published in the library. I agree to have this thesis published on the Web. The thesis text is available under a Creative Commons License (CC BY-SA 4.0). The source code is available under a GNU General Public License (GPLv3). The collected data is available under a Creative Commons License (CC BY-SA 4.0). ........................................................................................... (Place, Date) (Signature) iii Zusammenfassung Abstract Knowledge-based authentication methods are vulnerable to Shoulder surfing phe- nomenon. The widespread usage of these methods and not addressing the limi- tations it has could result in the user’s information to be compromised. User au- thentication method ought to be effortless to use and efficient, nevertheless secure. The problem that we face concerning the security of PIN (Personal Identification Number) or password entry is shoulder surfing, in which a direct or indirect ma- licious observer could identify the user sensitive information. To tackle this issue we present TouchGaze which combines gaze signals and touch capabilities, as an input method for entering user’s credentials. Gaze signals will be primarily used to enhance targeting and touch for selecting. In this work, we have designed three dif- ferent PIN entry method which they all have similar interfaces. For the evaluation, these methods were compared based on efficiency, accuracy, and usability. The re- sults uncovered that despite the fact that gaze-based methods require extra time for the user to get familiar with yet it is considered more secure. In regards to efficiency, it has the similar error margin to the traditional PIN entry methods. v Acknowledgement I take this opportunity to express my profound gratitude and deep regards to my thesis advisors Prof. Dr. Steffen Staab and Dr. Chandan Kumar for their exemplary guidance, monitoring and constant encouragement throughout the course of this project.The door to Dr. Kumar’s office was always open to me whenever I ran into a problems or had a question about my research or writing. He consistently allowed this paper to be my own work but steered me in the right direction whenever he thought I needed it. Their input and patience have been invaluable in helping me to learn how to do research, and to navigate some of the of the more emotionally challenging aspects of this project. Their creativity and wisdom are inspirational. I would also like to acknowledge Mr. Raphael Menges, who I am gratefully in- debted to for his effort to help me during the implementation phase and for his very valuable comments on this thesis. I would also like to thank the participants who were involved in the evaluation process for this project. Without their passionate participation and input, the exper- iment could not have been possible. Finally, I must express my very profound gratitude to my mother and dedicate this work to the memory of my father Dr. Mohammad Akbari, for providing me with unfailing support and continuous encouragement throughout my years of study and through the process of researching and writing this thesis. This accomplishment would not have been possible without them. Thank you. vii Contents 1 Introduction1 2 Background and Related work4 2.1 Technical information............................4 2.2 Eye Tracking and Touch..........................4 2.3 Personal Identification Number (PIN) Entry...............6 3 Research Problem 11 4 Methodology 13 4.1 Challenges.................................. 13 4.1.1 Security............................... 13 4.1.2 Usability............................... 13 4.1.3 Speed................................. 14 4.1.4 Error Rate.............................. 14 4.1.5 Interface............................... 14 4.1.6 Interaction.............................. 15 4.2 Approach................................... 15 4.2.1 Touch PIN entry concept...................... 15 4.2.2 Touch + Gaze PIN entry concept................. 16 4.2.3 TouchGaze PIN entry concept................... 16 4.3 Implementation............................... 17 4.3.1 Touch PIN entry method...................... 17 4.3.2 Touch + Gaze PIN entry method................. 18 4.3.3 TouchGaze PIN entry method................... 19 5 Evaluation 21 5.1 Experimental Set up............................ 21 5.2 Participants................................. 21 5.3 Apparatus.................................. 22 5.4 Counter Balancing............................. 22 5.5 Procedure.................................. 22 5.5.1 Quantitative Evaluation...................... 24 5.5.2 Qualitative Evaluation....................... 27 5.6 Results.................................... 28 5.7 Discussion.................................. 36 6 Conclusion 38 7 Future Work 40 8 Bibliography 42 viii Appendices 46 ix 1 Introduction Human-Computer Interaction (HCI) is focused on the joint performance of tasks be- tween humans and computers and how they communicate. There are two compo- nents in this exchange of information, Input and Output. Input components identify and sense the user’s desired task and information to be communicated to the com- puter. There are two types of Input methods, Direct and Indirect. The Indirect input refers to the method where the actions of the user will translate into data or com- mands to be entered into a system. The Indirect input methods have been around since the beginning of Web development and they are still usable and prominent. Keyboard and Mouse are prime examples of indirect interaction methods. The di- rect input refers to the devices which have no intermediary and the movement of the user’s body is equal to the input to the system. The Direct input methods have been introduced to facilitate users to have a natural feeling of interaction, for example, touch interaction. A touch screen refers to an electronic visual display which is able to detect the presence and location of a touch on the screen. The touchscreen technology can be found in Smartphones, Laptops, Tablets and even ATMs. The touchscreen enables the direct interaction with what is being displayed and requires no intermediary device like a mouse. Touchscreen as a Direct input method enables users to simply point at their target and select it, unlike indirect methods which the user is required to press multiple keys or move the cursor to select an object. Direct and Indirect in- put methods have been evaluated and the experiment done by D. Schmidt showed that accuracy and time efficiency is much more optimal when using direct condi- tions [SBG09]. Touch screens are easier to learn and more efficient compared to indirect input methods. It is important to mention that touch screens are known to be practical for selecting large targets and they are not very accurate when used for small targets and they tend to be susceptible to error. During an interaction with a touchscreen interface the user firstly looks at the screen and targets the desired ob- ject to be selected and then proceed to actually selecting the object by touching the location of the object on the screen. Eye tracking had been introduced as a way of direct interaction which could pave the way for new technologies and devices to be introduced for end-users. It is be- coming a popular way of interaction. Gaze coordinates could be used to pinpoint the target selected by the user on the screen, then proceed to do a command more efficiently. We hypothesize that the limitations and disadvantages mentioned for touchscreen interfaces can be improved by adding gaze capabilities. It is worth mentioning that gaze is used anyway when people use the touch method and a fin- ger or hand movement follows the user’s gaze. Therefore the targeting can be done via gaze and the selection by touch, which will improve the efficiency and also the screen is not obscured by hand during targeting. The multimodal approach will ben- efit the users with a user-friendly interface and aims to make the PIN-entry secure and to improve the accuracy of target selection and reduce unwanted selections. The 1 focus of this Master thesis is to analyze how the direct method of interaction could be optimized for PIN-entry, i.e, a multimodal approach of enhancing the popular Touch-based interaction using cues and context from gaze signals, so that it would increase accuracy, user-friendliness, efficiency, and security. It should be compara- ble to the current method of direct interaction and resolve its limitations. One can argue that this is not the most optimal and efficient way of interaction since it can be time-consuming for the user. The touchscreen interfaces are undesirable in some applications since the finger is considered to be a large pointer and cannot accu- rately point to small areas on the screen. It is also worth mentioning that the finger or hand can block the visibility of the screen during the interactions since most of the time hands have to move in front of the screen to reach the intended target. User authentication refers to the process in which the identity of the user is proven in a system, to gain access permission. Authentication processes are part of the user’s daily interactions, whether it is their email, online banking or online partici- pation to vote.