Brain Computer Interfaces for Games

Masaryk University Faculty of Informatics Brain Computer Interfaces for Games Master’s Thesis Bc. Roman Konečný Brno, Fall 2017 Masaryk University Faculty of Informatics Brain Computer Interfaces for Games Master’s Thesis Bc. Roman Konečný Brno, Fall 2017 This is where a copy of the official signed thesis assignment and a copy ofthe Statement of an Author is located in the printed version of the document. Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Bc. Roman Konečný Advisor: Doc. Fotis Liarokapis PhD i Acknowledgement I want to thank all the wonderful people who have supported me in my studies and writing this thesis. I also want to thank my supervisor doc. Fotios Liarokapis, PhD for his guidance, support and kindness. iii Abstract The aim of this master’s thesis is to create a novel application for controlling simple computer games through EEG technology and conduct an experiment focusing on the playability of such a game on 30 test subjects. Data from the device used for the brain computer interaction should be collected and analyzed, as well as the questionnaires used for measuring the user experience. The goal of the project is to propose a suitable EEG hardware for the purposes of the application, designing and developing the application itself, running the experiment and gathering and analyzing the data that was collected, especially the correlations between the achieved results and gender. iv Keywords Brain-Computer Interface, BCI, Unity, Game, Neurosky mindwave v Contents 1 Introduction 1 2 Background 3 2.1 Brain computer interfaces ...................3 2.1.1 Introduction to Brain Computer Interfaces . .3 2.1.2 Types of BCI . .3 2.2 EEG ..............................4 2.3 BCI Approaches ........................4 2.3.1 Recording ordinary brain activity . .4 2.3.2 Event Related Potentials . .5 2.3.3 Steady-State Visual Evoked Potentials . .5 2.3.4 Motor Imagery . .5 2.3.5 Slow Cortial Potential Shifts . .6 2.4 BCI Systems ..........................6 2.4.1 Wireless BCI systems for consumer use . .6 2.4.2 Wireless BCI systems for research uses . .8 2.5 NeuroSky MindWave ..................... 12 2.6 Related work .......................... 12 2.7 Used technologies ....................... 16 2.7.1 Unity . 16 2.7.2 Visual Studio . 16 2.7.3 GIMP . 17 3 Initial project 19 3.1 Used technologies ....................... 19 3.2 BCI input method ....................... 19 3.3 Game description ....................... 22 3.4 Issues ............................. 22 4 Methodology 25 4.1 Game description ....................... 25 4.1.1 Requirements . 25 4.1.2 Game design . 26 4.2 Implementation ........................ 28 4.2.1 Used assets . 29 4.2.2 Controls . 30 vii 4.2.3 Running . 32 4.2.4 Shooting . 32 4.2.5 User interface . 33 4.2.6 Logging . 37 4.3 Future work .......................... 37 4.3.1 Sounds . 38 4.3.2 Additional game modes . 38 4.3.3 New environments and archer skins . 38 4.3.4 Online leaderboard . 39 4.3.5 Porting to other platforms . 39 5 Experiment 41 5.1 Testing group ......................... 41 5.2 Stages of the experiment .................... 41 5.3 Issues during the experiment ................. 43 6 Result Analysis 45 6.1 Procedure ........................... 45 6.2 Demographics ......................... 45 6.3 NASA Task Load Index .................... 46 6.4 Experience questionnaire ................... 48 6.5 Open questions ........................ 49 6.6 Game results ......................... 51 7 Conclusion 57 Bibliography 59 A Content of the online folder 65 viii List of Tables 2.1 Frequencies Generated By Different Types of Activities in the Brain. [28] 14 4.1 Used assets 30 6.1 Age of the participants according to gender 45 6.2 Daily use of a computer according to gender 45 6.3 Occupation of the participants according to gender 46 6.4 Highest qualification achieved according to gender 46 6.5 NASA TLX Questions 47 ix List of Figures 2.1 Emotiv EPOC [15] 7 2.2 Neural Impulse Actuator [17] 8 2.3 B-Alert X [19] 9 2.4 Quasar DSI 10/20 [21] 10 2.5 Enobio [23] 11 2.6 g.MOBIlab+ [24] 11 2.7 MindWave diagram [27] 13 2.8 2D car game using SSVEP [29] 15 2.9 Tower defense game controlled by SSVEP [31] 15 2.10 Personal edition of Unity. 16 3.1 Input using both MI and SSVEP 20 3.2 Input using SSVEP only with 4 directions 21 3.3 Input using SSVEP only with 8 directions 21 3.4 Car game prototype 23 4.1 Initial idea of aiming in the shooting part of the game — dynamic crosshair size. On the left, low meditation level crosshair is present. On the right, a small crosshair is displayed which is connected to high meditation level. 28 4.2 Mechanism for controlling the shooting — actual game screenshot. 29 4.3 UML diagram depicting the usage of the NeuroSky MindWave controlling interface in an application 31 4.4 Screen shot of the running part of the game 33 4.5 Screen shot of the shooting part of the game — successful attempt 34 4.6 Screen shot of the game menu 35 4.7 Screen shot of the leaderboard 36 4.8 Screen shot of the game UI after finishing the run 37 5.1 Female participant during the experiment 42 6.1 Chart showing the results of the NASA TLX questionnaire according to gender 48 6.2 Chart showing the results of the experience questionnaire according to gender 50 xi 6.3 Time required for finishing the game according to gender 52 6.4 Time spent using attention/meditation according to gender 53 6.5 Time required for finishing the game and time spent using attention/meditation of each participant 53 6.6 Shooting attempts according to gender 54 6.7 Attention levels according to gender 54 6.8 Meditation levels according to gender 55 xii 1 Introduction Over last two decades, the computers and computer games became common. The interaction with the computer, however, is still very limited to the usage of computer peripherals such as keyboard and mouse, which requires the user to be able to operate such devices. One approach to overcome this obstacle for the users which cannot use their hands (or feet in specific situations) is to utilize the Brain Computer Interface. This allows direct communication between the computer and human brain without the need of using any other devices and enables disabled people to use the computer or control artificial limbs. As the research in this field still continues, there are already solutions that can be used and are affordable to a broader audience. The aim of this thesis is to create a novel application for controlling simple computer games through EEG technology. The device used for collecting the brain data should be easy to use, affordable and should provide good experience for the player. The game should be tested by a group of users, evenly split between males and females, and the collected results should be analyzed. The analysis should cover both the user experience — how well they could control the game, how easy it is to adapt to this new controlling paradigm etc., and the actual game results in terms of how well the males and females did in the game. For the brain computer interaction, a NeuroSky MindWave headset was used as it provides an affordable and simple way how to connect the brain with the computer. The initial idea was to use a different BCI approach, which is discussed in chapter 3. Unfortunately, due to the issues, it was not possible to use this approach, so the entire project had to be redesigned and different technologies had to be used. The structure of the thesis is as follows: Chapter 2 provides background information about Brain Computer Interfaces and technologies used for the game development. Chapter 3 describes the initial project idea, the design and the issues that forced the author to change the device. Chapter 4 describes the game, its design and the implementation. Chapter 5 summarizes the testing process and the results of the testing are presented in Chapter 6. Finally, the conclusion and future work is described in Chapter 7. 1 2 Background 2.1 Brain computer interfaces 2.1.1 Introduction to Brain Computer Interfaces Brain computer interface (BCI) is a method of communication between human and computer based on neural activity generated by the brain and is independent of its normal output pathways of peripheral nerves and muscles [1]. Brain computer interfaces measure brain activity, process it and produce control signals that reflect the user’s intent that can be then by connecting the device to the computer [2]. 2.1.2 Types of BCI Brain computer interfaces can be split into three main categories. An invasive approach uses implantable micro-electrodes that are placed directly into the brain cortex during a neurosurgery and has the highest signal quality [3]. These devices can provide functionality to para- lyzed people. Invasive BCIs are also used to restore vision of a subject by connecting the brain with external cameras and to re-enable the use of arms and legs by using brain controlled robotic prosthetics. As they rest in the grey matter, invasive devices produce the highest quality signals of BCI devices but are vulnerable to further expanding of scars on brain tissue which can lead to weakening or even losing the signal [4].

Load more