Brain Computer Interfaces for Games

Masaryk University Faculty of Informatics

Master’s Thesis

Bc. Roman Konečný

Brno, Fall 2017

Masaryk University Faculty of Informatics

Brain Computer Interfaces for Games

Master’s Thesis

Bc. Roman Konečný

Brno, Fall 2017

This is where a copy of the official signed thesis assignment and a copy ofthe Statement of an Author is located in the printed version of the document.

Declaration

Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source.

Bc. Roman Konečný

Advisor: Doc. Fotis Liarokapis PhD

Acknowledgement

I want to thank all the wonderful people who have supported me in my studies and writing this thesis. I also want to thank my supervisor doc. Fotios Liarokapis, PhD for his guidance, support and kindness.

iii Abstract

The aim of this master’s thesis is to create a novel application for controlling simple computer games through EEG technology and conduct an experiment focusing on the playability of such a game on 30 test subjects. Data from the device used for the brain computer interaction should be collected and analyzed, as well as the questionnaires used for measuring the user experience. The goal of the project is to propose a suitable EEG hardware for the purposes of the application, designing and developing the application itself, running the experiment and gathering and analyzing the data that was collected, especially the correlations between the achieved results and gender.

iv Keywords

Brain-Computer Interface, BCI, Unity, Game, Neurosky mindwave

Contents

1 Introduction 1

2 Background 3 2.1 Brain computer interfaces ...... 3 2.1.1 Introduction to Brain Computer Interfaces . . .3 2.1.2 Types of BCI ...... 3 2.2 EEG ...... 4 2.3 BCI Approaches ...... 4 2.3.1 Recording ordinary brain activity ...... 4 2.3.2 Event Related Potentials ...... 5 2.3.3 Steady-State Visual Evoked Potentials ...... 5 2.3.4 Motor Imagery ...... 5 2.3.5 Slow Cortial Potential Shifts ...... 6 2.4 BCI Systems ...... 6 2.4.1 Wireless BCI systems for consumer use . . . . .6 2.4.2 Wireless BCI systems for research uses . . . . .8 2.5 NeuroSky MindWave ...... 12 2.6 Related work ...... 12 2.7 Used technologies ...... 16 2.7.1 Unity ...... 16 2.7.2 Visual Studio ...... 16 2.7.3 GIMP ...... 17

3 Initial project 19 3.1 Used technologies ...... 19 3.2 BCI input method ...... 19 3.3 Game description ...... 22 3.4 Issues ...... 22

4 Methodology 25 4.1 Game description ...... 25 4.1.1 Requirements ...... 25 4.1.2 Game design ...... 26 4.2 Implementation ...... 28 4.2.1 Used assets ...... 29 4.2.2 Controls ...... 30

vii 4.2.3 Running ...... 32 4.2.4 Shooting ...... 32 4.2.5 User interface ...... 33 4.2.6 Logging ...... 37 4.3 Future work ...... 37 4.3.1 Sounds ...... 38 4.3.2 Additional game modes ...... 38 4.3.3 New environments and archer skins ...... 38 4.3.4 Online leaderboard ...... 39 4.3.5 Porting to other platforms ...... 39

5 Experiment 41 5.1 Testing group ...... 41 5.2 Stages of the experiment ...... 41 5.3 Issues during the experiment ...... 43

6 Result Analysis 45 6.1 Procedure ...... 45 6.2 Demographics ...... 45 6.3 NASA Task Load Index ...... 46 6.4 Experience questionnaire ...... 48 6.5 Open questions ...... 49 6.6 Game results ...... 51

7 Conclusion 57

Bibliography 59

A Content of the online folder 65

viii List of Tables

2.1 Frequencies Generated By Different Types of Activities in the Brain. [28] 14 4.1 Used assets 30 6.1 Age of the participants according to gender 45 6.2 Daily use of a computer according to gender 45 6.3 Occupation of the participants according to gender 46 6.4 Highest qualification achieved according to gender 46 6.5 NASA TLX Questions 47

List of Figures

2.1 Emotiv EPOC [15] 7 2.2 Neural Impulse Actuator [17] 8 2.3 B-Alert X [19] 9 2.4 Quasar DSI 10/20 [21] 10 2.5 Enobio [23] 11 2.6 g.MOBIlab+ [24] 11 2.7 MindWave diagram [27] 13 2.8 2D car game using SSVEP [29] 15 2.9 Tower defense game controlled by SSVEP [31] 15 2.10 Personal edition of Unity. 16 3.1 Input using both MI and SSVEP 20 3.2 Input using SSVEP only with 4 directions 21 3.3 Input using SSVEP only with 8 directions 21 3.4 Car game prototype 23 4.1 Initial idea of aiming in the shooting part of the game — dynamic crosshair size. On the left, low meditation level crosshair is present. On the right, a small crosshair is displayed which is connected to high meditation level. 28 4.2 Mechanism for controlling the shooting — actual game screenshot. 29 4.3 UML diagram depicting the usage of the NeuroSky MindWave controlling interface in an application 31 4.4 Screen shot of the running part of the game 33 4.5 Screen shot of the shooting part of the game — successful attempt 34 4.6 Screen shot of the game menu 35 4.7 Screen shot of the leaderboard 36 4.8 Screen shot of the game UI after finishing the run 37 5.1 Female participant during the experiment 42 6.1 Chart showing the results of the NASA TLX questionnaire according to gender 48 6.2 Chart showing the results of the experience questionnaire according to gender 50

xi 6.3 Time required for finishing the game according to gender 52 6.4 Time spent using attention/meditation according to gender 53 6.5 Time required for finishing the game and time spent using attention/meditation of each participant 53 6.6 Shooting attempts according to gender 54 6.7 Attention levels according to gender 54 6.8 Meditation levels according to gender 55

xii 1 Introduction

Over last two decades, the computers and computer games became common. The interaction with the computer, however, is still very limited to the usage of computer peripherals such as keyboard and mouse, which requires the user to be able to operate such devices. One approach to overcome this obstacle for the users which cannot use their hands (or feet in specific situations) is to utilize the Brain Computer Interface. This allows direct communication between the computer and human brain without the need of using any other devices and enables disabled people to use the computer or control artificial limbs. As the research in this field still continues, there are already solutions that can be used and are affordable to a broader audience. The aim of this thesis is to create a novel application for controlling simple computer games through EEG technology. The device used for collecting the brain data should be easy to use, affordable and should provide good experience for the player. The game should be tested by a group of users, evenly split between males and females, and the collected results should be analyzed. The analysis should cover both the user experience — how well they could control the game, how easy it is to adapt to this new controlling paradigm etc., and the actual game results in terms of how well the males and females did in the game. For the brain computer interaction, a NeuroSky MindWave headset was used as it provides an affordable and simple way how to connect the brain with the computer. The initial idea was to use a different BCI approach, which is discussed in chapter 3. Unfortunately, due to the issues, it was not possible to use this approach, so the entire project had to be redesigned and different technologies had to be used. The structure of the thesis is as follows: Chapter 2 provides background information about Brain Computer Interfaces and technologies used for the game development. Chapter 3 describes the initial project idea, the design and the issues that forced the author to change the device. Chapter 4 describes the game, its design and the implementation. Chapter 5 summarizes the testing process and the results of the testing are presented in Chapter 6. Finally, the conclusion and future work is described in Chapter 7.

2 Background

2.1 Brain computer interfaces

2.1.1 Introduction to Brain Computer Interfaces Brain computer interface (BCI) is a method of communication between human and computer based on neural activity generated by the brain and is independent of its normal output pathways of peripheral nerves and muscles [1]. Brain computer interfaces measure brain activity, process it and produce control signals that reflect the user’s intent that can be then by connecting the device to the computer [2].

2.1.2 Types of BCI Brain computer interfaces can be split into three main categories. An invasive approach uses implantable micro-electrodes that are placed directly into the brain cortex during a neurosurgery and has the highest signal quality [3]. These devices can provide functionality to paralyzed people. Invasive BCIs are also used to restore vision of a subject by connecting the brain with external cameras and to re-enable the use of arms and legs by using brain controlled robotic prosthetics. As they rest in the grey matter, invasive devices produce the highest quality signals of BCI devices but are vulnerable to further expanding of scars on brain tissue which can lead to weakening or even losing the signal [4]. Partially invasive BCI devices are characterized by splitting the devices into two parts. One part of the system is implanted inside the skull onto the grey matter, but second part of the device is located outside of the body of the subject. Partially invasive BCIs produce better resolution signals than non-invasive BCIs and have a lower risk of forming scar-tissue in the brain than fully-invasive BCIs [4]. A non-invasive BCI is a BCI technology which does not involve implanting a foreign device into the user’s brain. The most common non-invasive BCIs are using electroencephalography (EEG) to collect brain activity data — the electrical brain activity is recorded from electrodes placed on the scalp. Besides electrical activity, neural activity also produces other types of signals that can be used in a BCI. Mag-

3 2. Background netic fields can be recorded with magnetoencephalography (MEG), while brain metabolic activity, reflected in changes in blood flow, can be observed with positron emission tomography (PET), functional magnetic resonance imaging (fMRI), and optical imaging. Unfortu- nately, such alternative techniques require sophisticated devices that can be operated only in special facilities. Moreover, techniques for measuring blood flow have long latencies and thus are less appropriate for interaction [5].

2.2 EEG

An electroencephalograph is the recorded electrical activity generated by the brain. In general, EEG is obtained using electrodes placed on the scalp with (or without using dry electrodes) a conductive gel. In the brain, there are millions of neurons, each of which generates small electric voltage fields. Those electric voltage fields can be aggregated to create an electrical reading which electrodes on the scalp are able detect and record. The amplitude of an EEG signal typically ranges from about 1 uV to 100 uV in a normal adult, and it is approximately 10 to 20 mV when measured with subdural electrodes such as needle electrodes. [6].

2.3 BCI Approaches

There are several BCI paradigms that are used today. The difference between them is what control signals are obtained from the device. A suitable control signal has the following attributes: it can be precisely characterized for an individual, it can be readily modulated or translated to express the intention and it can be detected and tracked consistently and reliably. There are some approaches towards BCI, which are described below.

2.3.1 Recording ordinary brain activity The first approach is based on the multiple sensor EEG activities recorded in the course of ordinary brain activity. This approach is more comprehensive than others and does not require any particular

4 2. Background

stimulus. Nowadays there are many attractions in using the normal EEGs (or Spontaneous Signals (SSs)) for BCI. A BCI system of this kind generates a control signal at given intervals of time based on the classification of EEG patterns resulting from particular Mental Activity (MA).

2.3.2 Event Related Potentials Second approach based on Event Related Potentials (ERPs) that ap- pears in response to some specific stimulus. The most widely used ERP evoked potential (EP) is the P300 signal, which can be auditory, visual, or somatosensory. It has a latency of approximately 300 ms and is elicited by rare or significant stimuli and its amplitude is strongly related to the unpredictability of the stimulus [7]. Another type of Visual EP (VEP) is those for which the ERPs have a short latency, representing the exogenous response of the brain to a rapid visual stimulus. The ERPs can provide control when the BCI produces the appropriate stimuli. The information achieved through ERP extraction and measurement is not accurate enough for extraction of movement related features and they have vast variability in different subjects with various brain abnormalities and disabilities. More importantly, the subject has to wait for the relevant stimulus presentation [8].

2.3.3 Steady-State Visual Evoked Potentials Other approaches used for BCI is based on Steady-State Visual Evoked Potentials (SSVEP), which are natural responses for visual stimulations at specific frequencies. The SSVEP is characterized by an increase in EEG activity around the stimulus frequency. The advantage of SSVEP approach is that is has higher accuracy and higher information transfer rate (ITR) compared to other BCI approaches, such as P300. In addition, short/no training time and fewer EEG channels are required [9]

2.3.4 Motor Imagery Imagining a movement or performing an action mentally is known as Motor Imagery. The Motor imagery produces similar effects on the brain rhythm in the sensory-motor cortex as the real executed move-

5 2. Background ment [10]. The Mu rhythm power, which is associated with planning and execution of voluntary movements of the body [11], undergoes attenuation during the preparation and execution of movement. With an appropriate training and feedback, an individual can learn to control this mu power which could be used to produce the commands for BCI [12].

2.3.5 Slow Cortial Potential Shifts The latest approach is Slow Cortical Potential Shifts (SCPSs), which are shifts of cortical voltage lasting from a few hundred milliseconds up to several seconds. This can be achieved if the subjects are provided with feedback on the evolution of their SCP and if they are positively reinforced for correct responses [13].

2.4 BCI Systems

2.4.1 Wireless BCI systems for consumer use In the market of consumer-grade wireless BCI systems, there are many commercial companies such as Emotiv, Neurosky or OCZ technology. These companies have competitively released their own wireless BCI systems along with various applications related to gaming, utilities, and mental-state monitoring. In this section, the review for these wireless BCI devices for consumer use is presented.

Emotiv EPOC The device released by Emotiv is the EPOC headset. The EPOC headset (figure 2.1) [14] is a multi-channel wireless BCI system. This headset is equipped with 14 saline-based wet-contact resistive electrodes for measuring EEG, Electrooculogram (EOG), and facial Electromyogram (EMG). Additionally, the EPOC headset also has a 2-axis gyroscope for measuring the head rotation. Employing 2.4GHz wireless connectivity, this system provides wide accessibility for devices such as PCs, lap- tops, and smart phones. The package of the EPOC headset provides a bundle software which contains a suite of built-in signal processing algorithms for interpretation of EEG signals. The built-in algorithms

6 2. Background

Figure 2.1: Emotiv EPOC [15]

discern the user’s conscious intentions, emotional states, and facial expressions based on measured EEG, Electromyogram (EMG) and Electrooculogram (EOG) signals. Through this software, the users can interact with various applications related to virtual reality, game controlling, and brain state monitoring. These applications can be downloaded by accessing their web site.

Neural Impulse Actuator

OCZ technology, a PC component manufacturer such as solid-state drives (SSD) and power supplies, has released a game controller by utilizing an wireless BCI technology. The name of this controller is Neural Impulse Actuator (NIA) (figure 2.2) [16]. Using NIA, the users can control a PC game by translating facial expressions, eye movements, and concentrated brainwave activity, instead of using traditional input devices such as a keyboard and a mouse. This device now supports various PC games including shooting, role playing, virtual worlds, and racing.

Neurosky MindWave

Neurosky MindWave headset also belongs to this category. It is described more in detail in section 2.5 as it was used as the device for getting the data.

7 2. Background

Figure 2.2: Neural Impulse Actuator [17]

2.4.2 Wireless BCI systems for research uses

Wireless BCI systems have many advantages, such as freedom of user’s postures and convenient installation. Therefore, wireless BCI systems are useful in the research field as well. Some companies have been involved in research with universities and research institutes to develop wireless BCI systems for research uses. They include, but not limited to, Advanced Brain Monitoring, Quasar, Starlab and Guger technologies (G.tec). In this subsection, those wireless BCI systems are presented.

B-Alert X

Advanced Brain Monitoring has recently released the B-Alert X series wireless EEG systems [18] for mobile neurophysiological data acquisition and analysis. These systems include three models that have different numbers of channels, i.e., 4, 10, and 24. Among these models, the B-Alert X24 system (figure 2.3) is equipped with 24 channel electrodes for biopotential measurements, such as EEG, Electrocardiogram (ECG), Electromyogram (EMG), and Electrooculogram (EOG). This system measures and delivers real-time EEG signals via a Bluetooth connection. The system provides more than 8 hours of operation time and less than 10 minutes of installation time. Also, this system supports a variety of applications such as drowsiness, cognitive workload, and neurodynamics monitoring.

8 2. Background

Figure 2.3: B-Alert X [19]

Quasar BCI solution Quasar [20] has developed and released a wireless BCI solution. This solution includes Dry Sensor Interface (DSI) 10/20 (figure 2.4), wireless Data Acquisition (DAQ), and a suite of software of its own. DSI 10/20 is a wireless BCI headset which is equipped with up to 21 EEG sensors. The EEG sensors are dry electrodes and provide high input impedance good for measuring the high fidelity EEG signals. Wire- less DAQ is a peripheral device for signal transmission and onboard recording using a flash memory. QStreamer is a suite of software which contains data acquisition algorithms as well as different cognitive state classification algorithms. The classification algorithms estimate user’s mental states in terms of workload, engagement, and fatigue.

Enobio Enobio (figure 2.5) [22] is a wireless EEG acquisition device developed by Starlab. This device is a cap style with light weight feature (only 65g). It has multiple channels, supporting an option of 8, 20 or 32 channels in particular; each is equipped with a dry electrode. It can operate up to 16 hours long using a rechargeable Lithium Polymer battery. It is connected with a computer via a Bluetooth connection. A bundle software provides real time visualization of EEG signals such as power

9 2. Background

Figure 2.4: Quasar DSI 10/20 [21] spectrum and raw signal monitoring. This system has been applied to various applications associated with medical, neurofeedback, and cognitive state monitoring.

g.MOBIlab+

Guger technologies (G.tec) is a medical engineering company which provides comprehensive BCI solutions. This company has released a mobile biopotential acquisition system named as g.MOBIlab+ (figure 2.6) [24]. This system is available in two different modes: the 8 channel EEG acquisition mode and the multi-modal acquisition mode. In the multi-modal acquisition mode, this system can measure the EEG signals with other physiological signals such as Electrocardiogram (ECG), Electromyogram (EMG), and Electrooculogram (EOG). With a Bluetooth connection, it operates up to 100 hours using four AA batteries

10 2. Background

Figure 2.5: Enobio [23]

Figure 2.6: g.MOBIlab+ [24]

11 2. Background 2.5 NeuroSky MindWave

NeuroSky Mindwave is an EEG headset created by NeuroSky, Inc. launched in 2011, aiming on consumer market sector. It uses dry sensors, so no conductive gel is required for using the device (see figure 2.7 for a diagram of the headset). It measures brainwaves (Alpha, Beta, Delta, Theta) (see table 2.1 for explanation) and transfers it wirelessly via Bluetooth to consuming platform (a computer). It also offers output from NeuroSky proprietary eSense meter, which can calculate attention and meditation levels using the brainwaves measurements mentioned above [25]. This output was used in this thesis as the input source for the game. For both attention and meditation eSense, the value is reported on a scale of 1 to 100. In this scale, 5 intervals are established and one special value is present. The special value is 0, indicating that the device is unable to calculate attention or meditation level with a reasonable amount of reliability. The intervals are as follows: 1 to 20, 20 to 40, 40 to 60, 60 to 80 and 80 to 100. Those intervals are not used in this thesis as the game uses the whole range of the scale. Both attention and meditation levels are typically refreshed once a second. The term attention level is a value which indicates the intensity of user’s level of mental focus or attention. The more the user concentrate, the higher the value is. Distractions, wandering thoughts, lack of focus, or anxiety may lower the Attention meter levels. Meditation level is a value representing the level of mental calmness or relaxation of the user. Meditation is related to reduced activity by the active mental processes in the brain, so distractions, wandering thoughts or anxiety may lower the meditation level [26].

2.6 Related work

Even though the BCIs and their connection to computer games is a relatively new thing, there are already a plenty of interesting projects that are trying to connect various BCI devices with computer games using different approaches. For example, SSVEP was used in a simple 2D game, where the player used SSVEP to navigate a car on a track [29] (figure 2.8). The same approach was used in a maze game [30].

12 2. Background

Figure 2.7: MindWave diagram [27]

13 2. Background

Brainwave type Frequency range Mental states and conditions Delta 0.1 Hz to 3 Hz Deep, dreamless, non-REM sleep, unconscious Theta 4 Hz to 7 Hz Intuitive, recall, fantasy, imag- inary, dream Alpha 8 Hz to 12 Hz Relaxed, but not drowsy, tran- quil, conscious Low Beta 13 Hz to 15 Hz Formerly SMR, relaxed yet focused, integrated Midrange Beta 16 Hz to 20 Hz Thinking, aware of self and surroundings High Beta 21 hz to 30 Hz Alertness, agitation

Table 2.1: Frequencies Generated By Different Types of Activities in the Brain. [28]

A slightly different approach was presented in a tower defense game [31], where only one stimulus was used for determining if the player wants to use the highlighted game option or not (figure 2.9). MI approach was used in a tennis game, where the player controlled the movement (up, down or stay) of the character [32]. MI was also used in a modified pinball machine [33]. Although ERPis more commonly used in spellers, there is a speller based game called MindGame which is using ERP to move characters across a chess board [34]. An interesting approach was introduced in alphaWoW, a World of Warcraft modification in which neurofeedback is used for switching between elf and bear while playing a druid class. This is determined by user’s meditation level. Although the rest of the actions is still controlled via mouse and keyboard, the alphaWoW was a demonstration that BCI can be incorporated into the controls of a big and complex game.

14 2. Background

Figure 2.8: 2D car game using SSVEP [29]

Figure 2.9: Tower defense game controlled by SSVEP [31]

15 2. Background 2.7 Used technologies

2.7.1 Unity Unity is a game engine developed by Unity Technologies used for creating 2D and 3D video games. Unity offers multiple versions of the engine (plans), the most basic version called Personal Edition can be downloaded for free [35]. Unity offers three programming languages that can be used for creating games — C#, UnityScript (a modified version of JavaScript [36]) and Boo [37].

Figure 2.10: Personal edition of Unity.

2.7.2 Visual Studio Visual studio is an integrated development environment created by Microsoft for developing application for Android, iOS, Mac, Windows, web and cloud [38]. It also has support for Unity as a plug-in that can be installed — Visual Studio Tools for Unity [39]. With this tool, a developer is able to debug the application using Visual Studio, read the Unity documentation, use code snippets to speed up the writing of the code and much more [40].

16 2. Background

2.7.3 GIMP GIMP is a cross-platform open source image editor available on many operating systems (GNU Linux, Mac Os, Windows to name a few). The name GIMP is actually an acronym for GNU Image Manipulation Program [41]. Gimp offers wide range of features and capabilities that can be extended by plug-ins. A few examples of the features[42]:

∙ Full Alpha channel support

∙ Layers and channels

∙ Multiple undo/redo (limited only by disk space)

∙ Load, display, convert and save to many file formats

3 Initial project

As an initial idea, a different project was proposed. The basic principles were the same — to create a game which will be controlled by a BCI input and perform an analysis of the obtained data from a testing session with a group of users. In this chapter, the initial project is presented, including the technologies that were used, the input methods that were proposed, the design of the game and the issues that appeared during the process. It is also explained, why this project was not used in the end.

3.1 Used technologies

The device chosen for getting the EEG data was Enobio with 32 electrodes. This particular device was selected as it was affordable, available and provided the mobility as the connection between the device and the computer is wireless. For processing the EEG data, Open- VIBE software was used. OpenVIBE is free and open-source software devoted to the design, testing and use of brain-computer interfaces and can be used to acquire, filter, process, classify and visualize brain signals in real time [43]. The game was created in Unity game engine, code was written in Visual Studio and the models were created in Blender, which is an open-source 3D modeling software (the track), or downloaded from the Unity asset store (the car). For the connection between the OpenVIBE output and input used in the game, Unity Indie VRPN Adapter (UIVA) [44] created by Jia Wang and Robert W. Linde- man was used as OpenVIBE has already built in support for VRPN. UIVA supports multiple devices such as Microsoft Kinect or Nintendo WiiMote, but it does not provide functionality for OpenVIBE, so it was added by the author of the thesis as UIVA is an open-source project. This OpenVIBE to Unity interface was then used by others, e.g. [45].

3.2 BCI input method

When designing the game, it was agreed that the BCI input should mimic cursor movement (arrow) keys. That means that the system

19 3. Initial project

Figure 3.1: Input using both MI and SSVEP

should be able to provide input for up, down, left and right. As a first proposal, a mixture of MI and SSVEP was suggested. Input for left and right would be provided by MI, whereas up and down will be controlled with SSVEP as can be seen in figure 3.1. This approach, however, was discarded later on as it was agreed that mixing BCI approaches in one scenario is counterproductive. The user would need to be concentrated on SSVEP and MI simultaneously, which would lead to not optimal results. The decision was to remove MI and focus on SSVEP approach, so there would be 4 stimuli on the screen, as can be seen in figure 3.2. Each arrow represents one stimulus on the screen, so there would be input for up, down, left and right. Each of those arrows would be flashing with different frequencies as required by the SSVEP. There was also a consideration of adding diagonal inputs (see figure 3.3 for the purposes of the game, but it was more difficult to implement as the frequency options are limited and adding 4 more flashing objects to the corners of the screen would distract the user even more from what would be happening in the center of the screen.

20 3. Initial project

Figure 3.2: Input using SSVEP only with 4 directions

Figure 3.3: Input using SSVEP only with 8 directions

21 3. Initial project 3.3 Game description

The idea of the game is relatively simple — to have a racing game in which the player would control the car on a racing circuit. There will be no AI opponents and the goal of the game would be the time — that means how long will it take to finish the round. There will also be a line in the middle of the road. The goal of this line is to present to the player the ideal racing route and there is also a scoring element in this line — the further the car is from the line, the less points are given. The total score for the player would then consist of two elements — time of the race and the points that were given while driving. This system would reward the player in terms of both speed and accuracy, which is a motivation to control the car as best as the player can. The game prototype that was develop was fully playable. The game supported stereoscopic mode, so it was possible to play it in 3D, this mode was only optional and could be disabled. There was also implemented a system for adjusting the maximum speed of the car and maximal angle in which the car could steer as it was quite difficult to control the car, even with the keyboard. The keyboard input was captured via OpenVIBE and sent to the game using UIVA to test that the connection is working. It was achieved by attaching the UIVA server application to the OpenVIBE interface and integrating the UIVA client as an Dynamic Linked Library (DLL) file into the game to receive the data from the server. How the game prototype looked like can be seen in figure 3.4.

3.4 Issues

As described in previous section, there were some issues with the game prototype. First of all, it was very difficult to control the game. When testing the race, an ideal line was hard to follow and the steering and acceleration of the car were not ideal. This required implementing a system of adjusting the top speed of the car and the steering effi- ciency. When the values were set to a lower difficulty, so the top speed was very limited and there were restriction on the car steering, it got better, but the issue with the controlling the car was not removed. The testing was done using a keyboard input, which allowed the user to

22 3. Initial project

Figure 3.4: Car game prototype

combine multiple keys, so the car could go left and forward simultaneously. This was not possible if 4 direction SSVEP control setup was used, and the 8 direction setup was very chaotic on the screen, so a different approaches were proposed, such as replacing thecar with a hovercraft. The hovercraft would allow the user to rotate in- dependently from going forward or backward and, in theory, should simplify the game. Some basic testing was done to prove this, however, the hovercraft model was never created, as a different and a major issue was identified. The main issue, however, was not the game design or gameplay, but the the expected input source — BCI using SSVEP. It must be admitted that a lot of time and effort could have been saved if some basic BCI testing was done first, but it was not the case, so the game prototype was built before the BCI testing started. When following a basic SSVEP tutorial provided by OpenVIBE [46], in which the player controls a ship which can rotate clockwise, anticlockwise or shoot, and which is a part of the OpenVIBE solution, it was discovered that it is not possible to get meaningful output from the application. The reactions of the tutorial game were unpredictable, the ship was rotating and

23 3. Initial project shooting randomly no matter what the player was focusing on. The root cause of this faulty behavior was not discovered, though the most probable explanation is that either the Enobio device was not working properly or the room in which the testing was done was producing too much electrical noise that could have led to the biased input data. It is also possible that some step from the tutorial was misinterpreted, but this possible issue was minimized by repeating the tutorial multiple times in different sessions. The discovery of such an impact on the project meant that the device for getting the EEG data should be replaced and the game design adjusted accordingly, as the interface for controlling the game is tightly connected to the device used and the data the device is able to provide. One suggestion for the EEG device was NeuroSky MindWave headset, which does not provide neither SSVEP nor MI support and rather produces its own neurofeedback output, which can be used for creating an interface for controlling a simple computer game. After the experience with the Enobio headset, MindWave device was tested first using application provided by NeuroSky that is used for demonstration of the hardware. The testing was successful, the device was connected to the computer with no issues (plug and play approach), no initial training was needed for using the device, and the design of the headset allowed the user to attach it to the head and make it working in a minute or two, whereas the setup of Enobio would take 10 minutes or more. On the other hand, the device is very limited, compared to Enobio, but the fact that it is easy to use and, most importantly, is working correctly, was the main factor why the MindWave headset was chosen as an input source.

24 4 Methodology

4.1 Game description

4.1.1 Requirements One goal of the thesis was to design and create a game which would use NeuroSky MindWave headset for controlling the application, which would then be used in the experiment stage. During the initial brainstorming for game ideas, following requirements were established:

∙ Controls As the main input source, the NeuroSky MindWave headset should be used. It offers two separate inputs — attention and meditation level. The game should implement both those inputs in a reasonable way, so it would feel as natural as possible to control the game with those. Other input methods such as mouse or keyboard can be used as well, but not as a primary input source.

∙ Feedback The application should provide visual feedback to the player, so he can check how well is he doing in the game. This can be further separated into two main areas: – How well is he doing in general. – How well is he doing in the terms of controlling his attention or meditation.

∙ Competitiveness The player should be motivated to perform as good as he pos- sibly can, but not in a way that the stress factor could affect his performance. A non-direct competition is preferred, so the player can compare his results with other players, but should not be facing other players directly.

25 4. Methodology

∙ Easy to understand game mechanics The game mechanics should be easy to grasp, so the player is not concerned about what is the goal of the game and rather focus on achieving the best results possible. Preferably, the game should adopt some well known principles or resemble some popular sport or activity. The main goal here is to have a game that is fun to play without any complicated tutorial session.

∙ Easy, yet challenging gameplay The gameplay should be pretty straightforward without any unnecessary complications. There should not be many different scenarios, one for controlling attention and one for meditation should be enough. All the activities should be challenging though, so the player will not get bored.

∙ Length of the game Time required for completing the game should be reasonable. As we want rather easy gameplay, tasks which would take longer than 10 minutes can become very repetitive and frustrating. Ideally, it should be possible to finish the game in less than half of this time in most cases.

∙ Access to raw input data It should be possible to get the input data, ideally connected to other game variable such as time or score after the player finishes the game. Those logs should uniquely represent the attempt and could then be used for its further analysis. Alongside the questionnaires, the log files can be taken as another source of data.

4.1.2 Game design Having done the requirements, the next step was to design a game which would follow the requirements mentioned in section 4.1.1. It was decided to create an application that would resemble biathlon. Biathlon is a sport that combines cross-country skiing, or other forms of movement, with rifle shooting[47]. The main idea behind this choice

26 4. Methodology was that the game based on biathlon would use attention level for the movement part of the game and the meditation level for shooting. It was decided that the game should be in 2D as it would provide less distraction for the player without ruining the gameplay experience. The theme of the game was chosen based on available assets on Unity Asset Store and personal preferences. As the game takes place in the forest, the game was named Foresthlon. The gameplay consists of two parts that repeats 4 times. In the first part, the player runs forward to the shooting range as quickly as possible. When running, attention level is the input for the game as it sets the player’s speed. In the second part of the game, the player is at the shooting range and tries to hit the target. If he misses, he repeats the shooting, otherwise he continues in the game by running to the next shooting range. The player has unlimited ammo, so he can repeat the shooting as many times till he hits the target. There is, however, a 2 second delay between shooting and next action. The distance between shooting ranges was set to 100 in-game meters. That means that the total distance the player needs to run is 400 meters and he shoots at 4 shooting ranges. After the 4th successful shot, the game ends. The main goal of the game is to run and hit all the targets as quickly as possible as the main indicator of how successful the attempt is the time the player was able to finish the game. Number of attempts needed on each shooting range to pass is also stored, but it is not taken into account when comparing the result with other players. As the running part of the game is pretty straightforward to include the attention level as the source of the input, the only big game design choice remained how exactly should the shooting part implement the meditation level. The initial idea was to have a crosshair that would have variable size. If the meditation level is low, then the crosshair would be huge so it would be really hard to aim for the target and the higher the meditation level is, the smaller the crosshair gets, as is depicted in figure 4.1. This idea was discarded however, as itwould require rather complicated mouse interaction that could negatively affect player’s meditation level, and also some degree of randomness as the shot could hit any point inside the crosshair.

27 4. Methodology

Figure 4.1: Initial idea of aiming in the shooting part of the game — dynamic crosshair size. On the left, low meditation level crosshair is present. On the right, a small crosshair is displayed which is connected to high meditation level.

A different mechanism was used instead of the crosshair. Atthe shooting range, a box is displayed above the target which is separated into three parts and there is a moving circle inside this box. The middle part has a green color and the player is supposed to hit a defined key when the circle is inside the green area to make a successful shot. Movement speed of the circle is set according to player’s meditation level, so the more is the player meditated, the slower the circle moves in the box. Figure 4.2 shows how this mechanism is implemented to the game. After finishing the game, the player can see his final time andhis position on the leaderboard. Then, he is required to fill in his name, so his attempt can be saved into the leaderboard and the log file with his time, number of attempts needed and input values throughout the run is stored on the disk.

4.2 Implementation

Following section describes the implementation of the game. Game engine Unity was used for creating the game and the code was written in C# programming language using Microsoft Visual Studio 2015. For editing and creating the graphics elements, GIMP was used.

28 4. Methodology

Figure 4.2: Mechanism for controlling the shooting — actual game screenshot.

The core of the application is GameController class, which is re- sponsible for the game flow and connecting multiple aspects ofthe game together. An enum called GameStateEnum is used to control what is currently going on in the game and it has these options: ∙ PreStart — Main menu screen when the game starts. ∙ Running — The player is currently running, attention level is used as a source for input. ∙ Shooting — The player is currently at the shooting range, meditation level is used as a source for input. ∙ Pause — The game is currently paused, menu is displayed. ∙ End — The player finished the game and the final screen isdis- played.

4.2.1 Used assets The following table 4.1 contains list of external assets and other re- sources used in the game.

29 4. Methodology

Asset name Usage Source 2D Game Starter Assets Background [48] 2D Archers Sprites Archer model and anima- [49] tions ThinkGear Unity3D Pack- Connection between Neu- [50] age roSky MindWave headset and Unity Free Archery Target Board Shooting board sprite [51] Clip Art

Table 4.1: Used assets

4.2.2 Controls

The data from the NeuroSky MindWave headset is collected by the TGCConnectionController. It gets both attention and meditation level from the device and raises events to MovementInputController and ShootingInputController where the input is processed for the purposes of the game, but only for the current part of the game. That means that if the player is in the shooting part, only meditation level is processed and the same applies for the attention level while running. When in menu, no input is collected in the controllers. That allows to immediately store the input value for logging purposes. Those three classes can be, with some modification to make them more generic, rather than specialized entities, used as an extensible interface which can be used in any game or program that should be controlled using Neurosky Mindwave headset. For the purposes of the Foresthlon application, it was decided to include this interface in the solution, but it can be extracted into separate DLL which can be used in other applications. UML diagram shown in figure 4.3 describes how it can be used. The core class is the TGCConnectionController, which gets the data from the MindWave device. In the constructor of the class, the basic setup of the parameter (how frequently would the data be up- dated, which inputs would be used) is done, as well as the injec- tion of IAttentionInputController, IMeditationInputController, IRawDataInputController and IInputLogger instances. The logger

30 4. Methodology

Figure 4.3: UML diagram depicting the usage of the NeuroSky Mind- Wave controlling interface in an application

31 4. Methodology is always provided by the consuming application, the controllers can be the basic ones, or inherited classes from the base classes (AttentionInputController, MeditationInputController and RawDataInputController). TGCConnectionController is triggering events that are consumed by the input controllers, that can then be used within the application itself for controlling the application and logging the data.

4.2.3 Running The running part of the game is simulated by emitting the background particles which, combined with the archer’s running animation, makes the illusion of the moving archer. The speed of the archer is basically the speed at which the particles are emitted and the speed of the animation of the archer. A screen shot from the running part of the game is at figure 4.4. The movement speed is calculated from the input and its range is between 0 to 7 ingame meters per second. Once the player reaches the distance ShootingRangeEmitDistance (76.7 meters after start of the game and after each shooting), a shooting line is emitted and the emission of the trees is stopped so the shooting range can be spawned. Once the player is at the line, the shooting part of the game begins.

4.2.4 Shooting As the player approaches the line indicating the shooting part of the game, the box with moving circle is displayed above the target. The circle is the moving back and forth in the box at speed that is calculated from the meditation level by the following formula: circleMovementSpeedIncrement = 10 + ((100 − inputSpeed) * 4.4) (4.1) where inputspeed is a float variable that is in range between 0 and 100. circleMovementSpeedIncrement is then multiplied by Time.deltaTime [52] and is either added or subtracted from the current circle movement speed. The circle position inside the box is a float value between 0 and 100. To hit the target, the player has to press the Space key to shoot when the circle is in the green area. When the key is pressed, the current circle

32 4. Methodology

Figure 4.4: Screen shot of the running part of the game position is compared to the green interval (<40, 60>) and then either "HIT" (as shown in figure 4.5) or "MISS" text is displayed in the middle of the target. After the successful shot is made, number of attempts needed to pass the shooting range is stored in the GameController and the player proceeds further in the game.

4.2.5 User interface In-game UI The purpose of the in-game user interface is to provide feedback to the player on how well is he doing in the game. In the top left corner of the screen, information about the distance and the time of the run is provided. The distance tells the player how far is the player from the shooting range or the finish. A green bar is present in the top center of the screen. This bar represents current attention or meditation level of the player. Its purpose is to provide clearer way how the player can check his current performance rather than relying on his feeling of how fast the archer runs or the circle in the shooting bar moves. A label above the input bar states if the current input is attention or meditation.

33 4. Methodology

Figure 4.5: Screen shot of the shooting part of the game — successful attempt

In the top right corner, there is an icon indicating the connection to the NeuroSky MindWave headset. If the icon is green, the connection is ok. Yellow icon indicates that there is a problem with the connection and red means that the headset is not connected. This icon is connected to the main camera object so it can be visible on every screen of the game.

Main game menu has three purposes. One of them is to provide player a way how he can start or resume the game, restart the game if needed, check the leaderboard and quit the game. There is also a tutorial text stating the following: How to play: Press Space to start the game, and once you are at the shooting range, press space to shoot! Movement speed is adjusted by your attention level, speed of the shooting slider is adjusted by your meditation level. Have fun! Additionally, there are two buttons in the bottom left corner of the menu. Those are the buttons for reconnecting or disconnecting the headset. In most cases though, those buttons are not needed and are there just in case of an error getting the input data

34 4. Methodology

from the headset, so the connection can be reset. How the menu looks like is shown in figure 4.6.

Figure 4.6: Screen shot of the game menu

Leaderboard

Leaderboard is a screen that contains all the runs ordered by time descending. This allows the player to check the attempts of other players. Information that can be seen on the leaderboard consists of player’s position, name, time and attempts needed at each shooting range (figure 4.7). The leaderboard is stored locally on the disk. For this purpose, Application.persistentDataPath field [53] is used to get the folder name where the leaderboard is stored. In case no leaderboard is found, a new one is created. This file contains xml data in this format:

35 4. Methodology

Figure 4.7: Screen shot of the leaderboard

Final screen

When the player finishes the game, final screen is displayed sothe player can see how successful he was (figure 4.8). He is asked to enter his name and once he does that, he has access to menu, leaderboard or has the option to quit the game. After he submits his name, a new entry in the leaderboard is created as well as log file with the details of his run.

36 4. Methodology

Figure 4.8: Screen shot of the game UI after finishing the run

4.2.6 Logging

Once the player finishes the run and submits his name, a log file inthe xml format with the information about the attempt is saved to the Logs folder in the game directory. The file contains overall characteristics of the run (name, attempts and time) and all the input values with the timestamp. Those entries are in the following format:

4.3 Future work

The game as it is right now offers complete experience and is fully playable, there is however some functionality not implemented which could make the game even better. Following improvements can be split into two groups — improvements of the game itself (subsections 4.3.1, 4.3.2, 4.3.3 and 4.3.4) and extending the game so it can be reached by a broader audience (subsection 4.3.5)

37 4. Methodology

4.3.1 Sounds There are no sounds present in the current state of the game. This was a game design feature — in-game sounds can affect the player’s performance by distracting him from the ultimate goal of the experiment — controlling his attention/meditation level. However, adding sounds would definitely improve user’s experience and overall game quality. The following sounds ca be added in the future:

∙ Sounds of the environment (owls hooting, frogs croaking etc.),

∙ Background music,

∙ Archer’s footsteps,

∙ Bow sounds,

∙ Sound of missed shot or hit,

∙ Sound of clicked buttons.

4.3.2 Additional game modes For the purpose of the experiment, only one game mode is present. A proper extension to the game can be new game modes and modifiers, so the player has more options what to do in the game. A good example can be a limited ammo game mode, where the player will have limited quiver capacity and will either not be able to finish the game if the ammo is depleted, or will have to run additional distance to refill the quiver. Another game mode idea can include control stations, where the player would be forced to reach a certain attention level to proceed in the run.

4.3.3 New environments and archer skins This improvement would only affect the visuals (and sounds if present) of the game. For example, a winter theme would add more of a biathlon feeling to the game and the archer can be equipped with cross-country skis with additional animations.

38 4. Methodology

4.3.4 Online leaderboard Current implementation only supports a leaderboard that is stored locally on the disk. To make the game more enjoyable, a global leaderboard that is located in the cloud and can be accessed by every game instance across multiple computers (or other platforms if applica- ble) can be added so the player can compete with other players, not only those playing the game on the same machine. This can be either achieved by writing the whole functionality from scratch, or existing solution can be integrated with the game. For example, there is an asset called Very Simple Leaderboard which can be downloaded from the Unity Asset Store1.

4.3.5 Porting to other platforms The game was deployed on Microsoft Windows platform only. How- ever, Unity supports deployment on more than 25 platforms [54] so it can be fairly easy to port it to other systems as well. As we used NeuroSky MindWave headset, only a port to Mac operating system is available [55]. The game can be further ported to iOS and Android platforms as well but NeuroSky MindWave Mobile headset is required as NeuroSky MindWave headset cannot be connected to a smartphone running one of these operating systems.

1. https://www.assetstore.unity3d.com/en/#!/content/59227, accessed 2017-05-05

5 Experiment

5.1 Testing group

The testing group consisted of 15 males and 15 females with no previous experience with brain computer interfaces.

5.2 Stages of the experiment

The testing procedure consisted of these steps:

∙ Filling consent form Each participant was given a consent form to fill before the experiment started.

∙ Setting up the experiment NeuroSky MindWave headset was attached to the participant’s head and it was made sure that the connection between the device and the game is stable.

∙ Giving basic instructions A basic explanation what is expected from the participant was given. This consisted of explaining the game, introducing the headset and how the game is controlled. Then, the participant had an opportunity to see the leaderboard so he had a better understanding of time required to beat the game by other participants (except for the first participant, who did not have the opportunity to see previous results). The participants were not allowed to do a training run, as the experiment was designed to observe how quickly can the player adapt to a new controlling paradigm.

∙ Playing the game After giving the instructions, the participant played the game. During this phase, the interaction between the test subject and the observer was reduced to minimum so the player was not

41 5. Experiment

distracted. The observer’s role was primarily to check the connection status of the device. The game was run on a notebook (5.1), so the display was relatively small, but that did not affect the result, as one participant had a chance to play again using external display, but the this run was not stored. The gameplay demostration can be seen at this link.

Figure 5.1: Female participant during the experiment

∙ Checking the results After finishing the game, player’s time and his position on the leaderboard was checked. Then, the name was entered and his attempt was stored to the log file and the leaderboard.

∙ Filling the questionnaires As the participant finished the practical part of the experiment, he was handed a set of questionnaires to fill. Those questionnaires are described in chapter 6. During this stage, the observer

42 5. Experiment

was present to help with explaining the questions in the questionnaires, if the participant was unsure about the meaning of the question.

5.3 Issues during the experiment

During the testing, the biggest issue was with the connection between the headset and the computer. When the connection was set, it was very stable, but it happened a few times that the device was not recog- nized, so it had to be restarted. When this did not help, the computer was restarted as well and then the device connected. A minor issue was battery consumption, the device needed new batteries after 7 or 8 sessions on average, but as it was indicated by the device that battery is low, it never happened that the experiment had to be stopped because of the battery replacement. Only one participant was not able to take part in the experiment, because the headset was constantly falling from her head and the connection between the headset and the game could not be established. Another participant had to restart the game, because the connection was lost during the game. This was caused because of the shape of the headset, which did not fit well on all the participants and the adjusting options of the headset are very limited. It was sometimes difficult to find a proper positioning of the headset so the connection would be stable, but apart from those two cases, all the test subjects were able to connect and finish the game in one run.

6 Result Analysis

6.1 Procedure

NASA Task Load Index (TLX), a personal information sheet, experience questionnaire and two open questions were one part of the source of the data presented in this section. Logs generated by the game were also used for the analysis. The main focus of the analysis was on the difference between how males and females perceive the experiment and how successful they were in the game.

6.2 Demographics

15 males and 15 females took part in the experiment. They were asked to provide their age in intervals (18-25, 26-33, 34-41, 42-49 and over 50), their daily use of a computer on a scale from 1 to 5, current status (student or employed) and their highest achieved qualification (High school, BSc, MSc). Tables 6.1, 6.2, 6.3 and 6.4 represents the data.

Gender 18-25 26-33 34-41 42-49 50+ Male 7 8 0 0 0 Female 8 4 1 1 1

Table 6.1: Age of the participants according to gender

Gender 1 2 3 4 5 Male 0 0 0 2 13 Female 0 0 6 4 5

Table 6.2: Daily use of a computer according to gender

45 6. Result Analysis

Gender Student Employed Male 6 12 Female 8 7

Table 6.3: Occupation of the participants according to gender

Gender High School BSc MSc Male 5 4 6 Female 7 3 5

Table 6.4: Highest qualification achieved according to gender

6.3 NASA Task Load Index

NASA Task Load Index (TLX) is a multi-dimensional scale used for collecting the workload estimates from the testing subjects either while they are performing the given task or after the task was completed [56]. The questionnaire consists of 6 questions (see table 6.5 for the questions, their descriptions and scales) rated on a scale from 1 to 21. As the chart 6.1 suggests, the results of the NASA TLX do not differ much between the genders. Mental demand is almost the same for both males and females. Females felt that the physical demand was a little bit higher, opposed to temporal demand, where the males stated that the demand was higher for them. In terms of performance, there was no significant difference. Females also felt that they had to put more effort and were more frustrated by the task.

46 6. Result Analysis

Question Description Scale Mental Demand How mentally de- Very Low — Very High manding was the task? Physical Demand How physically de- Very Low — Very High manding was the task? Temporal Demand How hurried or Very Low — Very High rushed was the pace of the task? Performance How successful Perfect — Failure were you in accom- plishing what you were asked to do? Effort How hard did you Very Low — Very High have to work to ac- complish your level of performance? Frustration How insecure, dis- Very Low — Very High couraged, irritated, stressed, and an- noyed were you?

Table 6.5: NASA TLX Questions

47 6. Result Analysis

Figure 6.1: Chart showing the results of the NASA TLX questionnaire according to gender

6.4 Experience questionnaire

After filling the NASA TLX questionnaire, the participants were handed an experience questionnaire which consisted of 10 questions with a scale of 1 to 7. The questions are shown with the results of the questionnaire on chart 6.2. In terms of how easy did the participants find to control their attention or meditation level during the game, male participant tend to rate them better than females, as well as their feeling of having control of the archer. For performing a sport or an activity including active movement, both testing groups had the same results. All of the participants were aware of the sport of biathlon, so the game mechanics were easy to grasp. Males also stated that the mechanism which controlled movement and shooting was more natural to them. In terms of how involved the participants were with the experience, both testing groups stated that they were rather engrossed to the experience then not involved at all. Females experienced rather longer

48 6. Result Analysis

delay between their thoughts and the expected outcome. Most of the males stated that the adjustment to the experience took less than a minute, while for the females it took longer to adapt. Males also were more proficient in moving and interacting with the application at the end of the experiment. As for the visual display quality and the possible distraction from performing the task, females were less distracted as opposed to males. Males also rated them better when evaluating how well could they concentrate on actually playing the game rather than on controlling their attention and meditation level, but females rated themselves rather good in this aspect.

6.5 Open questions

The participants were asked to explain what they did to control their attention and meditation level. 17 responded that they were focusing on the game screen for controlling their attention level, usually just on one exact point (the archer’s legs was the dominant point of interest — 6 participants explicitly stated that they were focused on the archer’s legs while running). As for controlling the meditation level, most of the participants tried to calm down, control their breathing and not thinking about anything specific. Watching the green area above the shooting board also helped the participants to control their meditation level. A few participants stated that the game did not reflected their thoughts. Some of the respondent’s answers are listed below. "Before start I tried to control myself to calm down and focus only on the game. Then I started and I was thinking about one thing only — to beat the best time. During aiming I was trying to calm down." "At first, I was trying to control archer with my view. As I went further the control of the archer became easier when I realized it is all about focus, not about taking a look at the objects. The aiming was precisely about focus, really good experience to see how computer can reflex human brain activity." "I tried to focus on one point in the game during movement, but it wasn’t as successful as I expected. Sometimes it seemed that the speed is set randomly. I was more successful during the shooting when I saw concrete results of my effort." "When I was running I focused on a piece of grass. When I controlled myself it got worse, more focus = worse result. As for the meditation level,

49 6. Result Analysis

Figure 6.2: Chart showing the results of the experience questionnaire according to gender

50 6. Result Analysis

I tried to focus on nothing, just breathing and it went slower. Anything on my mind = it went faster." The participants also had the chance to express any additional thoughts and comments about the experiment. Not all respondents took this opportunity, but some valuable feedback was collected. One respondent was complaining about the connection indicator: "I think the connection indicator on top right part of the screen sometimes makes it hard to concentrate. My opinion is that it could be4 hidden when the connection is fine and should only display when there is a problem (the game can be paused as well). And the attendee should not be aware of that so doesn’t focus on that, it should be the researchers responsibility." One participant was not happy with the experience as the attention and meditation levels were not reflecting her effort: "It was really frustrating. When I wanted to move it didn’t work. When I wanted to calm down it really did not work." Apart from those two comments, all the other participants were excited with the experience and would like to try it again. "I liked the experience, especially the shooting part. I would like to expand the game to be longer." "Game was pretty easy to control and it was intuitive to play. Graphics looks good, controlling was weird because I have not much experience with mind controlling and similar stuff." "I am not a gamer, but it was interesting experience."

6.6 Game results

For the purpose of the analysis of the game logs, a simple application was developed. The program loads the log files from the disk and calculates the mean and the median of both attention and meditation levels of the player as well as the time spent in game using each input source. Those calculations are then saved to an XML file for further processing. As the chart 6.3 suggests, males overall had better results compared to females. The same result is also noticeable when we split the time into two parts — time spent running and time spent shooting (chart 6.4). On average, the males spent 20 seconds less in the running part of the game and 30 seconds less in the shooting part. Most of the players

51 6. Result Analysis

Figure 6.3: Time required for finishing the game according to gender spent more time running, only 7 participants spent more time in the shooting part (3 males and 4 females — chart 6.5. Number of attempts chart 6.6 provides the comparison of attempts needed on each shooting range and in total between males and females. The crucial, quite surprisingly, was the third shooting — the players needed more attempts to pass than on any shooting. The results reflects the times needed to finish the game — males needed less attempts then females. From observation, this can be linked to the playstyle — females had the tendency to wait longer before taking the shot, which often resulted in decreased meditation level as they become more excited when they actually wanted to shoot. Males tend to shoot even if the circle was moving fast. The meditation levels do not differ much between the genders as shown in chart 6.8. This does not apply for the attention level, as males did better in this matter (chart 6.7.

52 6. Result Analysis

Figure 6.4: Time spent using attention/meditation according to gender

Figure 6.5: Time required for finishing the game and time spent using attention/meditation of each participant

53 6. Result Analysis

Figure 6.6: Shooting attempts according to gender

Figure 6.7: Attention levels according to gender

54 6. Result Analysis

Figure 6.8: Meditation levels according to gender

7 Conclusion

This thesis presents a study of a simple game application controlled by BCI device. The game was designed such as it would provide a pleasant user experience, easy to grasp gameplay and a competitive factor. According to the feedback received both written and spoken, the participants of the testing were really interested by this novel approach of controlling the game and the application itself was classified as fun to play experience which would the testers played again if they had the chance. They also stated that the device chosen was appropriate as it does not require the use of the gel and no previous training. Comparing the success of the game results between males and females, better outcomes were identified in the group of male participants. This can relate to the fact that males were more focused on the actual game result (beating the best time) rather than the way how the game was controlled as well as taking the risk in the shooting parts of the game — not waiting for the ideal speed of the moving circle. The fact that males play computer games more than females could have affected the results as they could have more experience with playing games similar to the one presented in this thesis or have better grasp of what was required for finishing the game [57]. The initial idea of using Enobio device and SSVEP for controlling the game failed, as the output from the device was not deterministic. Instead of Enobio, NeuroSky MindWave hardware was used, which is not so complex and the presented controlling interface is not as rich as it could, if more sophisticated solution was used. The reasons for why the original approach failed remained unclear, so there is a space for having another look on this approach and utilizing its benefits. The presented controlling interface is not very novel, but it is working and it simplifies the process of creation games controlled by MindWave headset, so the creator can focus more on the game itself, rather than integration with the BCI. Future improvements of the game were mentioned in subsection 4.3. For the testing part, more runs in a single session can be done by one player to see if more attempts will result in better times. Also, expanding the testing group with more participants and analyzing different demographic criteria such as age and daily use of computer

57 7. Conclusion can be done as a further work. Using different BCI device and comparing the results and user experience with NeuroSky Mindwave headset is another path that can be followed.

58 Bibliography

1. HE, Bin. Neural engineering. 1st ed. New York: Kluwer Academic/Plenum, 2005. ISBN 978-0-306-48609-8. 2. GRAIMANN, Bernhard; ALLISON, Brendan; PFURTSCHELLER, Gert (eds.). Brain-Computer Interfaces: Revolutionizing Human-Computer In- teraction. 1st ed. Springer Science & Business Media, 2010. ISBN 978-3-642-02091-9. 3. TSIHRINTZIS, George A.; JAIN, Lakhmi C. (eds.). Multimedia services in intelligent environments integrated systems. 1st ed. Berlin: Springer, 2010. ISBN 978-3-642-13396-1. 4. ANUPAMA, H. S.; CAUVERY, N. K.; LINGARAJU, G. M. Brain Com- puter Interface and its Types - A Study. International Journal of Ad- vances in Engineering & Technology. 2012, vol. May. ISSN 2231-1963. 5. MILLAN, Jose del R.; FERREZ, Pierre W.; BUTTFIELD, Anna. Non In- vasive Brain-Machine Interfaces [online]. 2005 [visited on 2017-05-01]. Available from: http://boo.sourceforge.net/BooManifesto.pdf. 6. Brain Wave Signal (EEG) of NeuroSky, Inc. [online]. 2009 [visited on 2017-05-10]. Available from: http : / / www . frontiernerds . com / files/neurosky-vs-medical-eeg.pdf. 7. DONCHIN, E.; SPENCER, K. M.; WIJESINGHE, R. The mental pros- thesis: assessing the speed of a P300-based brain-computer interface. IEEE Trans Rehabil Eng. 2000, vol. 8. 8. BAYLISS, Jessica D.; BALLARD, Dana H. A Flexible Brain-Computer Interface. 2017. 9. FAZEL-REZAI, Reza (ed.). Brain-Computer Interface Systems - Recent Progress and Future Prospects [online]. 1st ed. InTech, 2013 [visited on 2017-10-02]. ISBN 978-953-51-1134-4. Available from: http://www. intechopen.com/books/brain- computer- interface- systems- recent-progress-and-future-prospects/a-review-of-p300- ssvep- and- hybrid- p300- ssvep- brain- computer- interface- systems.

59 BIBLIOGRAPHY

10. HEMA, C.; PAULRAJ, M.; YAACOB, S.; ADOM, A.; NAGRAJAN, R. An Analysis of the Effect of EEG Frequency Bands on the Classification of Motor Imagery Signals. 2010. 11. JASPER, H.; PENFIELD, W. Electrocorticograms in man: effect of the voluntary movement upon the electrical activity of the precentral gyrus. 1949. 12. PFURTSCHELLER, G.; NEUPER, C. Motor Imagery and Direct Brain- Computer Communication. 2001. 13. KUEBLER, A.; KOTCHOUBEY, B.; SALZMANN, H. P.; GHANAYIM, N.; PERELMOUTER, J.; HOMBERG, V. Self-regulation of slow cortical potentials in completely paralyzed human patients. 1998. 14. Emotiv EPOC Neuroheadset [online] [visited on 2017-10-10]. Available from: http : / / emotiv . com / store / hardware / epoc - bci / epoc - neuroheadset/. 15. Emotiv EPOC - image [online] [visited on 2017-10-10]. Available from: https://www.emotiv.com/epoc/. 16. Neural Impulse Actuator [online] [visited on 2017-10-10]. Available from: https://niagamecontroller.weebly.com. 17. Neural Impulse Actuator - image [online] [visited on 2017-10-10]. Avail- able from: http://newlaunches.com/archives/neural_impulse_ actuator_gives_you_mind_power_over_pc_games.php. 18. Advanced Brain Monitoring - B-Alert X [online] [visited on 2017-10-10]. Available from: http://advancedbrainmonitoring.com/neurotechnology/. 19. Advanced Brain Monitoring - B-Alert X - image [online] [visited on 2017-10-10]. Available from: http : / / www . advancedmedicalequipment . com / balertx24.html. 20. Quasar BCI solution [online] [visited on 2017-10-10]. Available from: http://www.quasarusa.com/. 21. Quasar BCI solution - image [online] [visited on 2017-10-10]. Available from: http://www.quasarusa.com/products_dsi.htm. 22. Enobio [online] [visited on 2017-10-10]. Available from: http://www. neuroelectrics.com/products/enobio/.

60 BIBLIOGRAPHY

23. Enobio - image [online] [visited on 2017-10-10]. Available from: http: //www.neuroelectrics.com/image/555x557xc/products_main/ Enobio_01.jpg. 24. g.MOBIlab+ [online] [visited on 2017-10-10]. Available from: http:// www.gtec.at/Products/Hardware-and-Accessories/g.MOBIlab- Specs-Features. 25. Technical Specs - NeuroSky MindWave Mobile [online] [visited on 2017-05-10]. Available from: https://store.neurosky.com/pages/mindwave. 26. NeuroSky - eSense Meters [online]. 2014 [visited on 2017-05-10]. Avail- able from: http://developer.neurosky.com/docs/doku.php?id= esenses_tm. 27. MindWave Diagram [online]. 2011 [visited on 2017-05-10]. Available from: http://support.neurosky.com/kb/mindwave/mindwave- diagram. 28. SALVEKAR, Devashish; NAIR, Amrita; BRIGHT, Dany; BHISIKAR, S. A. Mind Controlled Robotic Arm [online]. 2015 [visited on 2017-05-10]. Available from: http : / / www . iosrjournals . org / iosr - jece / papers/NCIEST/Volume%202/8.%2036-44.pdf. 29. MARTINEZ, P.; BAKARDJIAN, H.; CICHOCKI, A. Fully online mul- ticommand brain-computer interface with visual neurofeedback using SSVEP paradigm. Intell. Neuroscience. 2007, vol. 2007. 30. CHUMERIN, N.; MANYAKOV, N.; VAN VLIET, M.; ROBBEN, A.; COMBAZ, A.; VANHULLE, M. Steady state visual evoked potential- based computer gaming on a consumer-grade EEG device. IEEE Trans. on Computational Intelligence and AI in Games. 2012, vol. 99. 31. VAN VLIET, M.; MANYAKOV, N.; CHUMERIN, N.; ROBBEN, A.; COMBAZ, A.; VAN HULLE, M. Designing a brain-computer interface controlled video-game using consumer grade EEG hardware. Biosignals and Biorobotics Conf. 2012. 32. LOPETEGUI, E.; ZAPIRAIN, B.; MENDEZ, A. Tennis computer game with non-invasive bci. Int. Conf. on Computer Games. 2005. 33. M., KRAUDELAT; GRZESKA, K.; SAGEBAUM, M.; BLANKERZ, B.; VIDAURRE, C; MULLER, K.; SCHRODER, M. Playing Pinball with non-invasive BCI. Advances in Neural Information Processing Systems. 2009, vol. 21.

61 BIBLIOGRAPHY

34. FINKE, A.; LENHARDT, A.; RITTER, H. The MindGame: a P300-based brain-computer interface game. Neural Netw, 2009, vol. 9. 35. Welcome to Unity - Choose a plan that’s right for you [online] [visited on 2017-05-08]. Available from: https://store.unity.com/. 36. UnityScript versus JavaScript [online] [visited on 2017-05-08]. Avail- able from: http://wiki.unity3d.com/index.php/UnityScript_ versus_JavaScript. 37. OLIVIERA, Rodrigo Barreto de. NASA-TASK LOAD INDEX (NASA- TLX); 20 YEARS LATER [online]. 2006 [visited on 2017-05-18]. Avail- able from: https://humansystems.arc.nasa.gov/groups/TLX/ downloads/HFES_2006_Paper.pdf. 38. Visual Studio IDE [online] [visited on 2017-05-07]. Available from: https://www.visualstudio.com. 39. Visual Studio Tools for Unity [online] [visited on 2017-05-07]. Avail- able from: https : / / marketplace . visualstudio . com / items ? itemName=SebastienLebreton.VisualStudio2015ToolsforUnity. 40. Overview of Visual Studio Tools for Unity [online] [visited on 2017-05-08]. Available from: https://docs.microsoft.com/en-us/visualstudio/ cross - platform / overview - of - visual - studio - tools - for - unity. 41. About GIMP [online] [visited on 2017-05-08]. Available from: https: //www.gimp.org/about/introduction.html. 42. KYLANDER, Karin. GIMP: the official handbook: the Gimp user’s manual version 1.01. 1st ed. Scottsdale, Arizona: Coriolis Open Press, 1999. ISBN 1-57610-520-2. 43. LECUYER, Anatole; RENARD, Yann. OpenViBE: Open-Source Software for Brain-Computer Interfaces [online]. 2009 [visited on 2017-10-15]. Available from: https://ercim-news.ercim.eu/en78/rd/openvibe- open-source-software-for-brain-computer-interfaces. 44. WANG, Jia; LINDEMAN, Robert W. Unity Indie VRPN Adapter (UIVA) [online] [visited on 2017-10-15]. Available from: http://web.cs. wpi.edu/~gogo/hive/UIVA/.

62 BIBLIOGRAPHY

45. HLINKA, Michal. Motor Imagery based Brain-Computer Interface used in a simple Computer Game. 2017. Available also from: https://is. muni.cz/th/422686/fi_b/. 46. BONNET, L. SSVEP: Steady-State Visual-Evoked Potentials [online]. 2011 [visited on 2017-10-16]. Available from: http://openvibe.inria. fr/steady-state-visual-evoked-potentials/. 47. International Biathlon Union - Rules 2016 [online] [visited on 2017-05-06]. Available from: http : / / res . cloudinary . com / deltatre - spa - ibu/image/upload/fl_attachment/z83vzuicw7ebrgdvs4ii. 48. 2D Game Starter Assets [online] [visited on 2017-03-03]. Available from: https://www.assetstore.unity3d.com/en/#!content/24626. 49. 2D Archers Sprites [online] [visited on 2017-03-10]. Available from: https://www.assetstore.unity3d.com/en/#!/content/18748y. 50. ThinkGear Unity3D Package [online] [visited on 2017-03-22]. Available from: http://developer.neurosky.com/docs/doku.php?id= using_thinkgear_with_unity. 51. Free archery target board clip art [online] [visited on 2017-03-15]. Avail- able from: http://www.clipartlord.com/category/sports-clip- art/archery-clip-art/. 52. Unity - Scripting API: Time.deltaTime [online] [visited on 2017-05-07]. Available from: https://docs.unity3d.com/ScriptReference/ Time-deltaTime.html. 53. Unity - Scripting API: Application.persistentDataPath [online] [visited on 2017-05-07]. Available from: https : / / docs . unity3d . com / ScriptReference/Application-persistentDataPath.html. 54. Unity - Multiplatform - Publish your game to over 25 platforms [online] [visited on 2017-05-05]. Available from: https://unity3d.com/ unity/multiplatform. 55. EEG Sensors - EEG Headsets | NeuroSky [online] [visited on 2017-05-05]. Available from: http://neurosky.com/biosensors/eeg-sensor/ biosensors/. 56. HART, Sandra G. The boo Programming Language [online]. 2004 [visited on 2017-05-08]. Available from: http://boo.sourceforge.net/ BooManifesto.pdf.

63 BIBLIOGRAPHY

57. Essential Facts about the Computer and Video Game Industry: 2015 Sales, Demographic and Usage Data [online]. 2015 [visited on 2017-05-20]. Available from: http://www.theesa.com/wp-content/uploads/ 2015/04/ESA-Essential-Facts-2015.pdf.

64 A Content of the online folder

∙ Foresthlon game — application and source files.

∙ LogAnalyzer application — executable and source files.

∙ Data — log files, calculations xml file from LogAnalyzer, Leader- board xml file and excel file with the data collected fromthe questionnaires.

∙ Questionnaires — the questionnaire forms.

∙ InitialProjet — source code, executables and OpenVibe solution used in initial project (chapter 3)