Classify Frustration Using a Consumer Grade EEG Device for Neuromarketing Purposes

Classify Frustration Using a Consumer Grade EEG Device For Neuromarketing Purposes

Nicolette Stassen April 16, 2017

Abstract Algorithms that may classify emotions such as frustration using a consumer grade EEG device provide possibilities in the ﬁeld of neuromarketing. This paper will focus on how machine learning techniques may be used to classify frustration using participants’ brainwaves as input. Data has been collected using an Emotiv EPOC+ headset while 31 subjects played an adapted coin collector game. The data is analyzed using several feature extraction and machine learning techniques. Although more research is needed, reliable models have been found for both testing on an independent test set, and for testing on only a small amount of data from a subject. These models may be implemented for neuromarketing purposes.

Keywords. EEG, emotion recognition, classifying frustration, machine learning

1 1 Introduction

Neuromarketing research is used increasingly by companies in order to analyze the reactions of potential customers on their advertisements or websites. Normal marketing research uses surveys to ask customers post hoc about their experience, whereas neuromarketing research registers subconscious emotions in real time. To recognize and classify these emotions, algorithms are becom- ing increasingly important. Which emotions influence customer experience and buyer’s intent are not known, but positive and negative emotions with high arousal are known to have an effect (Kim & Lennon, 2013). Within neuromarketing research, negative emotions with high arousal, such as frustration, are used to determine deterrents in advertisements. Insights from users’ affects can be used to improve the effectiveness of said adverts. Neuromarketing research focuses on measuring brain activity as determining personal affects using physiological measurements, such as skin conductance, is difficult. These measurements are multifaceted phenomenons and may show elevation in sweating during frustration, as well as in other emotional states (Figner, Murphy, et al., 2011). Research within the neuromarketing field tries to find the consumer’s unconscious feelings by analyzing the reactions of the brain on several advertisements. Techniques such as functional magnetic reso- nance imaging (fMRI) and electroencephalography (EEG) may be used to measure these unconscious feelings. fMRI measures changes in brain blood flow whereas EEG measures electronic activity in the brain. Both methods are used in neuromarketing research. For example, Neurensics is a neuromarketing company that mainly uses fMRI to record the human brain and how people react to several commercials 1. It does so as fMRI can measure the deep neural structures involved in emotion processing. Braingineers is a neuromarketing company mainly using EEG to record the human brain 2. EEG measures more superficial parts of the brain, but has several advantages over fMRI. Firstly, EEG hardware costs significantly less than fMRI hardware. Secondly, EEG hardware is lightweight and portable. Thirdly, fMRI exposes participants to high-intensity magnetic fields and EEG does not. Lastly, EEG has a higher temporal resolution, which makes it very suitable for neuromarketing purposes. In contrast to fMRI, it can pinpoint a moment of emotion to a much smaller time window and find the exact aspects of a certain commercial that might be frustrating to a consumer. However, the disadvantage of EEG is that it is not yet certain whether it can accurately predict emotions, as it monitors more superficial parts of the brain. Emotion recognition based on EEG is a relatively new research field in the areas of affective computing and neuromarketing. EEG devices measure brainwaves. Professional EEG devices with up to 128 channels can be used to determine frustration and other personal affects (Reuderink, Mühl,& Poel, 2013; Marox et al., 2001). However, professional EEG devices are inconvenient to

1http://www.neurensics.nl/ 2http://www.braingineers.com/

2 use, as the electrodes have a higher chance of being placed in the wrong locations and the participant’s hair must have been washed right before the study. These restrictions make them unpractical for neuromarketing purposes. Con- sumer EEG devices, such as the Emotiv EPOC+ 3, consist of only 14 channels, excluding two reference channels, and are easy to set up. Until now, only few studies report being able to classify emotions using consumer EEG devices, and even fewer have successfully classified frustration (Tarnowski, Kolodziej, Ma- jkowski, & Rak, n.d.; Garrett, Peterson, Anderson, & Thaut, 2003; Félix,2011; Liu & Sourina, 2014; Petrantonakis & Hadjileontiadis, 2010b). Of these studies, frustration has been classified with a highest found accuracy of 63%. The Emo- tiv company provides detection algorithms to measure six different emotional states, among others frustration. They scale the output of their algorithms based on historical patterns for each individual user, therefore it takes a few hours for the system to settle down for a new subject 4. This makes the Emotiv algorithms not ideal for neuromarketing purposes. This research project will focus on whether it is possible to classify frustration using a consumer grade EEG device. The Emotiv EPOC+ EEG device will be used to measure brainwaves and a machine learning algorithm will be designed to classify individual’s frustration levels. Frustration will be induced by playing a simple coin collector game that is to be developed using the Unity game engine 5. This game will do so by making the user’s controls unresponsive at fixed time intervals. Whether frustration will have been induced will be verified using pupil dilation, keyboard loggers, and behavioural analysis. The goal of this research is twofold. Firstly, it is to design an accurate algorithm that may be used to measure frustration in new subjects, without the need to refit the model first. As brainwaves differ between participants, it is not known whether it is possible to design a general model to predict frustration (Lake & Bryden, 1976; Springer & Deutsch, 1998; Bourne, 2005). Therefore, this research does not only aim to design an algorithm to measure frustration in new subjects, without the need to refit the model first, but also to design an algorithm that can classify frustration using only a few seconds of data to train on. Several challenges may be faced during this process, as classifying frustration using consumer grade EEG devices and the effects of frustrating stimuli on brainwaves are relatively unknown. Other challenges include the adaptation of feature extraction classifiers to allow for recognition of data patterns over multiple sensors and over time, as well as designing software that would connect the measurement devices in a way that does not obstruct the collection of data. These problems may be solved using serial analysis of sensor data columns over different lengths and the use of multiple computers running on synchronized software respectively. The following section will provide background information on frustration, EEG, brainwaves, and machine learning techniques. The subsequent sections

3https://www.emotiv.com/epoc/ 4https://emotiv.zendesk.com/hc/en-us/articles/200782279-How-do-you-measure-emotions-in-the-first-place-so-you-can-compare-the-outputs-and-come-up-with-a-number- 5https://unity3d.com/

3 will discuss the experimental setup, methodology, and results from the study to collect the data for the algorithm. The following sections will explain the exploratory data analysis, pre-processing of the data, and feature extraction techniques used in this research. Afterwards, the used feature selection techniques and machine learning classiﬁers will be explained and the results of the algorithms will be presented. Finally, these results will be discussed and a conclusion will be drawn.

4 2 Background 2.1 Frustration Frustration is an emotion that, in psychology, is defined as a response to op- position and can be defined using the valence-arousal model (Schlosberg, 1954; Lang, Greenwald, Bradley, & Hamm, 1993). Frustration is located in the model as having high arousal and negative valence. It is related to anger and dis- appointment and arises when an individual is unable to reach his or her goal (De Botton, 2001; Berkowitz, 1989). Two types of frustration exist, namely internal and external frustration (Brisset & Nowicki, 1973). Internal frustration arises when an individual lacks confidence to fulfill personal goals or desires. External frustration involves factors outside of the individual’s control, such as traffic. Individuals may cope with frustration in different ways. For example, some may use violence, whereas others may use passive-aggressive behaviour. Though individuals handle frustration in different ways, most responses involve aggression (Berkowitz, 1989). Frustration is difficult to measure, as individuals have different tolerances for frustration and everyone shows a different reaction to frustrating events (Szasz, Szentagotai, & Hofmann, 2011). Frustration is measurable for approximately nine seconds after induction (Heraz, Daouda, & Fras- son, 2008). However, as individuals react differently to frustration, this duration may differ per individual. Although measuring frustration is difficult, it is thought to be measurable physiologically (Yu, Mobbs, Seymour, Rowe, & Calder, 2014; Bradley, Miccoli, Escrig, & Lang, 2008; Feild, Allan, & Jones, 2010). Even though physiological responses of the body, such as an increased heart rate and pupil dilation, are used in research to determine whether someone is frustrated, these responses have the disadvantage of being multifaceted and therefore correlates with other factors as well. Whether the participant is frustrated is uncertain, as he or she could also be experiencing another emotion.

2.2 Electroencephalography Frustration, as explained in subsection 2.1, is an emotion. Emotion processing is considered to be measurable in the limbic system, a brain structure located directly under the cerebrum. As the limbic system is a deep lying structure, the use of fMRI has been the most common non-invasive technique to measure emotions in human beings. With the use of fMRI, researchers have demonstrated that frustration is indeed measurable in the brain (Abler, Walter, & Erk, 2005; Yu et al., 2014). However, according to modern neuroscience research, not only the limbic system seems to be involved in emotion processing. LeDoux (2000) suggests the existence of some kind of emotion circuit in the brain. If this emotion circuit does exist, it will mean that brainwaves will be produced during emotion processing. Brainwaves are synchronized electrical pulses that are induced by communicating neurons.

5 EEG measures the electronic activity on the scalp in a non-invasive manner. Studies using EEG recorders found that emotions such as frustration are indeed measurable in mammals (Kimsey, Dyer, & Petri, 1974; Cernea, Kerren, & Ebert, 2011; Reuderink et al., 2013; Isotani et al., 2002; Marox et al., 2001). These studies were conducted using both professional and consumer grade EEG devices, such as the Emotiv EPOC+ 6. This particular device uses dry electrodes to connect to the skull. Dry electrodes do not require any gels or recently washed hair to accurately measure brainwaves. This makes the Emotiv EPOC+ an easy device to use for nonprofessionals and for commercial use. The Emotiv chip digitizes raw brain signals and uses algorithms to translate these signals into brainwave patterns. These patterns are so called EEG frequency bands. The frequency bands measured by the Emotiv headset are: theta, alpha, low beta, high beta, and gamma 7. A different consumer grade EEG device manufactorer is the NeuroSky Com- pany 8. Its devices have been used in studies to measure different kinds of emotions with positive results (de Man, 2014; de Man & Stassen, 2017). How- ever, as NeuroSky headsets use only one sensor, this headset is not thought to be capable of measuring more complex emotions such as frustration. In contrast, the Emotiv EPOC+ uses fourteen sensors, excluding two reference points. The locations of these sensors are shown in figure 1.

Figure 1: Locations of the sensors from the Emotiv EPOC+

6https://www.emotiv.com/epoc/ 7https://github.com/Emotiv/community-sdk/ 8http://www.neurosky.com/

6 2.3 The nature of brainwaves As explained before, EEG measures electronic activity on the scalp. This activity reflects the functional activities that emerge from the brain. The fluctuation in electronic activity can be recorded by measuring the relative voltage between the electrodes. Figure 2 shows these electrical fluctuations measured at sixteen different EEG sensors.

Figure 2: Example of EEG measurement

Figure 2 shows a lot of ﬂuctuation in activity, as is common in EEG. The signals received from the EEG device can be transformed using Fourier transformation using the Fast Fourier Transform (FFT) method. Fourier transformation is based on the belief that signals can be seen as a combination of sinusoids, each with unique amplitude, frequency, and phase values, as can be seen in ﬁgure 3.

Figure 3: Example of Fourier Transform

7 Brain waves that have been transformed using FFT show different frequency bands and can be easier to analyze. The four types of frequency ranges measured by the Emotiv EPOC+ are explained below. Theta waves The theta wave spans the range between four and eight Herz (Hz). The amplitude is mostly greater than 20 micro Volts (µV). The theta wave is correlated with emotional stress, among others frustration (Ochoa, 2002). Alpha waves The alpha wave lies within the range from eight to twelve Hz. The amplitude is mostly between 30 and 50 µV. Alpha waves are thought to indicate a state of relaxed awareness (Ochoa, 2002). Beta waves The beta wave spans the range between 12 and 25 Hz. It has a low voltage between five and thirty µV. Beta brain waves are associated with active thinking (Ochoa, 2002). Emotiv separates beta waves into low (12 to 16 Hz) and high (16 to 25 Hz) beta waves. Gamma waves The gamma wave lies within the range from 25 to 45 Hz. This band is thought to correlate with motor activities (Yuval-Greenberg, Tomer, Keren, Nelken, & Deouell, 2008; Whitham et al., 2008, 2007; Crone, Miglioretti, Gordon, & Lesser, 1998). All brainwaves have different reaction times. When a brainwave is more active, it results in higher amplitudes in the Fourier transformed data. Brainwaves show patterns during cognitive tasks (Gevins et al., 1979). These patterns can be seen over different time elements, locations, or frequency bands. Studies have found differences in brainwave patterns based on the sex and dominant handedness of participants (Lake & Bryden, 1976; Springer & Deutsch, 1998; Bourne, 2005). EEG measurements are sensitive to artifacts. Several methods, such as notch filtering, high pass filtering, low pass filtering, and electromyography (EMG) and electrooculography (EOG) artifact removal needs to be performed to clean noise from EEG data. The implementation of these methods will be explained in sections 6.2 and 6.4.

2.4 Machine learning techniques Machine learning is defined as a technique that makes computers learn without being programmed explicitly and it can be used to find patterns in large data sets (Samuel, 1959; Bishop, 2006). Machine learning algorithms have been used for pattern recognition in brainwaves and have shown positive results in classifying different emotions using EEG signals (Tarnowski et al., n.d.; Garrett et al., 2003; Félix,2011; Liu & Sourina, 2014; Petrantonakis & Hadjileontiadis, 2010b). These studies use several machine learning techniques. The techniques that report the best results are k-nearest neighbour (k-NN) and Support Vector Machine (SVM). This research also considers a General Linear Model (GLM), because in contrast to SVM, it aims not to produce a separating hyperplane,

8 but a function that describes the probability of frustration. It describes a combination of features that indicate frustration, and may therefore also predict frustration if one of those features is absent. GLM is a technique that has been used in brain research, mostly in fMRI studies, and might give more insight in which brainwaves and locations are suggesting frustration (Nakai, Bagarinao, Matsuo, Ohgami, & Kato, 2006). The three techniques, SVM, k-NN, and GLM, will be used in this research and are discussed in further detail in section 8. From all these studies classifying frustration using EEG and machine learning techniques, the highest accuracy score that has been found is 63%. However, these studies do not test on an independent test set. For some cases in this research project the test set will be independent. Therefore, the parameters of the classifiers will be independent of the test set, as one of the goals of this research is to find an algorithm that can classify frustration on new subjects without further training. The second goal of this research, to investigate whether it is possible to build a reliable model using only a small amount of calibration data, is based on the variance of brainwaves between individuals. All individuals react differently to stimuli, and brainwaves differ between individuals as well (Lake & Bryden, 1976; Bourne, 2005; Springer & Deutsch, 1998). Some individuals might have general lower or higher brain activity levels than others. This might result in misclassification by a general model. Figure 4 shows a theoretical example where the activity levels of two subjects in two features are observed and classified using linear SVM kernel functions. The different colours denote different participants, frustration and non-frustration points are denoted by + and x signs respectively. The left subfigure shows the relative inaccuracy of a general model, that would not occur if separate kernels would be constructed for both subjects. However, whether this would actually occur is unknown and using separated models per person could result in overfitting or less robust models for some people.

Figure 4: Theoretical example of frustration classiﬁcation with general and personalized SVM models

9 3 Experimental Setup

In this research, EEG measurements are used to classify frustration using a supervised machine learning algorithm. However, as frustration is a subconscious emotion, it is not something that can be determined beforehand. Therefore, it is necessary to provoke the frustrating feeling and to use objective and subjective measurements to make sure participants were frustrated at the intended times. This section will explain how frustration is provoked, what objective and subjective measurements were used, and why some design choices are made.

3.1 The Game As will be explained below, a simple coin collector game will be developed in order to provoke frustration. The goal of the game will be to collect as many coins as possible within a 2.5 minute time span. The game will be adapted to provoke frustration. During the first 60 seconds of playing, the subject will play the game without intervention to get used to the controls and to the game. After 60 seconds, the input keys will be unresponsive for five seconds, but the game character will keep moving in his current direction. After 70, 95, 108, and 145 seconds, the input keys will be periodically unresponsive either. These instances will have durations of three seconds. These values have been chosen after extensive testing with several participants to find the optimal way to provoke frustration without the player noticing the game has been adapted. During the entire game, the player’s score and the amount of remaining time are displayed in the upper left corner of the playing field. When the input keys are unresponsive, subjects are expected to get frustrated. This is an adaptation of the experimental design by Reuderink, Nijholt, and Poel (2009). In this study a Pacman game has been adapted in order to provoke external frustration. To make the game barely playable, unresponsive keys are introduced. The participant’s goal in this study will be to collect as many coins as possible. However, if the keys are unresponsive, they will not be able to reach their goal and the participants might get frustrated. Programming the game In order to provoke frustration in users, a simple coin collector game will be developed. This game will be programmed using the Unity game engine. The standard Unity character Ethan will be used as a player. A terrain with several hills will be used. Every time the player will collect a coin, a new coin will spawn nearby enough for the player to find and the player will gain a point in his total score. 12 coins will be in the field at all times. To adapt the game, Unity scripts will be written to make sure that the controls will not work for the aforementioned amount of time. Whether the participant is pressing the controls during this time does not matter. The timer will keep running when the game is unresponsive. Other scripts will be written to handle the collecting and spawning of coins. The game will end after 2.5 minutes and the last frame will be frozen in order to record the participant’s

10 score.

3.2 Objective measurements 3.2.1 Electroencephalography Preprocessed EEG measurements will be used as input for the classification algorithm. Research with EEG shows that different brain areas are correlated with several emotions (Isotani et al., 2002). The researcher suggests that frustration is measurable in different brain regions as well. As explained in section 2, other researchers have developed algorithms to classify frustration before. Programming with Emotiv EPOC+ In order to develop the algorithm, a program that collects the user’s EEG frequency bands will have to be developed. This program will be written in Java using Eclipse Neon and will use the Emotiv community SDK to extract the data from the Emotiv EPOC+. The program will collect data every two milliseconds and extract the average frequency band powers per channel with a step size of 0.5 seconds and a window size of two seconds. To improve the legibility and usability of the data, timestamps will be added to each sample and all values in each sample will be labeled by their channel and frequency band. The Emotiv function that recognizes blinking will be used to determine EOG artifacts for each sample. Emotiv has provided interfaces to each of the functions in their Emotiv Development Kit (EDK), which contains the code that collects the samples, calculates the average frequency band powers, and determines some possible EOG artifacts. In order to fully understand the meaning and usefulness of the data returned from this kit, these interfaces will have to be examined to construct an overview of the data transformation sequences.

3.2.2 Galvanic Skin Response and Heart Rate Multiple methods to objectively measure frustration in users are known. Gal- vanic Skin Response (GSR) and Heart Rate (HR) are common methods to measure frustration (Zhai & Barreto, 2006; Scheirer, Fernandez, Klein, & Pi- card, 2002; Zhai & Barreto, 2005). In this research, measuring the GSR will not be possible, as participants will be using their hands in order to play the game. This will lead to noisy results from the device. A professional HR monitor is not available. Other HR monitor devices, such as smart watches, do not measure the HR often enough to return reliable results. Therefore, HR will not be used in this research either.

3.2.3 Pupil dilation As GSR and HR measurements are not possible within this research, other objective measurements will need to be used. Section 2.1 explained the deﬁnition of frustration using the valence-arousal model. Studies have shown that the Sympathetic Division of the Autonomic Nervous System (ANS) inﬂuences the

11 physiological variables when users are in stress (Barreto, Zhai, Rishe, & Gao, 2007; Bradley et al., 2008). Barreto et al. (2007) used several physiological measurements, including pupil dilation, to predict whether users were stressed or relaxed. They found that the mean value of the pupil diameter was signiﬁcantly higher when the user was stressed. Because eyetrackers that are able to measure pupil dilation will be available, this information will be used to objectively determine when users will have been feeling frustrated. The mean of the diameter of the pupils during the normal progress of the game will be compared to the mean of the diameter during frustrating events. As literature states that stress is measured according to the valence-arousal model, this method will be tested on two participants. Stress can indicate high arousal, which is also found in frustration. However, it is likely that this method will not be a good objective measurement to determine frustration, as people may be aroused during the non-frustrating parts of the game as well.

3.2.4 Keyboard pressure Frustration is not only measurable using the user’s physiological responses. Kapoor, Burleson, and Picard (2007) used pressure applied to a computer mouse to determine frustration. They chose this method as it has been shown in research that people’s frustration is often displaced towards inanimate objects. As an example, people slam the door, kick a table, or forcefully press the keys of a computer keyboard (Kapoor et al., 2007; Haner & Brown, 1955; Yu et al., 2014). Therefore, a second objective measurement of frustration will be used in this research. A sensitive microphone will be used to measure whether people will have pressed the buttons on the keyboard louder during frustrating events.

3.2.5 Facial expressions Krumhuber and Scherer (2011) found that facial expressions vary within different emotions. Therefore, all participants will be ﬁlmed during the game. Afterwards, these videos will be analyzed to determine whether participants will have acted in a frustrated manner. Within the videos, the focus will be on the participant’s behaviour. The participant’s facial expressions will be taken into account when determining moments of frustration.

3.3 Subjective measurements To subjectively measure the experienced level of frustration, all participants will be asked to ﬁll in a survey after having played the game. This survey will contain questions on how they would have felt during the game, whether they would have noticed if anything had been wrong, and whether this would have inﬂuenced their state of mind.

12 4 Methodology 4.1 Subjects The sample consisted of 31 persons aged between 18 and 53 years old (M = 25.28, SD = 6.69). 62.5% of these participants were female and 37.5% were male. 9.4% were left-handed participants, whereas 90.6% of the participants had a dominating right hand. No participants had a pacemaker that might inﬂuence the data.

4.2 Measures To measure objectively whether participants are frustrated during the game, multiple devices are used. A smart phone video camera is used to ﬁlm the participants in order to analyze their behaviour. A sensitive microphone, the Blue Snowball 9, is used in combination with the recording program Audacity 10 to measure how loudly participants press the keyboard and whether participants press the keyboard more often than in non-frustrating events. The Mirametrix eyetracker 11 is used to measure the participants’ pupil dilation. A Google Form is used as a questionnaire to measure the participants’ subjective feelings. An Emotiv EPOC+ headset 12 is used to measure the subjects’ brainwaves.

4.3 Procedure All participants from the sample have already been invited to participate in a diﬀerent research case. After that research, they are asked to play the video game. Although in total 31 participants were asked to play the video game, before the subjects play the game, they are told that in total 30 participants will play the game. The participant with the highest amount of collected coins will receive a monetary price of 50 euros. They are also told to play the game in silence, as a microphone is recording, and that any questions they may ask will not be answered during the game. The researcher explains that the participants can use the arrow keys to move the game character around and use the space bar to make the character jump. The keyboard behaviour, pupil dilation, behaviour of the participant, and EEG are measured during the entire game. After playing, the participant is asked to ﬁll in the aforementioned questionnaire. The researcher will note the participant’s score and name. All questionnaires and measurements are stored anonymously.

4.4 Data Analysis This research study subdivides its data analysis into several parts. The Emotiv EPOC+ EEG device provides Fourier transformed EEG measurements per half

9http://www.bluemic.com/products/snowball/ 10http://www.audacityteam.org/download/ 11http://www.mirametrix.com/ 12https://www.emotiv.com/epoc/

13 second with brain activity bands ranging from theta to gamma. Subsections 6.2 and 6.4 thoroughly explain how the data is and has been preprocessed. The analysis of the results of the machine learning data will be explained in subsection 8.4. This section will merely focus on the analysis of the microphone, video camera, eyetracker, and questionnaire data. The Audacity program is used to playback the recorded audio to analyze keyboard behaviour. Figure 5 shows the playback meter where the values are shown in negative decibel (dB). Values found during frustrating events will be compared to values found during non-frustrating events.

Figure 5: Playback meter of Audacity

To analyze pupil dilation, the mean of the diameter of the pupils during playing the game will be compared to the mean of the diameter during frustrating events. This analysis will be performed for the first two participants, as described in subsection 3.2.3. The video feed will be analyzed by classifying each participant’s facial expressions and behaviour. The classification will focus on frustrating behaviour, such as the increased pressing of keys on the keyboard, pulling the hands away from the keyboard, shifting in one’s seat due to muscle tension, smiling or laughing under frustrating events, and moving one’s eyes or looking at the keyboard for confirmation of one’s actions (Yu et al., 2014; Scheirer et al., 2002; Hoque & Picard, 2011; Ehmke & Wilson, 2007; Krumhuber & Scherer, 2011). Lastly, the results of the questionnaire will be analyzed to check whether participants did feel frustrated when the input keys did not respond, as to confirm the classification of frustration by the researcher. Participants are considered to be frustrated when they are stuck in a mountainous area, or when the input keys are unresponsive, as both events prevent the player from reaching their goal to collect as many coins as possible. All results will be analyzed using the statistical R-language in the program RStudio. All confidence intervals (CI) will be calculated using the Wilson procedure without a correction for continuity and will be based on 95% certainty. Most results will be summarized by giving the percentage of occurrence with their 95%CI.

14 5 Results data collection

As explained in subsection 4.4, frustration is deﬁned when input keys are unresponsive, or when participants are stuck in a mountainous area. Before being able to determine moments of frustration, the results of the objective and subjective measurements need to be analyzed. The following subsections will discuss the video analysis, audio analysis, and pupil dilation results respectively. For both video and audio analysis, two participants had to be excluded as no reliable data sets had been recorded.

5.1 Video analysis The video analysis indicates that 9.7%, 95%CI [0.03,0.25], of the participants showed no noticeable diﬀerences. 58.1%, 95%CI [0.41,0.74], of the participants pressed the keyboard more often during frustrating events. 16.1%, 95%CI [0.07,0.33], of the subjects retracted their hands from the keyboard during these events. 19%, 95%CI [0.09,0.36], of the participants shifted in their chair. 25.8%, 95%CI [0.14,0.43], of the subjects laughed or smiled right after frustrating events, possibly as a token of relief. Lastly, 19%, 95%CI [0.09,0.36], of the participants looked at the keyboard or showed they were moving their eyes over the screen more than in non-frustrating events. Figure 6 visualizes these results. The dots denote the proportion of the occurrence of the problems. The bars show the 95% conﬁdence intervals.

Figure 6: Visualization of the results of general problems in elderly people

5.2 Audio analysis Figure 7 shows that it is not unlikely that both during frustrating and non- frustrating events the audio samples from the participants stem from a Normal population. Therefore, a paired t-test is performed to test whether the underlying distribution of X1, ...X29 ∼ F , denoting the frustrating events, is smaller

15 than that of Y1, ..., Y29 ∼ G, denoting the non-frustrating events. The hypothesis is as follows: H0 : F > G,

H1 : F ≤ G.

Figure 7: Normal QQ plots of audio samples for all participants

The resulting p-value after performing the paired t-test is 2.9 ∗ 10−7. This is below the signiﬁcance threshold of 0.05. Therefore, H0 can be rejected and H1 may be assumed.

5.3 Pupil dilation Subsection 3.2.3 explained how the samples of only the ﬁrst two participants would be used to investigate the usefulness of this measurement in determining frustrating events during this research. Figures 8 and 9 suggest that not all pupil dilation of the two participants during frustrating and non-frustrating events stem from a Normal population. Therefore, a Mann-Whitney test is performed to test whether X1, ...X931 ∼ F , denoting the non-frustrating events, is greater than Y1, ..., Y218 ∼ G, denoting the frustrating events. The hypothesis is as follows: H0 : F < G,

H1 : F ≥ G. For the first participant, the resulting p-value after performing the Mann- Whitney test is 1 for the left eye and 0.9892 for the right eye. Both are above the significance threshold, so H0 cannot be rejected and H1 may not be assumed. For the second participant, the resulting p-value after performing the Mann- Whitney test is 2.2 ∗ 10−16 for the left eye and 0.9892 for the right eye. The p-value for the left eye is below the significance threshold, so H0 can be rejected. The p-value for the right eye is above the significance threshold. Therefore H0 cannot be rejected. Taking all results into account, this measurement is discarded as it seems to be unreliable in measuring frustration.

16 Figure 8: Normal QQ plots of pupil dilation samples of ﬁrst participant

5.4 Survey results 96.9%, 95%CI [0.84,0.99], of the participants indicated feeling frustrated during the game. They indicated to be motivated to collect as many coins as possible within the duration of the game. They also indicated feeling frustrated when the input keys were unresponsive or when getting stuck in a mountainous area.

17 Figure 9: Normal QQ plots of pupil dilation samples of second participant

18 6 Exploratory data analysis

As explained in section 5, significant differences have been found between frustrating and non-frustrating events. Therefore, this data set was chosen as a valid base for the algorithm. However, several difficulties exist. Brainwaves have different reaction times for emotions and accurately classifying frustration is therefore not possible for only one point in time. Heraz et al. (2008) state in their research that frustration is noticeable for approximately nine seconds. Therefore, when a frustrating stimulus presented itself to the player, all data samples from that moment until nine seconds after the frustrating event had ended are classified as frustration. Before being able to use the data as an input for the machine learning process, the data needs to be analyzed and preprocessed. This will be explained in the following subsections.

6.1 Problem definition, assumptions, and similar problems The first research goal requires that the machine learning algorithm is able to train on the EEG data from previous subjects to predict frustration in new subjects without further training. Therefore, a general model will need to be developed and the test set needs to remain independent during the cross validation. The second research goal requires the algorithm to be able to train on data from a small time window, but still be able to accurately predict longer time windows. EEG samples are subject to different constraints and philosophies than common data domains. In most fields, outliers are unwanted in a data set. There- fore, these might be removed using interpolation. However, in the domain of EEG, outliers might represent peaks in brainwave activity. Higher activity might indicate frustration. Therefore, interpolating the data might remove important features. Simultaneously, the three dimensionality of brain activity needs to be taken into account. Time, amplitude, and location may form predictive patterns for frustration. Amplitude is represented in the collected value, different locations are included as separate columns is the data set. Time needs to be included into the data set by observing subsequent samples as a time series and reporting measurements. Unfortunately, no other solutions to this problem exist. Some problems look similar, but none provide a clear way on how to build the algorithm.

6.2 Preprocessing of the data by Emotiv Multiple steps have been taken by Emotiv before any collected data is returned to the user. This section will explain all executed steps and the available artifacts and their removal.

19 Several artifacts exist for EEG measurements (Patil, 2010). Firstly, cardiac and pulse artifacts are measurable in the T3-A1, T5-O1, and P3-O1 sensors. These regions are not specifically monitored by the Emotiv EPOC+, but the artifacts can be removed using high pass filtering, as will be explained in section 6.2.1. Secondly, movement artifacts show mostly in the frontal and temporal lobes, specifically in T3. The Emotiv EPOC+ does not measure this region either. Therefore, these artifacts cannot be specifically removed. A way to deal with these is explained in subsection 6.4.2. Thirdly, electrode artifacts are electrode pops, movements, and sweat artifacts. The following subsections describe how to remove these. Fourthly, external noise is ambient frequency noise, set at 60 Hz in Europe. Lastly, eye artifacts are blinking and ocular artifacts. Blinking is mostly measured in the pre-frontal region of the brain, namely in the sensors AF3, AF4, F7, and F8. Ocular artifacts from lateral gaze have a frequency less than 1 Hz.

6.2.1 High pass filtering High pass filtering is used to remove some of the artifacts. High pass filtering is performed on the raw data by Emotiv. A high pass filter at 0.16Hz characteristic is used to remove DC offset.

6.2.2 Fourier Transformation To remove regular artifacts with set frequencies, the raw EEG data can be transformed to speciﬁc frequency bands by representing the raw signal time domain as the weighted sum of sines and cosines. This can be achieved using either Fourier transformation via FFT or by Wavelet transformation. Emotiv uses FFT to transform to the theta (4-8 Hz), alpha (8-12 Hz), low beta (12-16 Hz), high beta (16-25 Hz), and gamma (25-45 Hz) frequency bands.

6.2.3 Hanning Windowing Fourier transformation knows some limitations as the analysis is based on a ﬁnite set of data. When a measured signal is periodic and a whole number of periods periods ﬁll the time interval completely, FFT is performed well. How- ever, in EEG signals, the signal is not a whole number of periods. This can lead to abrupt waveforms and sudden changes in the measured signal. To handle this limitation of Fourier transformation, the data needs to be smoothed or interpolated. Emotiv uses Hanning Windowing for each two second window frame to smooth the data. Windowing reduces the amplitudes of discontinuities. With this method, time records are multiplied by a two second window with an amplitude that varies gradually toward zero at the edges of this window. This eventually results in a continuous waveform without sudden transitions.

20 Figure 10: Boxplot of the theta band values for all sensors.

6.3 Collected data The data as collected from the Emotiv EPOC+ consists of samples, one sample for roughly every 500 milliseconds. Each sample consists of a timestamp, an array of real-valued numbers, and a boolean that displays whether a blink has been registered. Each real-valued number corresponds to the absolute average band power in one channel in one frequency band. The gamma frequency band is known for its high correlation with motorical activity (Yuval-Greenberg et al., 2008; Whitham et al., 2008, 2007; Crone et al., 1998). The data in this research has been collected using a game that forces people to type and otherwise be motorically active. Because this could influence the final model by serving as a confounding variable, this frequency band is omitted from the data set. Boxplots of the collected data are shown in figures 10 through 13. The data displayed in these figures has been collected from the first participant only. The visible outliers in these figures might either indicate peaks in brain activity, or might be caused by eye, movement, or other artifacts. Subsection 6.4.2 will explain more on outlier detection and removal. After these artifacts have been removed from the data set, the column that denotes blinking is discarded, as it is of no further use in the research.

6.4 Preprocessing of the data The following subsections explain how the collected data is preprocessed and what design choices were made in implementing these methods.

21 Figure 11: Boxplot of the alpha band values for all sensors.

Figure 12: Boxplot of the low beta band values for all sensors.

22 Figure 13: Boxplot of the high beta band values for all sensors.

6.4.1 Interpolation Interpolation estimates the approximate value of data points based on their neighbours. This method can use diﬀerent functions to model data points in a sample. Because of the nature of EEG data, increases in brain activity may indicate either brainwave peaks, or may be caused by artifacts. If any spikes are caused naturally, interpolation will almost always return values that are between those of their preceding and succeeding samples. This would remove the peak and discard possibly important information.

6.4.2 Outlier detection and Kalman filtering The preprocessing methods described in subsection 6.2 take almost all artifacts into account. Testing with the Emotiv EPOC+ indicated that the four pre-frontal sensors (AF3, AF4, F7, and F8) returned unrealistic values during blinking. Electrooculography (EOG) artifacts, such as those caused by blinking, are well known in EEG measurements. Other well known artifacts are teeth clenching, smiling, frowning, or head movement. The outliers seen in figures 10 through 13 might be caused by these artifacts. Figure 14 shows the effect of blinking and jaw clenching on raw EEG data. If this raw data were to be processed using FFT and the absolute values were to be taken, the instances of blinking and jaw clenching would result in high-valued outliers of the band activity values. Several studies show that the maximum activity in a specific brain band is dependent on a large number of factors (Don, Ponton, Eggermont, & Masuda, 1993; Mourtazaev, Kemp, Zwinderman, & Kamphuisen, 1995; Besson, Fa¨ıta,& Requin, 1994). Even though a study by Teplan et al. (2002) indicates a maximum value in raw EEG data of approx-

23 Figure 14: The effect of blinking and jaw clenching on raw EEG data. imately 100µV, this is not applicable, as the collected data has already been processed by Emotiv and the original values are unknown and irretrievable. As the goal of this research is to design an algorithm that applies to all people in general, instead of adapting parameters based on subject characteristics, outliers are considered to be caused by artifacts if their absolute activity value is more than 1.5 times the interquartile range (IQR) over the 75th percentile. This measure varies per subject, but takes the subjects baseline activity level and variance into account. The data is filtered to retrieve the original values. Several filtering methods are applicable with blinking artifacts. Recursive Least Square (RLS) and Least Mean Square (LMS) are most frequently used for EEG data. However, these filters are used to smooth raw data, which makes them inapplicable to this research project. Several studies have shown that Kalman filtering performs better than RLS and LMS (Banerjee, Basu, & Chakraborty, 2007; Omidvarnia, Atry, Setarehdan, & Arabi, 2005; Oikonomou, Tzallas, Tsalikakis, Fotiadis, & Konitsiotis, 2009). LMS is an adaptive filter that changes its filter weights inversely proportional to the Mean Square Error (MSE). Because of its stochastic nature, the filter is discarded for this research. It would try to find all possible combinations to train the classifier and would lead to unmanageable computation time. RLS is an adaptive filter that minimizes a weighted linear least squares cost function by recursively adapting coefficients as input signals are received. RLS considers input signals deterministically. However, RLS is also inapplicable to this research, as it needs data to train to and this data is unavailable. Kalman filtering observes a series of measurements containing statistical noise and other inaccuracies over time. It produces estimates of unknown variables by using Bayesian inference. The advantage of Kalman filtering is its prediction step. Therefore, Kalman filtering is considered most applicable to this research.

6.4.3 Correlation Highly correlating factors can cause problems in machine learning and modelling algorithms. These features can cause overﬁtting and compromise the reliability

24 and robustness of the model. Therefore, all features that correlate more than 95% are removed from the data set. This correlation threshold is based on the relative lack of knowledge on brainwave correlation in emotions. Deleting too many features could cause the deletion of valuable information. Features that were found to be correlated are theta, alpha, and low beta on location T7 and theta, alpha, low beta, and high beta on location P7. All but the theta values in these sensors were removed from the data set.

6.5 Determining frustration Because the algorithm will use supervised learning, frustration needs to be determined within the collected data. Heraz et al. (2008) estimate an average duration of emotions in brainwaves of between six and nine seconds. This research determines frustration from the beginning of a frustrating stimulus until eight seconds after the event has ended. During the determination process, a boolean column is added tot the data that denotes whether a participant was frustrated during the corresponding 0.5 second time window.

6.6 Evaluating the algorithm Previous studies that reported using EEG to classify emotions used accuracy as a performance metric of their algorithm. This research uses the F1-score metric instead, as it incorporates precision and recall and takes unbalanced samples into account, making it a more robust performance metric. A theoretical random model would return an F1-score of 0.4 on a test set from this data set. This score will be used to value the models that are returned by the algorithm.

6.7 Data processing scripts The program that applies Kalman filtering and determines frustration has been programmed in Python using Sublime Text 3. This language has been selected because of its large array of external libraries, as well as its compatibility with multiprocessing. Firstly, the data from samples that were collected during the second after a blink is discarded, as explained in subsection 6.4.2. Secondly, artifacts are removed by finding values of more than 1.5 times the IQR over the 75th percentile, as explained in subsection 6.3. Kalman filtering is used to smooth the data, and all values that have been removed are replaced with their smoothed counterparts, provided that these values are below the aforementioned threshold. Otherwise, the original values remain. The start and end time of the test have been recorded by hand, as well as the start and duration of frustrating events, as described in subsections 4.4 and 6.5. The program compares all timestamps to the registered start of the experiment, until it is to be exceeded. Then, samples are classified as frustration

25 where applicable, until the end of the experiment is reached. All data outside the experiment is discarded. All scripts have been programmed using multiprocessing. The computer used is a Mac Pro with eight 2.7 Giga Herz (GHz) cores. All eight cores are used near-optimally to speed up the calculations.

26 7 Feature Extraction

This section will compare several feature extraction techniques. EEG processing uses a variety of techniques, such as Wavelet-Based Features, Statistical-Based Features, and Higher Order Crossing (HOC) Based Features. This research will not use Wavelet-Based Features as a feature extraction technique, as it is only applicable on raw data. As Emotiv has already performed FFT on the data, this technique is not useful for this research. This study will use Statistical-Based Features and HOC-Based Features to find the optimal classification algorithm. The following subsections will discuss all specific techniques used. Each technique will be explained and an overview of the structure of the returned data will be presented. This structure takes the removal of five features from subsection 6.4.3 into account. In general, each column in the data structure is a feature. This feature is formed by an activity band in a sensor and can be seen as an infinite time series Xt. All feature extraction methods observe a subset of Xt XN (n), n = 1, ..., N, denoting the N last measurements of this activity band in this sensor. This approach is called using a boxcar, where for each new observation that is recorded, the oldest observation in XN (n) is discarded. To ensure the correct operation of this approach, N samples must have been recorded before the experiment is started. Thus, the time series contains precisely N elements at any given moment.

7.1 Statistical-Based Features The Statistical-Based Feature extraction methods used in this research are a standard statistical benchmark, windowed standard deviation, and windowed regression. The following subsections explain these methods.

7.1.1 Standard statistical benchmark A statistical feature vector has only been proposed for physiological signals. This vector has been expanded using the features of physiological signals with derived EEG signals. Each feature is used to form a Statistical Feature Vector (FV ). For each XN , a corresponding FV is constructed with the following components.

1. The mean of the raw signal

N 1 X µ = X (n); X N N n=1

2. The standard deviation of the raw signal v u N u 1 X σ = t (X (n) − µ )2; X N N X n=1

27 3. The mean of the absolute values of the ﬁrst diﬀerences of the raw signal

N−1 1 X δ = |X (n + 1) − X (n)|; X N − 1 N N n=1

4. The mean of the absolute values of the first differences of the standardized signal N−1 1 X δX δ¯ = |X¯ (n + 1) − X¯ (n)| = ; X N − 1 N N σ n=1 X 5. The mean of the absolute values of the second differences of the raw signal

N−2 1 X γ = |X (n + 2) − X (n)|; X N − 2 N N n=1

6. The mean of the absolute values of the second diﬀerences of the standardized signal

N−2 1 X γX γ¯ = |X¯ (n + 2) − X¯ (n)| = . X N − 2 N N σ n=1 X

In these formulas, X¯N is the standardized signal of XN (n) and

¯ T FV = (µX , σX , δX , δX , γX , γ¯X ) .

The complete feature vector for this observation is obtained by concatenating the obtained FV for each activity band in each sensor. Because the optimal value of N is unknown, the calculations are run for N ∈ {15, 20, 25}. The result ﬁts a data structure where each sample consists of a timestamp, an array of real-valued numbers, and a binary value whether the participant was frustrated. For 14 sensors, four activity bands, and six feature measurements, the array of real-valued numbers has a length of 306 values.

7.1.2 Windowed standard deviation Brain activity causes raw EEG signals to fluctuate heavily. These fluctuations are visible in Fourier transformed data as well. Here, intense fluctuations are visible as high-valued outliers in the absolute values of this data. Therefore, a sudden increase in absolute values would indicate an increase in fluctuation in the underlying raw signal data. These fluctuations can be measured using standard deviation. The approach used in this research is called windowed, because the standard deviation is calculated over the N most recent samples. A feature vector is proposed that uses windowed standard deviation. This procedure uses a time series XN (n), n = 1, .., N, N = 2, ..., 25 for each activity

28 band in each sensor. The standard deviation of the time series σXN is calculated according to v u N u 1 X σ = t (X(n) − µ )2, ∀N XN N XN n=1

N 1 P where µXN = N X(n). The resulting feature vector for each activity band n=1 T FV = (σXN | N = 2, ..., 25) . The complete feature vector for this observation is constructed by concatenating the obtained FV for each activity band in each sensor. Because it is unknown whether a too large value of N would result in overﬁtting, as it has more features than a large value of N, subvectors of FV are taken for each value x ∈ {15, 20, 25}. This results in a data structure where each sample consists of a timestamp, an array of real-valued numbers, and a binary value denoting whether the participant was frustrated. For 14 sensors, four activity bands, and x − 1 feature measurements, the array of real-valued numbers has a length of 714, 969, or 1224 values.

7.1.3 Windowed regression Using the same logic as in the previous subsection, heavy brain activity would result in large absolute values in the FFT processed data. Sudden increases and decreases in these values would be measurable using regression. The approach used in this research is called windowed, because the regression is calculated over the N most recent samples. A feature vector is proposed that uses windowed regression. This procedure uses a time series XN (n), n = 1, .., N, N = 2, ..., 25 for each activity band in each sensor. The regression of the time series δXN is calculated according to

δXN = XN (1) − XN (N), ∀N.

The resulting feature vector for each activity band in each sensor FV = (δXN | N = 2, ..., 25)T. The complete feature vector for this observation is constructed by concatenating the obtained FV for each activity band in each sensor. Using the same logic as in the previous subsection, subvectors of FV are taken for each value of x ∈ {15, 20, 25}. This results in a data structure where each sample consists of a timestamp, an array of real-valued numbers, and a binary value denoting whether the participant was frustrated. For 14 sensors, four activity bands, and x − 1 feature measurements, the array of real-valued numbers has a length of 714, 969, or 1224 values.

7.2 Windowed Higher Order Crossings Studies have shown that a HOC-Based Features technique is eﬀective at feature extraction for EEG analysis (Petrantonakis & Hadjileontiadis, 2010a; Kedem & Yakowitz, 1994; Petrantonakis & Hadjileontiadis, 2010b). Almost all observed time series display local and global oscillations over time. This behaviour can

29 be expressed through the mean-crossing count. Consider a series Yt, t = 1, ..., N with mean µY . For Zt = {Yt − µY : t = 1, .., N}, the mean-crossing count is equal to the zero-crossing count. In general, applying filters to a time series causes a change in oscillations and therefore the zero-crossing count. Higher Order Crossings (HOC) is a technique counts zero-crossings over time. HOC is applied as follows. Let ∇ be the backward difference operator defined by ∇Zt ≡ Zt − Zt−1. ∇ is a high-pass filter. A sequence of high-pass filters can be defined as

k−1 =k ≡ ∇ , k > 0

0 with identity ﬁlter =1 ≡ ∇ . This leads to

k X k − 1 k − 1 (k − 1)! = (Z ) = ∇k−1Z = (−1)j−1Z with = . k t t j − 1 t−j+1 j − 1 (j − 1)!(k − j)! j=1

A binary time series Xt(k) is constructed under 1 if =k(Zt) ≥ 0 Xt(k) = , k > 0, t = 1, ..., N. 0 if =k(Zt) < 0

The desired higher order crossings are estimated by counting the changes in X1(k), ..., XN (k), N X Dk = |Xt(k) − Xt−1(k)|. t=2 The feature vector FV for each activity band in each sensor is constructed as T FV = (Di | i = 1, ..., N) . Two variations of HOC are applied in this research. In the first variation, the mean of Yt is taken and Zt is constructed as described above. In the second variation, the mean of the values in all samplesµ ¯ is taken for each activity band in each sensor and Zt is constructed usingµ ¯ as µY . These variations are called Variable Mean HOC and Static Mean HOC respectively. The complete feature vector for each observation is constructed by concatenating the obtained FV for each activity band in each sensor. Using the same logic as in the previous sections, subvectors of FV are taken for each value of x =∈ {15, 20, 25}. This results in a data structure where each sample consists of a timestamp, an array of real-valued numbers, and a binary value whether the participant was frustrated. For 14 sensors, four activity bands, and x − 1 feature measurements, the array of real-valued numbers has a length of 714, 969, or 1224 values. HOC was originally intended to be used on raw EEG signals (Petrantonakis & Hadjileontiadis, 2010b, 2010a). However, because heavy fluctuations in raw activity result in large fluctuations in average band power, HOC could still be useful, albeit using different parameters than would be used for raw EEG signals.

30 7.3 Feature extraction scripts The program that performs feature extraction is programmed in Python using Sublime Text 3. This language has been selected because of its large array of external libraries, as well as its compatibility with multiprocessing. Because of the large amount of calculations necessary, the program attempts to use as much caching as possible, as well as to optimally use all cores. The used computer is a Mac Pro with eight 2.7GHz cores.

31 8 Machine Learning

The following subsection describes the general decisions and assumptions related to the implementation of the Machine Learning algorithms. This research uses k- Nearest Neighbour (k-NN), Support Vector Machine (SVM), and General Linear Model (GLM) algorithms. These algorithms will be explained in the subsequent subsections.

8.1 General decisions and layout This subsection will explain several issues that have been encountered during the design and implementation of the Machine Learning algorithms.

8.1.1 Subject separation The aim of this research is to develop an algorithm that can be used to predict frustration with data from completely new subjects, or to predict frustration using only a few seconds of training data. In case of the first research goal, the data from each subject needs to be separated, as a model that could train on data from later test subjects would be unrepresentative of the later performance of the model. The separation of subjects throughout the machine learning process solves this problem. This ensures the generality of the model. The data is split using a 70/30 ratio, where 70% of the data is used to train the model. For the second research goal, exploratory testing has indicated that having an unrepresentative distribution of frustrated and non-frustrated samples could result in under- or overclassification. Adding two sequences of frustrated samples would also force the removal of several subjects whose data contains only two sequences of frustration. Therefore, the first 40 seconds of non-frustrated samples, as well as the complete first frustration sequence are taken as a training set.

8.1.2 Model parameters Each Machine Learning technique offers different parameters to tune its decision function. k-NN offers the possibility of varying the number k, as well as the manner of weighing each neighbour. An SVM can use various kernel functions with corresponding parameters to be tuned. Because the optimal parameters are unknown, a gridsearch technique is employed to fit the model for each valid combination of parameters and return the model with the best results. All combinations are valued using 3-fold cross validation for comparison.

8.1.3 Calculation time Fitting the models for each machine learning and feature extraction technique, as well as for each combination of parameters, causes the script run times to exceed multiple days and has caused the computers that were used to run the scripts on to crash on multiple occasions. To limit the number of processes that

32 have to be run, the depth parameter N, which was explained in section 7, is limited to the value set {15, 20, 25}.

8.2 Machine Learning techniques The following subsections will explain the Machine Learning techniques used in this research.

8.2.1 k-Nearest Neighbour k-NN is a technique that observes the k data points in the memory of the model that bear the closest resemblance to the data point that is to be classified. It evaluates resemblance by calculating the Euclidean distance between all points and the new data point. A variation on the classic k-NN weighs the k neighbours by relative closeness to the new data point. This method causes data points that are nearer to have more influence on the outcome of the classification process. In this technique, k can be any natural number greater than zero, but the in this research, k ∈ {5, 10, 25, 50}. The weighing measure can be either ’uniform’ for the classical k-NN technique, or ’distance’ for the variation. Each combination is considered by the grid search technique mentioned in section 8.1.2. k-NN is known to be prone to class imbalance, which can be solved by using different voting mechanisms (Coomans & Massart, 1982). However, the data in this research consists of 40% frustration and 60% non-frustration samples, so class imbalance is not an issue and these voting mechanisms can be discarded.

8.2.2 Support Vector Machine SVM uses a kernel function to divide the m-dimensional space into n subspaces, where m is the number of features and n is the number of labels. In this research, n = 2 and 306 ≤ m ≤ 1224, depending on the feature extraction technique used. Thus, the kernel function simply splits the feature space. The kernel function can be either ’linear’, ’poly’, ’rbf ’, or ’sigmoid’, where each kernel function has its own parameters. The linear kernel has no parameters, but a polynomial kernel can vary its degree, its kernel coefficient, and its independent term. The radial basis function kernel can only vary its kernel coefficient, and the sigmoid kernel can choose different values for its kernel coefficient and its independent term. These parameters are not changed from their default values by the grid search technique. Therefore, the degree of the 1 polynomial function is 3, the kernel coefficient is m , and the independent term is equal to 0.0. Independent of the kernel function is the penalty parameter C for the error term. This penalty parameter is varied between 0.1, 1.0, and 6.0. Each combination of kernel function and penalty parameter is considered by the grid search technique mentioned in section 8.1.2.

33 To combat the extraordinary run time of this algorithm, the maximum number of iterations is limited to 1000.

8.2.3 General Linear Model GLM attempts to ﬁt a function to the dataset by tuning the parameters corresponding to each variable or feature. Thus, a GLM in this research has 306 ≤ m ≤ 1224 variables. Because the aim of this research is to classify frustration, the Logistic Regression model has been selected. This model is specialized in classiﬁcation, instead of regression. This problem has a binary nature, either a person is frustrated or not, so varying any parameters that are meant for multinomial problems is unnecessary. Therefore, the algorithm uses all default parameters, including the ’liblinear’ solver. No grid search technique is applied here.

8.3 Feature Selection The large number of features present in the data set would cause the regular machine learning techniques to take an unacceptable amount of time to complete for all parameters and techniques. A large number of features could also cause overﬁtting and non-robust models, thus reducing the overall quality of the model. Several techniques exist to reduce or transform the dimensionality of the problem, using feature selections. The following subsections will discuss the techniques used in this research.

8.3.1 Principal component analysis To reduce overfitting, principal component analysis (PCA) is applied to extract the most important components of initial input features. PCA is a technique that reduces the amount of features in the data set by transforming the data to fit the n-dimensional space described by the n eigenvectors that have the highest corresponding eigenvalue. These eigenvectors form an orthagonal space (Jolliffe, 2002). In this m-dimensional problem, each eigenvector is a vector of length m as well. The number of components to add to the model is a complex decision, as each component is less descriptive than the previous component, but might still improve the model. On the other hand, each extra component that is added might induce overfitting and lower overall model performance.

8.3.2 Component selection through a search algorithm A simple, greedy algorithm is implemented to constructively increase the number of components used by PCA and applying gridsearch and 3-fold cross validation to each transformed set. It subsequently constructs a 95% conﬁdence interval and compares it to the current best solution. If the new solution im- proves over the current best solution by more than a threshold value τ, the model is improved.

34 This algorithm, which bears resemblance to local search, is na¨ıve and will often get stuck in locally optimal solutions (Lenstra, 2003). These are solutions sets that have a value that is not optimal, but no neighbouring sets with higher solutions. In order to prevent the algorithm from getting stuck in these local optima, the algorithm is extended with forced moving. If the model is unable to improve by more than τ, forced moving is applied. This move is performed at most n times. Because each additional dimension in the PCA transformed data explains less variance than the previous one, the total number of dimensions is limited to m. If the number of dimensions exceeds m, forced moving is no longer applied. Forced moving does not necessarily improve the solution and can induce overﬁtting. Therefore, the resulting 3-fold cross validation results from all vis- ited solutions are stored and the best solution is selected as the ﬁnal model. The aforementioned technique is run with parameters τ = 0.005, n = 50, m = 100.

8.4 Data analysis For the algorithms that train on several subjects and test on independent subjects, for each combination of depth, feature extraction, and machine learning techniques, a 3-fold cross validation on the training set will be performed. The resulting confidence intervals will be compared. Confidence intervals that are significantly worse will be discarded. The remaining models will be fitted to the entire training set and will be used to predict the independent test set. The models in this research will be prone to classifying many or all samples as either frustration or non-frustration. Therefore, the models will be valued on multiple scores that have been selected to prevent this. For each model, the f1- scores of its frustrated and non-frustrated parts, as well as the weighted average of these scores, will be observed. If the weighted average of both f1-scores is under 0.5 or if either f1-score is under 0.4, the model will be discarded. If either of these conditions is true, the model does not outperform a random algorithm, or it classifies many or all samples the same. For the algorithms that train and test on data from the same subject, the resulting confidence intervals of the f1-scores and the weighted average will be compared as described above. 95% confidence intervals will be constructed by training and testing each model on each subject separately and storing the results. For each subject, the gridsearch technique will be applied, so selected parameters may differ. If no significant differences may be found in the models, the model with the highest f1 score will be chosen to be visualized. The important features will be extracted for the subjects that have scored the highest and lowest on this model. Moreover, if a subject would have only one frustration point, he or she will be discarded. For the best performing models from both techniques, the classification of each sample will be plotted against the determined frustration. The range of the classification function will be in {0, 1}. To increase the applicability of the algorithm in the neuromarketing field, the function will be smoothed to increase readability and transformed to range [0, 1]. The weighted moving average of the

35 classification will be calculated, as it signals general trends of frustration and non-frustration. If no significant differences may be found in the models, the model with the highest f1-score will be chosen be visualized. For these models, the most prominent components will be extracted from each remaining model and these will be transformed back to their respective features using PCA. These features will be compared to literature. The selected features will be determined differently for each Machine Learning technique. For k-NN, the features will be weighted by the sum of their influence on the PCA components that will have been used in the model, whereas for SVM and GLM, the PCA components will be multiplied by their influence on the respective decision functions.

8.5 Machine learning scripts The program that performs machine learning is programmed in Python using Sublime Text 3. This language has been selected because of its large array of external libraries, such as the used machine learning library Scikit-learn, as well as its compatibility with multiprocessing. Because of the large amount of calculations necessary, the program attempts to use as much caching as possible, as well as to optimally use all cores. The used computer is a Mac Pro with eight 2.7GHz cores.

36 9 Machine learning results

This section will discuss the results of the machine learning algorithms. The ﬁrst subsection will discuss the results of the 3-fold cross validation of the models for training on new subjects, as well as the results of the signiﬁcantly better algorithms on the independent test set. It will subsequently display the valid results of these independent tests and display the important features of the model as explained in subsection 8.4. The second subsection will discuss the results of the models on the independent test sets for the models that have been trained and tested on data from the same subjects.

9.1 Generalized models This subsection will display the results of the models that have been trained and tested on data from separate subjects. Firstly, the results of the 3-fold cross validation will be presented. Secondly, the results of the signiﬁcantly better algorithms on the independent test set will be displayed. Lastly, the results of the valid models will be displayed graphically and the important features will be shown.

9.1.1 3-fold cross validation Tables 1, 2, and 3 display the results of the generalized 3-fold cross validation for SVM, k-NN, and GLM respectively. The results will be displayed in tables that contain the mean f1-scores, 95% conﬁdence intervals, and the optimal number of components. The tables display the selected parameters for the k-NN and SVM technique. Results indicate that, for the SVM models, all feature extraction methods are signiﬁcantly better than the results from any methods used by the other algorithms. The combinations of techniques and depth that remain can be written as {(SVM, x, y): ∀x ∈ X, ∀y ∈ Y } with X = {SB, SD, Regression, VMHOC, SMHOC},Y = {15, 20, 25}.

9.1.2 Independent test set The signiﬁcantly better algorithms that have been mentioned in the previous subsection have been tested on the independent test set. The results of these tests are displayed in table 4. The results indicate that all but the SVM VMHOC 15 combination produce invalid results as explained in subsection 8.4. Therefore, all other combinations are discarded.

9.1.3 Model analysis Figure 15 shows the visualizations of the model results using the weighted average against the determined frustration. Table 5 shows the most important features for the model. The table displays the frequency band, the sensor location, the depth, and the normalized importance.

37 Table 1: Results for SVM with training on separated subjects.

Method Chosen Mean 95% Conﬁdence Number of parameters f1-score Interval components SB 15 Kernel: linear, 0.58 95%CI [0.56,0.59] 30 C: 0.1 SB 20 Kernel: poly, 0.58 95%CI [0.57,0.59] 58 C: 1.0 SB 25 Kernel: poly, 0.58 95%CI [0.57,0.6] 12 C: 6.0 SD 15 Kernel: sigmoid, 0.58 95%CI [0.56,0.59] 3 C: 6.0 SD 20 Kernel: linear, 0.58 95%CI [0.56,0.6] 15 C: 0.1 SD 25 Kernel: sigmoid, 0.58 95%CI [0.56,0.6] 1 C: 6.0 Regression 15 Kernel: poly, 0.58 95%CI [0.56,0.59] 93 C: 0.1 Regression 20 Kernel: poly, 0.58 95%CI [0.56,0.59] 42 C: 6.0 Regression 25 Kernel: poly, 0.58 95%CI [0.56,0.6] 84 C: 1.0 VMHOC 15 Kernel: linear, 0.57 95%CI [0.56,0.58] 1 C: 6.0 VMHOC 20 Kernel: poly, 0.56 95%CI [0.56,0.57] 7 C: 0.1 VMHOC 25 Kernel: poly, 0.58 95%CI [0.56,0.59] 7 C: 0.1 SMHOC 15 Kernel: poly, 0.57 95%CI [0.56,0.58] 2 C: 6.0 SMHOC 20 Kernel: poly, 0.57 95%CI [0.55,0.58] 9 C: 0.1 SMHOC 25 Kernel: linear, 0.57 95%CI [0.56,0.58] 2 C: 0.1

9.2 Personalized models This subsection will display the results of the models that have been trained and tested on data from the same person. These results are shown in tables 6, 7, and 8 for SVM, k-NN, and GLM respectively. No model is signiﬁcantly better than another one. As the k-NN SB 25 model has the highest f1-score, this model will be used in further analyses.

38 Table 2: Results for k-NN with training on separated subjects.

Method Chosen Mean 95% Conﬁdence Number of parameters f1-score Interval components SB 15 Neighbours: 5, 0.41 95%CI [0.35,0.46] 9 Weights: uniform SB 20 Neighbours: 10, 0.42 95%CI [0.41,0.43] 1 Weights: distance SB 25 Neighbours: 5, 0.42 95%CI [0.41,0.43] 1 Weights: distance SD 15 Neighbours: 5, 0.39 95%CI [0.38,0.4] 1 Weights: distance SD 20 Neighbours: 5, 0.41 95%CI [0.4,0.43] 1 Weights: distance SD 25 Neighbours: 5, 0.44 95%CI [0.43,0.45] 5 Weights: distance Regression 15 Neighbours: 5, 0.39 95%CI [0.39,0.4] 2 Weights: distance Regression 20 Neighbours: 5, 0.4 95%CI [0.39,0.41] 1 Weights: distance Regression 25 Neighbours: 5, 0.42 95%CI [0.4,0.43] 1 Weights: distance VMHOC 15 Neighbours: 5, 0.4 95%CI [0.38,0.42] 1 Weights: distance VMHOC 20 Neighbours: 5, 0.39 95%CI [0.38,0.4] 1 Weights: distance VMHOC 25 Neighbours: 5, 0.4 95%CI [0.35,0.45] 52 Weights: uniform SMHOC 15 Neighbours: 5, 0.4 95%CI [0.39,0.41] 5 Weights: uniform SMHOC 20 Neighbours: 5, 0.39 95%CI [0.39,0.4] 4 Weights: distance SMHOC 25 Neighbours: 5, 0.39 95%CI [0.38,0.4] 1 Weights: distance

9.2.1 Model analysis Figures 16 and 17 show the visualizations of the model results using the weighted average against the determined frustration. As this model is using the k-NN technique, all persons use the same features therefore there is no distinction in persons. Table 9 shows the most important features for the model. The table displays the frequency band, the sensor location, the used feature, and the normalized importance.

39 Table 3: Results for GLM with training on separated subjects.

Method Mean 95% Conﬁdence Number of f1-score Interval components SB 15 0.37 95%CI [0.29,0.45] 65 SB 20 0.4 95%CI [0.32,0.48] 89 SB 25 0.4 95%CI [0.31,0.5] 87 SD 15 0.32 95%CI [0.25,0.4] 76 SD 20 0.34 95%CI [0.28,0.41] 76 SD 25 0.38 95%CI [0.3,0.45] 76 Regression 15 0.1 95%CI [0.06,0.14] 56 Regression 20 0.09 95%CI [0.04,0.14] 54 Regression 25 0.08 95%CI [0.04,0.13] 53 VMHOC 15 0.28 95%CI [0.27,0.3] 78 VMHOC 20 0.29 95%CI [0.26,0.32] 38 VMHOC 25 0.32 95%CI [0.3,0.34] 61 SMHOC 15 0.3 95%CI [0.26,0.33] 73 SMHOC 20 0.3 95%CI [0.28,0.32] 42 SMHOC 25 0.33 95%CI [0.31,0.36] 67

Table 4: Results of the signiﬁcantly better models with separated and independent testing.

Method Mean Non-frustration Frustration f1-score f1-score f1-score SVM SB 15 0.21 0.10 0.44 SVM SB 20 0.16 0.00 0.49 SVM SB 25 0.16 0.00 0.49 SVM SD 15 0.16 0.00 0.49 SVM SD 20 0.16 0.00 0.49 SVM SD 25 0.16 0.00 0.49 SVM Regression 15 0.16 0.00 0.49 SVM Regression 20 0.16 0.00 0.49 SVM Regression 25 0.16 0.00 0.49 SVM VMHOC 15 0.53 0.55 0.49 SVM VMHOC 20 0.58 0.74 0.24 SVM VMHOC 25 0.25 0.13 0.49 SVM SMHOC 15 0.17 0.01 0.49 SVM SMHOC 20 0.19 0.04 0.49 SVM SMHOC 25 0.19 0.02 0.49

40 (a) Participant 5 (b) Participant 12 (c) Participant 20

(d) Participant 21 (e) Participant 22 (f) Participant 23

(g) Participant 24 (h) Participant 25 (i) Participant 27

(j) Participant 29

Figure 15: Weighted average of VMHOC-15 classiﬁcation model

41 Table 5: Selected features for the VMHOC 15 algorithm

42 Table 6: Results for SVM with personalized training and testing.

Method Mean Non-frustration Frustration f1-score f1-score f1-score SB 15 0.51 0.47 0.42 95%CI [0.48, 0.54] 95%CI [0.4, 0.54] 95%CI [0.37, 0.47] SB 20 0.56 0.51 0.42 95%CI [0.51, 0.61] 95%CI [0.43, 0.59] 95%CI [0.35, 0.5] SB 25 0.53 0.51 0.42 95%CI [0.49, 0.56] 95%CI [0.44, 0.58] 95%CI [0.37, 0.48] SD 15 0.56 0.52 0.41 95%CI [0.52, 0.6] 95%CI [0.43, 0.61] 95%CI [0.35, 0.47] SD 20 0.55 0.54 0.43 95%CI [0.51, 0.59] 95%CI [0.46, 0.61] 95%CI [0.38, 0.48] SD 25 0.54 0.51 0.41 95%CI [0.49, 0.59] 95%CI [0.43, 0.59] 95%CI [0.34, 0.48] Regression 15 0.55 0.58 0.36 95%CI [0.52, 0.58] 95%CI [0.52, 0.64] 95%CI [0.31, 0.4] Regression 20 0.54 0.51 0.38 95%CI [0.52, 0.56] 95%CI [0.44, 0.59] 95%CI [0.31, 0.44] Regression 25 0.56 0.55 0.37 95%CI [0.53, 0.6] 95%CI [0.48, 0.63] 95%CI [0.31, 0.42] VMHOC 15 0.54 0.52 0.41 95%CI [0.5, 0.58] 95%CI [0.44, 0.59] 95%CI [0.35, 0.46] VMHOC 20 0.55 0.51 0.42 95%CI [0.51, 0.58] 95%CI [0.44, 0.58] 95%CI [0.36, 0.48] VMHOC 25 0.54 0.52 0.37 95%CI [0.51, 0.58] 95%CI [0.44, 0.6] 95%CI [0.32, 0.42] SMHOC 15 0.52 0.49 0.42 95%CI [0.49, 0.55] 95%CI [0.41, 0.56] 95%CI [0.36, 0.47] SMHOC 20 0.53 0.51 0.42 95%CI [0.49, 0.56] 95%CI [0.44, 0.58] 95%CI [0.38, 0.47] SMHOC 25 0.54 0.54 0.39 95%CI [0.5, 0.58] 95%CI [0.47, 0.62] 95%CI [0.34, 0.44]

43 Table 7: Results for k-NN with personalized training and testing.

Method Mean Non-frustration Frustration f1-score f1-score f1-score SB 15 0.54 0.53 0.42 95%CI [0.5, 0.58] 95%CI [0.46, 0.6] 95%CI [0.36, 0.48] SB 20 0.53 0.52 0.44 95%CI [0.49, 0.57] 95%CI [0.45, 0.59] 95%CI [0.39, 0.49] SB 25 0.54 0.53 0.46 95%CI [0.49, 0.59] 95%CI [0.46, 0.6] 95%CI [0.39, 0.52] SD 15 0.57 0.62 0.37 95%CI [0.53, 0.61] 95%CI [0.57, 0.66] 95%CI [0.31, 0.43] SD 20 0.57 0.6 0.41 95%CI [0.53, 0.62] 95%CI [0.55, 0.66] 95%CI [0.35, 0.47] SD 25 0.55 0.57 0.42 95%CI [0.51, 0.6] 95%CI [0.51, 0.63] 95%CI [0.35, 0.48] Regression 15 0.57 0.63 0.3 95%CI [0.53, 0.6] 95%CI [0.59, 0.67] 95%CI [0.25, 0.35] Regression 20 0.57 0.62 0.32 95%CI [0.53, 0.61] 95%CI [0.57, 0.67] 95%CI [0.28, 0.36] Regression 25 0.58 0.6 0.31 95%CI [0.54, 0.62] 95%CI [0.54, 0.65] 95%CI [0.25, 0.38] VMHOC 15 0.57 0.63 0.29 95%CI [0.53, 0.6] 95%CI [0.59, 0.67] 95%CI [0.25, 0.32] VMHOC 20 0.56 0.63 0.32 95%CI [0.53, 0.59] 95%CI [0.6, 0.67] 95%CI [0.28, 0.37] VMHOC 25 0.56 0.62 0.34 95%CI [0.53, 0.59] 95%CI [0.57, 0.67] 95%CI [0.3, 0.39] SMHOC 15 0.56 0.62 0.32 95%CI [0.53, 0.6] 95%CI [0.59, 0.66] 95%CI [0.27, 0.37] SMHOC 20 0.56 0.62 0.35 95%CI [0.52, 0.6] 95%CI [0.58, 0.67] 95%CI [0.31, 0.4] SMHOC 25 0.57 0.62 0.36 95%CI [0.53, 0.6] 95%CI [0.57, 0.67] 95%CI [0.32, 0.4]

44 Table 8: Results for GLM with personalized training and testing.

Method Mean Non-frustration Frustration f1-score f1-score f1-score SB 15 0.52 0.54 0.4 95%CI [0.47, 0.56] 95%CI [0.49, 0.6] 95%CI [0.35, 0.45] SB 20 0.53 0.53 0.42 95%CI [0.49, 0.57] 95%CI [0.47, 0.59] 95%CI [0.37, 0.48] SB 25 0.53 0.49 0.44 95%CI [0.47, 0.59] 95%CI [0.41, 0.57] 95%CI [0.37, 0.52] SD 15 0.52 0.55 0.39 95%CI [0.49, 0.55] 95%CI [0.49, 0.6] 95%CI [0.34, 0.45] SD 20 0.5 0.53 0.41 95%CI [0.46, 0.55] 95%CI [0.48, 0.58] 95%CI [0.35, 0.47] SD 25 0.52 0.52 0.4 95%CI [0.47, 0.56] 95%CI [0.47, 0.58] 95%CI [0.34, 0.46] Regression 15 0.54 0.6 0.36 95%CI [0.51, 0.57] 95%CI [0.56, 0.64] 95%CI [0.33, 0.39] Regression 20 0.53 0.6 0.33 95%CI [0.5, 0.57] 95%CI [0.56, 0.65] 95%CI [0.29, 0.37] Regression 25 0.53 0.59 0.34 95%CI [0.49, 0.57] 95%CI [0.54, 0.64] 95%CI [0.3, 0.39] VMHOC 15 0.5 0.56 0.41 95%CI [0.47, 0.53] 95%CI [0.51, 0.6] 95%CI [0.37, 0.45] VMHOC 20 0.49 0.54 0.42 95%CI [0.46, 0.52] 95%CI [0.5, 0.57] 95%CI [0.37, 0.46] VMHOC 25 0.5 0.53 0.44 95%CI [0.46, 0.54] 95%CI [0.49, 0.58] 95%CI [0.38, 0.49] SMHOC 15 0.51 0.57 0.4 95%CI [0.48, 0.55] 95%CI [0.54, 0.61] 95%CI [0.36, 0.45] SMHOC 20 0.53 0.58 0.41 95%CI [0.49, 0.57] 95%CI [0.54, 0.63] 95%CI [0.36, 0.46] SMHOC 25 0.57 0.61 0.45 95%CI [0.53, 0.6] 95%CI [0.56, 0.66] 95%CI [0.4, 0.49]

45 (a) Participant 1 (b) Participant 3 (c) Participant 4

(d) Participant 5 (e) Participant 6 (f) Participant 7

(g) Participant 8 (h) Participant 9 (i) Participant 10

(j) Participant 11 (k) Participant 12 (l) Participant 13

(m) Participant 14 (n) Participant 15 (o) Participant 16

Figure 16: Weighted average of SB-25 classiﬁcation model (part 1)

46 (a) Participant 17 (b) Participant 18 (c) Participant 19

(d) Participant 20 (e) Participant 21 (f) Participant 22

(g) Participant 23 (h) Participant 26 (i) Participant 27

(j) Participant 28 (k) Participant 29 (l) Participant 30

(m) Participant 32

Figure 17: Weighted average of SB-25 classiﬁcation model (part 2)

47 Table 9: Selected features for the SB 25 algorithm

Frequency band Location Feature Weight Theta T7 sgX -1.039 Theta T7 sd 1.0 Theta T7 dX 0.939 Theta FC5 dX 0.831 Theta P8 dX 0.818 Theta O1 dX 0.784 Theta O2 dX 0.727 Theta F4 dX 0.723 Theta T7 sdX 0.697 Theta P7 dX 0.641

48 10 Discussion

This section will analyze and discuss the results of the major findings presented in the previous section. The first subsection will discuss the combination of techniques that has yielded the best results against a test set of completely new subjects. It will also analyze the results to see where the algorithm misclassified. The second subsection will do the same for the best combination of techniques against previously known participants. The third subsection will compare these two models and discuss which model is most suited for neuromarketing purposes. The final subsection will discuss some design choices that might have influenced the model.

10.1 Model results for generalized models This subsection will discuss the models that have been trained and tested on completely new subjects. Significant differences have been found in all models that used SVM. Out of these models, only the model that performed feature extraction through variable mean HOC with depth 15 could be considered reliable, as it is the only model that has performed better than a random model would have. This would indicate that changes in activity in the Fourier transformed, absolute brain activity would be a predictive factor for frustration. The depth of 15 indicates that frustration is noticeable for no more than 7.5 seconds. Visualizations of this reliable model are shown in figure 15. The figures show that even though the model performs better than a random model would, it still misclassifies. The results of the model are analyzed by observing the instances where the model was wrong, and by analyzing the participant’s behaviour at that moment. This has shown that during false positive classifications, participants were talking, frowning, laughing, or using their facial muscles in some other way. Using facial muscles in EEG causes higher recorded activity and could act as a confounding variable, as mentioned in section 6.2. The Emotiv algorithms indicate the same. The ten most important features from the VMHOC 15 model are mostly from the frontal alpha frequency bands, as is shown in table 5. Correlation between frustration and alpha brainwave patterns in the frontal area of the brain correlate with frustration is consistent with found literature (Schmidt & Trainor, 2001; Harmon-Jones, Vaughn-Scott, Mohr, Sigelman, & Harmon-Jones, 2004; Harmon-Jones, 2003). The features indicate that patterns exist between and within locations and frequencies, as different depths are used to classify frustration, although this does make the model prone to confounding variables like muscle activity.

10.2 Model results for personalized models The personalized models perform well at classifying frustration within subjects, even if the training set only consists of one frustration point and less than a minute of baseline activity. No model is considered signiﬁcantly better than

49 the other algorithms. Therefore, the model with the highest f1-score, the k-NN SB 25 model, has been selected to be visualized. This depth indicates that frustration is noticeable for 12.5 seconds. Figures 16 and 17 show the visualized results of the model. The figures indicate that the performance of the model varies heavily from person to person. As explained in subsection 8.4, the most important features in a k-NN algorithm depend on the variance of the data set that is explained by each. As a property of PCA is that it finds the orthagonal basis that maximizes explained variance, these variances can be extracted from the fitted PCA model. The ten features that explain most of the variance from the k-NN SB 25 model are shown in table 9. The features in this table are mostly from the theta frequency band and originate from sensors located in the temporal, parietal, and occipital brain areas. As explained in subsection 2.3, theta brainwaves are thought to be involved in emotion processing such as frustration. The features indicate that, among others, the mean of the absolute values of the first differences and the measurements from the T7 location explain a lot of variance in brain activity and could be important in classifying frustration.

10.3 Method comparison The resulting models from both methods perform better than a random algorithm would. However, both methods result in models that apply different techniques and depths. For individual participants, the model suggests that frustration is noticeable in the brain for about 12.5 seconds, while when testing on new subjects, frustration is only noticeable for about 7.5 seconds. In the general model, the frontally located alpha frequency band seems to be important in classifying frustration. More specifically, the features extracted from alpha frequency bands located at sensors AF3 and AF4 are most present in table 5. However, the importance of the same sensors in the personalized models is much lower, as can be seen in appendix A in table 11. The highest absolute importance for features extracted from alpha AF3 and alpha AF4 are -0.037 and 0.039 respectively. In the personalized model, the temporally, parietally, and occipitally located theta frequency bands seem most important in explaining variance within the data set and could therefore be important in classifying frustration. Table 9 indicates that the features with the heighest weights mostly originate from Theta T7. The highest absolute importance for features extracted in the general model is 0.661. Appendix A contains all features from the general model and their importance in table 10. Both the alpha and theta frequency bands are thought to be involved in classifying frustration. However, some differences exist between the models. In the general model, most misclassifications seem to be influenced by muscle artifacts. In the personalized model, this is not as clear. For some subjects, the model is very precise in its classifications, whereas for others, the model tends to over- or underclassify frustration. It might be that the higher alpha frequency band is more sensitive to muscle artifacts. For the personalized model,

50 it would also be possible that the model classified frustration correctly where the researchers had misclassified. For neuromarketing purposes, it would be possible to create either a general or a personalized model. However, the model would have to be calibrated for each subject in the latter case. This process could obstruct the neuromarketing process if it would take too long. Therefore, it might not be useful if preprocessing and fitting the model were to take more than two minutes. This research does show that reliable models may be created using one frustration point and 40 seconds of baseline samples. The general model indicates correlation with muscle movement. If the model would be used to classify data obtained through passive neuromarketing, such as commercial analysis, this might not cause many problems. However, neuromarketing might also be implemented in investigating online usability. These purposes would require participants to type, scroll, click, or move other muscles, and might cause misclassified frustation. For these purposes, the personalized models might be more reliable, as correlation with muscle artifacts seems less present. Within this research, analysis is performed on multiple participants, and the effects of misclassifications could be diminished.

10.4 Design choices In the general model, false positives may be explained by confounding variables such as facial muscle contractions. These artifacts could not be removed in this research, as raw data was not available. However, when using raw data, these artifacts can be recognized and dealt with. As only Fourier transformed data was available, some steps could not be performed. Independent Component Analysis (ICA) is a decomposition technique that could have reduced noisy factors in raw data. This research discarded features if they resulted in over 0.95 correlation. Future research could discard features at higher correlation levels to determine whether this would better prevent overfitting. Moreover, data from Emotiv is returned every half second. This value could be varied, as it might be either too much, or too little data to fit a model on. Measuring frustration can be very difficult. If frustration would be correlated with, for example, cortisol, this might indicate that frustration is a long term process and that it should be measured as such, i.e., using an experimental block design. Furthermore, frustration might have a learning factor or could grow over time. This research has focused on building a classification model. Future research could investigate whether a regression model would be more effective at classifying frustration. It could also investigate whether previously experienced frustration influences brain activity. The method that trains and tests a model on data from the same subject uses one frustration sequence and 40 seconds of subsequent non-frustration. The optimal amount of frustration samples needed to create a robust model is unknown. Future research might investigate the ideal ratio in this distribution, which may vary between subjects, as well as whether differences between

51 frustration sequences are noticeable or significant. This research has used the weighted average of the f1-scores to measure the classification processes of both models. However, this might not have been the ideal measurement technique. Because brainwaves are time-dependent, and could be seen as a time series, only the first few seconds of a frustration process might be recognizable. The rest of the frustration process might not be predictable. On the other hand, brain activity may have a small delay, so frustration might be noticeable only some time after it has been induced. Lastly, brainwaves and their patterns seem to differ between individuals. A study by Leon-Carrion et al. (2006) shows that the time course and activation of the brain differs between genders when processing emotional stimuli. Several studies also indicate an effect of dominant handedness on brain activity patterns (Lake & Bryden, 1976; Springer & Deutsch, 1998). Future research could exam- ine whether models that incorporate differences in gender and hand dominance, or models that would be fitted for one group specifically, would perform better.

52 11 Conclusion

Classiﬁcation of frustration using a consumer grade EEG device for neuromarketing purposes seems possible. Constructed models perform better than theoretical random models would. However, no other models exist, so the constructed models cannot be compared. This research has found that a general model seems to highly correlate with muscle artifacts, which is not as present in personalized models. In these models, it is merely necessary to use one sequence of frustration samples and 40 seconds of non-frustrated samples to build a reliable model. Therefore, models created in this research are useful for neuromarketing purposes. Even though this model shows promising results, using raw data instead of processed date is thought to improve model performance.

Acknowledgements. Special thanks go out to my supervisor Mark Hoogen- doorn, for the support of my research and for all the help given. Furthermore, a thank you is in order for my colleagues at Braingineers, Lo Wang Cheung, Max van Kaathoven, Eva Lammers, Roderick Reichenbach, and Marion Veth, for providing the practicalities and help for this research.

53 References

Abler, B., Walter, H., & Erk, S. (2005). Neural correlates of frustration. Neu- roreport, 16 (7), 669–672. Banerjee, A., Basu, K., & Chakraborty, A. (2007). Prediction of eeg signal by digital filtering. In Proceedings of international conference on intelligent systems & networks, jagadhri, india. Barreto, A., Zhai, J., Rishe, N., & Gao, Y. (2007). Significance of pupil diameter measurements for the assessment of affective state in computer users. In Advances and innovations in systems, computing sciences and software engineering (pp. 59–64). Springer. Berkowitz, L. (1989). Frustration-aggression hypothesis: examination and re- formulation. Psychological bulletin, 106 (1), 59. Besson, M., Fa¨ıta,F., & Requin, J. (1994). Brain waves associated with musical incongruities differ for musicians and non-musicians. Neuroscience letters, 168 (1), 101–105. Bishop, C. M. (2006). Pattern recognition. Machine Learning, 128 , 1–58. Bourne, V. J. (2005). Lateralised processing of positive facial emotion: Sex differences in strength of hemispheric dominance. Neuropsychologia, 43 (6), 953–956. Bradley, M. M., Miccoli, L., Escrig, M. A., & Lang, P. J. (2008). The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology, 45 (4), 602–607. Brisset, M., & Nowicki, S. (1973). Internal versus external control of rein- forcement and reaction to frustration. Journal of personality and social psychology, 25 (1), 35. Cernea, D., Kerren, A., & Ebert, A. (2011). Detecting insight and emotion in visualization applications with a commercial eeg headset. In Proceed- ings of sigrad 2011. evaluations of graphics and visualization—efficiency; usefulness; accessibility; usability; november 17-18; 2011; kth; stockholm; sweden (pp. 53–60). Coomans, D., & Massart, D. L. (1982). Alternative k-nearest neighbour rules in supervised pattern recognition: Part 1. k-nearest neighbour classification by using alternative voting rules. Analytica Chimica Acta, 136 , 15–27. Crone, N. E., Miglioretti, D. L., Gordon, B., & Lesser, R. P. (1998). Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. ii. event-related synchronization in the gamma band. Brain, 121 (12), 2301–2315. De Botton, A. (2001). The consolations of philosophy (Vol. 63). Vintage. de Man, J. (2014). Analysing emotional video using consumer eeg hardware. In International conference on human-computer interaction (pp. 729–738). de Man, J., & Stassen, N. (2017). Analyzing fear using a single-sensor eeg device. In Intelligent technologies for interactive entertainment: 8th international conference, intetain 2016, utrecht, the netherlands, june 28–30, 2016, re- vised selected papers (pp. 86–96).

54 Don, M., Ponton, C. W., Eggermont, J. J., & Masuda, A. (1993). Gender differences in cochlear response time: an explanation for gender amplitude differences in the unmasked auditory brain-stem response. The Journal of the Acoustical Society of America, 94 (4), 2135–2148. Ehmke, C., & Wilson, S. (2007). Identifying web usability problems from eye-tracking data. In Proceedings of the 21st british hci group annual conference on people and computers: Hci... but not as we know it-volume 1 (pp. 119–128). Feild, H. A., Allan, J., & Jones, R. (2010). Predicting searcher frustration. In Proceedings of the 33rd international acm sigir conference on research and development in information retrieval (pp. 34–41). Félix,T. M. R. (2011). Emotion recognition using eeg during frustrating human– computer interaction: the influence of artifact removal via ica. Figner, B., Murphy, R. O., et al. (2011). Using skin conductance in judgment and decision making research. A handbook of process tracing methods for decision research, 163–184. Garrett, D., Peterson, D. A., Anderson, C. W., & Thaut, M. H. (2003). Com- parison of linear, nonlinear, and feature selection methods for eeg signal classification. IEEE Transactions on neural systems and rehabilitation engineering, 11 (2), 141–144. Gevins, A., Zeitlin, G., Yingling, C. D., Doyle, J. C., Dedon, M., Schaffer, R. E., . . . Yeager, C. L. (1979). Eeg patterns during ‘cognitive’tasks. i. methodology and analysis of complex behaviors. Electroencephalography and clinical neurophysiology, 47 (6), 693–703. Haner, C. F., & Brown, P. A. (1955). Clarification of the instigation to action concept in the frustration-aggression hypothesis. The Journal of Abnormal and Social Psychology, 51 (2), 204. Harmon-Jones, E. (2003). Clarifying the emotive functions of asymmetrical frontal cortical activity. Psychophysiology, 40 (6), 838–848. Harmon-Jones, E., Vaughn-Scott, K., Mohr, S., Sigelman, J., & Harmon-Jones, C. (2004). The effect of manipulated sympathy and anger on left and right frontal cortical activity. Emotion, 4 (1), 95. Heraz, A., Daouda, T., & Frasson, C. (2008). Decision tree for tracking learner’s emotional state predicted from his electrical brain activity. In Interna- tional conference on intelligent tutoring systems (pp. 822–824). Hoque, M., & Picard, R. W. (2011). Acted vs. natural frustration and delight: Many people smile in natural frustration. In Automatic face & gesture recognition and workshops (fg 2011), 2011 ieee international conference on (pp. 354–359). Isotani, T., Lehmann, D., Pascual-Marqui, R. D., Fukushima, M., Saito, N., Yagyu, T., & Kinoshita, T. (2002). Source localization of brain electric activity during positive, neutral and negative emotional states. In Inter- national congress series (Vol. 1232, pp. 165–173). Jolliffe, I. (2002). Principal component analysis. Wiley Online Library. Kapoor, A., Burleson, W., & Picard, R. W. (2007). Automatic prediction of frustration. International journal of human-computer studies, 65 (8),

55 724–736. Kedem, B., & Yakowitz, S. (1994). Time series analysis by higher order crossings. IEEE press New York. Kim, J., & Lennon, S. J. (2013). Effects of reputation and website quality on online consumers’ emotion, perceived risk and purchase intention: Based on the stimulus-organism-response model. Journal of Research in Interactive Marketing, 7 (1), 33–56. Kimsey, R. A., Dyer, R. S., & Petri, H. L. (1974). Relationship between hippocampal eeg, novelty, and frustration in the rat. Behavioral biology, 11 (4), 561–568. Krumhuber, E. G., & Scherer, K. R. (2011). Affect bursts: dynamic patterns of facial expression. Emotion, 11 (4), 825. Lake, D. A., & Bryden, M. P. (1976). Handedness and sex differences in hemispheric asymmetry. Brain and language, 3 (2), 266–282. Lang, P. J., Greenwald, M. K., Bradley, M. M., & Hamm, A. O. (1993). Look- ing at pictures: Affective, facial, visceral, and behavioral reactions. Psy- chophysiology, 30 (3), 261–273. LeDoux, J. E. (2000). Emotion circuits in the brain. Annual review of neuroscience, 23 (1), 155–184. Lenstra, J. K. (2003). Local search in combinatorial optimization. Princeton University Press. Leon-Carrion, J., Damas, J., Izzetoglu, K., Pourrezai, K., Mart´ın-Rodr´ıguez, J. F., y Martin, J. M. B., & Dominguez-Morales, M. R. (2006). Differential time course and intensity of pfc activation for men and women in response to emotional stimuli: a functional near-infrared spectroscopy (fnirs) study. Neuroscience letters, 403 (1), 90–95. Liu, Y., & Sourina, O. (2014). Real-time subject-dependent eeg-based emotion recognition algorithm. In Transactions on computational science xxiii (pp. 199–223). Springer. Marox, E., Rodr´ıguez, H., Yañez,G., Bernal, J., Rodr´ıguez, M., Fernández, T., . . . Guerrero, V. (2001). Broad band spectral measurements of eeg during emotional tasks. International journal of neuroscience, 108 (3-4), 251–279. Mourtazaev, M., Kemp, B., Zwinderman, A., & Kamphuisen, H. (1995). Age and gender affect different characteristics of slow waves in the sleep eeg. SLEEP-NEW YORK-, 18 , 557–557. Nakai, T., Bagarinao, E., Matsuo, K., Ohgami, Y., & Kato, C. (2006). Dynamic monitoring of brain activation under visual stimulation using fmri—the advantage of real-time fmri with sliding window glm analysis. Journal of neuroscience methods, 157 (1), 158–167. Ochoa, J. B. (2002). Eeg signal classification for brain computer interface applications. Ecole Polytechnique Federale de Lausanne, 7 , 1–72. Oikonomou, V. P., Tzallas, A. T., Tsalikakis, D. G., Fotiadis, D. I., & Konitsi- otis, S. (2009). The use of kalman filter in biomedical signal processing. INTECH Open Access Publisher.

56 Omidvarnia, A. H., Atry, F., Setarehdan, S. K., & Arabi, B. N. (2005). Kalman filter parameters as a new eeg feature vector for bci applications. In Signal processing conference, 2005 13th european (pp. 1–4). Patil, S. P. (2010). What every clinician should know about polysomnography. Respiratory care, 55 (9), 1179–1195. Petrantonakis, P. C., & Hadjileontiadis, L. J. (2010a). Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis. IEEE Transactions on affective computing, 1 (2), 81–97. Petrantonakis, P. C., & Hadjileontiadis, L. J. (2010b). Emotion recognition from eeg using higher order crossings. IEEE Transactions on Information Technology in Biomedicine, 14 (2), 186–197. Reuderink, B., Mühl,C., & Poel, M. (2013). Valence, arousal and dominance in the eeg during game play. International Journal of Autonomous and Adaptive Communications Systems, 6 (1), 45–62. Reuderink, B., Nijholt, A., & Poel, M. (2009). Affective pacman: A frustrating game for brain-computer interface experiments. In International conference on intelligent technologies for interactive entertainment (pp. 221–227). Samuel, A. L. (1959). Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3 (3), 210–229. Scheirer, J., Fernandez, R., Klein, J., & Picard, R. W. (2002). Frustrating the user on purpose: a step toward building an affective computer. Interacting with computers, 14 (2), 93–118. Schlosberg, H. (1954). Three dimensions of emotion. Psychological review, 61 (2), 81. Schmidt, L. A., & Trainor, L. J. (2001). Frontal brain electrical activity (eeg) distinguishes valence and intensity of musical emotions. Cognition & Emo- tion, 15 (4), 487–500. Springer, S. P., & Deutsch, G. (1998). Left brain, right brain: Perspectives from cognitive neuroscience. WH Freeman/Times Books/Henry Holt & Co. Szasz, P. L., Szentagotai, A., & Hofmann, S. G. (2011). The effect of emotion regulation strategies on anger. Behaviour research and therapy, 49 (2), 114–119. Tarnowski, P., Ko lodziej, M., Majkowski, A., & Rak, R. J. (n.d.). An inno- vative approach to classification of emotions in eeg signal for the use in neuromarketing research. Teplan, M., et al. (2002). Fundamentals of eeg measurement. Measurement science review, 2 (2), 1–11. Whitham, E. M., Lewis, T., Pope, K. J., Fitzgibbon, S. P., Clark, C. R., Love- less, S., . . . Willoughby, J. O. (2008). Thinking activates emg in scalp electrical recordings. Clinical Neurophysiology, 119 (5), 1166–1175. Whitham, E. M., Pope, K. J., Fitzgibbon, S. P., Lewis, T., Clark, C. R., Love- less, S., . . . others (2007). Scalp electrical recording during paralysis: quantitative evidence that eeg frequencies above 20hz are contaminated by emg. Clinical Neurophysiology, 118 (8), 1877–1888.

57 Yu, R., Mobbs, D., Seymour, B., Rowe, J. B., & Calder, A. J. (2014). The neural signature of escalating frustration in humans. Cortex, 54 , 165–178. Yuval-Greenberg, S., Tomer, O., Keren, A. S., Nelken, I., & Deouell, L. Y. (2008). Transient induced gamma-band response in eeg as a manifestation of miniature saccades. Neuron, 58 (3), 429–441. Zhai, J., & Barreto, A. (2005). Instrumentation for automatic monitoring of aﬀective state in human-computer interaction. In Flairs conference (pp. 207–213). Zhai, J., & Barreto, A. (2006). Stress detection in computer users based on digital signal processing of noninvasive physiological variables. In Engineering in medicine and biology society, 2006. embs’06. 28th annual international conference of the ieee (pp. 1355–1358).

Appendices

A All selected features for the general and personalized model

Table 10: All selected features for the general VMHOC 15 algorithm

Frequency band Location Depth Weight Alpha AF4 12 1.0 Alpha AF4 14 0.993 Alpha AF4 15 0.976 Alpha AF3 12 0.963 Alpha AF3 14 0.962 Alpha F3 14 0.956 Low beta F3 15 0.952 Alpha FC6 14 0.946 Alpha AF4 13 0.945 Alpha P8 12 0.944 Low beta F3 13 0.936 Alpha F4 14 0.931 Alpha P8 14 0.929 Alpha AF3 8 0.912 Alpha AF4 8 0.907 Alpha AF3 15 0.897 Low beta F3 12 0.892 Alpha AF3 13 0.89 Alpha AF3 10 0.886 Alpha P8 13 0.885 Low beta F3 14 0.885

58 Low beta AF4 13 0.882 Alpha P8 15 0.881 Theta T8 12 0.875 Theta P8 14 0.874 Alpha F8 14 0.872 Alpha F3 12 0.871 Alpha FC6 13 0.871 Theta O2 14 0.87 Alpha F4 12 0.866 Alpha F4 15 0.865 Theta FC6 12 0.858 Alpha AF4 11 0.855 High beta F3 8 0.853 Alpha O2 14 0.852 Low beta AF4 12 0.851 Alpha FC6 12 0.851 Alpha FC6 15 0.849 Alpha AF4 10 0.845 Alpha F8 15 0.845 Alpha T8 14 0.844 Low beta F4 12 0.844 High beta F4 13 0.842 Alpha FC5 12 0.842 Alpha F4 13 0.841 High beta F4 14 0.841 High beta FC5 14 0.839 Alpha O2 13 0.839 Low beta F4 15 0.838 High beta F4 15 0.832 Alpha F8 8 0.83 Low beta AF3 12 0.83 Theta FC5 8 0.83 Low beta F4 10 0.829 Theta F8 8 0.828 Theta O2 12 0.826 Alpha T8 12 0.825 Alpha F7 14 0.824 Low beta FC6 12 0.822 Theta F7 8 0.821 Low beta F8 12 0.82 Theta P8 12 0.818 Alpha F8 13 0.818 Alpha O2 12 0.818

59 Low beta FC6 14 0.817 Theta FC5 12 0.817 Theta F3 14 0.816 High beta F3 13 0.814 Alpha F7 10 0.811 Alpha F3 8 0.81 Low beta FC5 13 0.809 Low beta AF4 14 0.807 Alpha FC6 11 0.806 Low beta F7 13 0.806 Low beta F4 8 0.805 Alpha F3 15 0.803 Theta F8 10 0.803 Low beta F3 8 0.802 High beta FC5 12 0.802 Low beta FC5 12 0.8 Low beta F4 14 0.8 Theta F7 12 0.8 Theta FC6 8 0.8 Alpha O2 15 0.799 Low beta F4 13 0.798 Alpha F3 10 0.798 Low beta AF3 13 0.795 Low beta F3 11 0.795 Low beta FC6 13 0.794 Low beta AF3 8 0.794 Theta AF4 8 0.793 Low beta AF3 14 0.793 Alpha F7 12 0.793 Theta FC5 13 0.793 Theta FC6 14 0.793 High beta F3 12 0.792 Low beta F8 14 0.792 High beta FC5 13 0.792 Alpha F8 12 0.792 Alpha F3 13 0.791 Theta T8 14 0.791 Theta O2 13 0.79 High beta F3 14 0.79 Low beta FC5 15 0.79 Low beta F8 8 0.789 Low beta AF4 8 0.786 Theta F8 14 0.785

60 Alpha AF3 11 0.785 High beta FC5 15 0.785 Theta O2 15 0.784 Theta F7 14 0.783 Low beta AF4 10 0.783 Low beta F3 10 0.783 Low beta F8 13 0.782 Alpha F8 11 0.782 Theta F8 12 0.782 Alpha F4 8 0.781 Alpha FC5 14 0.78 Low beta F7 12 0.779 Low beta FC5 8 0.778 Alpha F7 13 0.778 Theta P8 15 0.776 Theta F7 10 0.776 High beta F3 10 0.776 Theta FC5 14 0.776 Low beta AF4 11 0.774 Theta T8 8 0.774 Alpha FC6 8 0.773 Alpha P8 10 0.773 High beta F4 10 0.772 Alpha O1 14 0.772 Low beta FC5 14 0.772 High beta O2 15 0.771 Alpha P8 8 0.771 Alpha O2 8 0.771 Alpha FC5 13 0.768 Theta AF3 8 0.768 Theta T8 10 0.767 Low beta AF4 15 0.766 High beta F3 15 0.765 Alpha F7 15 0.765 Low beta FC6 8 0.765 Low beta FC6 10 0.764 High beta AF4 12 0.764 Alpha F7 8 0.763 High beta AF4 10 0.762 Alpha FC5 10 0.762 High beta FC5 8 0.762 Low beta F8 10 0.761 Theta FC5 15 0.76

61 High beta AF3 8 0.76 Theta FC5 10 0.759 Low beta FC6 15 0.759 Alpha F4 11 0.758 Alpha FC5 15 0.756 Theta F7 15 0.756 Low beta F8 11 0.756 Alpha FC5 8 0.754 Alpha P8 11 0.751 Alpha AF3 6 0.75 High beta F4 8 0.75 Theta F4 12 0.749 Theta F4 14 0.749 Theta F3 15 0.748 Theta FC6 15 0.747 High beta F4 12 0.747 High beta FC6 8 0.746 Alpha AF4 6 0.746 Theta P8 13 0.745 Low beta AF3 15 0.744 Theta AF3 10 0.744 High beta FC5 10 0.743 Theta F3 12 0.743 Alpha F8 10 0.743 Theta FC6 10 0.743 Low beta F7 15 0.742 Low beta F4 11 0.742 Low beta AF3 11 0.74 Alpha F3 11 0.74 Theta O1 14 0.74 Alpha AF4 9 0.739 Theta P8 10 0.738 Theta P8 8 0.738 Alpha FC6 10 0.736 Alpha AF3 9 0.736 Theta O2 10 0.736 Theta F7 6 0.734 Low beta AF3 9 0.734 High beta AF4 8 0.733 High beta AF3 10 0.732 Theta O2 8 0.732 Alpha FC5 11 0.731 Theta F8 15 0.731

62 High beta AF3 12 0.729 Low beta F7 8 0.728 Theta AF4 10 0.728 Low beta T8 13 0.727 High beta P8 15 0.725 Low beta F7 10 0.725 High beta F7 13 0.725 Low beta O1 14 0.725 High beta F3 11 0.724 Low beta AF3 10 0.724 Alpha F4 10 0.724 High beta F8 15 0.724 Low beta O1 13 0.723 High beta AF4 15 0.723 High beta O2 14 0.722 Theta AF3 15 0.721 Low beta F8 15 0.717 Alpha O2 10 0.716 High beta F8 12 0.715 Alpha T8 8 0.715 Alpha O1 13 0.715 Alpha T8 13 0.714 High beta AF4 14 0.714 Low beta F7 14 0.714 Theta AF3 6 0.713 High beta F7 15 0.712 Theta AF3 12 0.711 Low beta F7 11 0.711 High beta P8 13 0.708 Theta AF4 12 0.707 High beta F8 8 0.707 Theta AF4 13 0.707 High beta AF3 13 0.706 High beta AF4 13 0.705 Low beta FC5 10 0.705 Theta F3 10 0.705 Theta F3 13 0.704 Alpha O1 15 0.703 High beta F8 13 0.703 High beta F7 12 0.703 Theta FC5 6 0.703 Alpha F7 6 0.702 Low beta T8 8 0.701

63 Theta FC6 13 0.701 Theta F7 13 0.699 Theta T8 15 0.699 Theta F4 15 0.699 Theta FC5 11 0.698 Alpha F7 11 0.696 Theta F4 8 0.696 Theta F4 13 0.695 Theta O1 13 0.695 High beta P8 14 0.695 Low beta F4 9 0.694 Low beta FC5 11 0.694 Alpha F3 6 0.694 Theta O1 12 0.693 Alpha O2 11 0.692 Alpha F7 9 0.691 High beta O2 13 0.691 Theta P8 11 0.691 High beta FC6 15 0.69 Low beta O1 15 0.69 High beta F4 11 0.69 Low beta T8 15 0.69 Alpha T8 10 0.689 Low beta FC6 11 0.688 Low beta O1 11 0.687 High beta FC6 13 0.687 Theta AF4 15 0.686 High beta F7 11 0.685 Theta AF3 13 0.684 High beta FC6 12 0.683 Alpha F8 6 0.683 Low beta F3 9 0.681 Theta P7 12 0.681 High beta F7 8 0.677 High beta AF3 14 0.677 Alpha T8 15 0.676 Theta O2 11 0.676 High beta O2 8 0.676 Low beta T8 12 0.675 Theta P7 10 0.675 Low beta O2 8 0.675 Theta O1 8 0.674 Theta AF3 14 0.674

64 High beta AF3 15 0.673 High beta FC6 10 0.673 Theta AF4 14 0.673 Theta F8 6 0.673 Alpha O1 12 0.672 High beta P8 12 0.672 Low beta F8 9 0.67 Theta T8 13 0.669 Alpha F4 6 0.667 Theta T8 11 0.666 Theta F4 10 0.666 Low beta T8 14 0.666 Theta F3 8 0.665 Low beta P8 13 0.665 Theta FC6 6 0.665 Low beta O2 14 0.664 Low beta O2 12 0.664 High beta O2 12 0.663 Alpha F8 9 0.662 Theta T7 14 0.661 Theta F8 13 0.66 Low beta AF4 6 0.66 Theta O1 10 0.66 Theta P7 8 0.659 High beta FC5 11 0.658 Theta AF4 6 0.658 High beta AF4 11 0.656 Alpha F3 9 0.655 Low beta O2 13 0.655 Low beta O1 8 0.652 Low beta AF4 9 0.651 Low beta F4 6 0.649 Low beta P8 15 0.647 High beta F8 14 0.646 Alpha T8 11 0.644 High beta FC6 14 0.643 Alpha FC6 6 0.643 Low beta O1 12 0.642 High beta O1 15 0.642 Low beta F8 6 0.641 Theta T7 13 0.64 Theta F4 11 0.64 Theta P7 14 0.637

65 Alpha F4 9 0.637 Theta T7 12 0.637 High beta F7 14 0.636 Low beta AF3 6 0.635 Alpha FC6 9 0.635 High beta F8 10 0.634 Low beta T8 10 0.634 Low beta P8 14 0.633 Low beta F7 6 0.633 Theta P7 15 0.632 Alpha P8 9 0.631 Theta F3 11 0.63 Theta F7 11 0.629 High beta O2 10 0.628 Theta T7 15 0.626 Theta T7 8 0.625 Low beta O2 15 0.624 Low beta O2 10 0.623 High beta AF3 11 0.622 Theta P7 13 0.622 Alpha P8 6 0.621 High beta P8 8 0.621 High beta T8 12 0.62 High beta T7 15 0.62 Low beta FC6 6 0.62 Theta O2 6 0.618 Theta FC5 9 0.617 Low beta T8 11 0.617 High beta O2 11 0.617 High beta F4 9 0.617 High beta O1 14 0.616 Theta FC6 11 0.615 High beta F3 6 0.615 High beta F7 6 0.614 Theta P8 6 0.614 Theta F8 9 0.614 High beta F7 10 0.613 Low beta FC5 6 0.613 High beta O1 12 0.613 Alpha O2 6 0.612 Theta FC6 9 0.609 Alpha FC5 6 0.609 Theta T8 6 0.607

66 High beta F4 6 0.606 Alpha O1 11 0.606 Theta AF3 11 0.605 Theta AF4 11 0.604 High beta F8 11 0.604 Low beta F7 9 0.604 Low beta O1 10 0.602 Theta O2 9 0.599 Theta F4 6 0.598 Low beta P8 12 0.598 Alpha O2 9 0.597 Low beta F3 6 0.596 Theta F3 9 0.594 Low beta P8 8 0.593 High beta FC5 6 0.592 High beta FC6 6 0.591 Theta F8 11 0.59 Low beta FC5 9 0.585 Theta P8 9 0.585 Theta O1 15 0.584 High beta T7 14 0.584 Theta F7 9 0.582 Theta O1 11 0.581 Low beta FC6 9 0.581 High beta O1 13 0.58 High beta F8 6 0.575 High beta P8 11 0.573 Alpha T8 9 0.572 Theta T7 10 0.571 High beta AF3 6 0.57 High beta FC6 11 0.569 High beta F7 9 0.567 Low beta O1 9 0.566 High beta T8 14 0.565 Theta O1 9 0.565 High beta T8 8 0.564 Alpha FC5 9 0.562 Low beta O2 6 0.562 Low beta O1 6 0.562 High beta T7 13 0.562 High beta O1 8 0.56 Theta F4 9 0.56 Low beta T8 9 0.559

67 Alpha T8 6 0.558 Alpha O1 10 0.557 High beta T8 11 0.555 High beta AF3 9 0.555 High beta FC5 9 0.555 Theta F3 6 0.555 High beta AF4 6 0.555 Theta AF3 9 0.554 High beta F3 9 0.552 Theta AF4 9 0.551 High beta AF4 9 0.551 High beta T8 10 0.55 Theta T8 9 0.549 Low beta O2 9 0.548 High beta P8 10 0.548 Theta P7 9 0.548 High beta T7 12 0.547 Low beta T8 6 0.547 High beta P8 9 0.543 High beta T8 15 0.543 Alpha O1 8 0.542 High beta T8 13 0.54 Theta T7 11 0.538 Theta P7 11 0.537 Low beta O2 11 0.537 High beta FC6 9 0.534 High beta O2 6 0.534 Theta T7 9 0.532 Theta P7 6 0.532 Low beta P8 11 0.531 Low beta P8 9 0.531 High beta O1 10 0.53 High beta T7 8 0.528 High beta O2 9 0.525 Theta O1 6 0.522 Alpha O1 9 0.517 High beta F8 9 0.513 High beta T7 11 0.508 Low beta P8 10 0.507 High beta T8 9 0.504 High beta T8 6 0.503 Theta FC6 7 0.501 Alpha F3 7 0.493

68 Theta F4 7 0.49 Theta T7 6 0.489 Alpha O1 6 0.486 Low beta P8 6 0.481 Alpha AF3 7 0.473 Alpha AF4 7 0.471 Low beta AF3 7 0.471 Theta O1 7 0.467 High beta T7 10 0.462 Low beta T8 7 0.462 Alpha T8 7 0.461 Theta O2 7 0.459 Alpha F7 7 0.459 Theta AF3 4 0.451 High beta P8 6 0.45 High beta O1 9 0.449 High beta O1 11 0.448 Alpha AF4 4 0.448 Alpha FC5 7 0.446 Alpha F4 7 0.445 High beta O1 6 0.44 Alpha P8 7 0.44 Theta P8 7 0.438 High beta T7 9 0.436 Theta F8 7 0.436 High beta F8 7 0.436 Low beta AF4 7 0.435 High beta O2 7 0.433 Theta T7 7 0.432 Alpha O2 7 0.432 Theta P7 7 0.432 Theta F7 7 0.431 High beta FC6 7 0.43 Theta FC5 7 0.428 Low beta F7 7 0.424 Low beta F4 7 0.423 Theta F8 4 0.422 Alpha O1 7 0.422 Theta AF3 7 0.421 Theta F3 7 0.42 Alpha F8 7 0.417 High beta FC5 7 0.417 Theta T8 7 0.415

69 Low beta FC5 7 0.411 Theta FC5 4 0.411 Theta AF4 7 0.409 Low beta F8 7 0.407 Theta AF4 4 0.407 High beta T7 6 0.406 Alpha AF3 4 0.399 Low beta FC6 7 0.397 High beta F7 7 0.396 High beta AF4 7 0.393 Low beta F3 7 0.391 High beta T8 7 0.389 Alpha F7 4 0.386 Low beta P8 7 0.382 Low beta O2 7 0.382 Theta F7 4 0.381 Theta FC6 4 0.373 Theta T8 4 0.369 Alpha F8 4 0.369 Alpha FC6 7 0.368 High beta AF3 7 0.365 Theta O2 4 0.364 Theta F4 4 0.364 High beta P8 7 0.362 High beta T7 7 0.361 Theta P8 4 0.359 High beta F8 4 0.355 Alpha FC6 4 0.353 High beta F4 7 0.349 Low beta F8 4 0.349 High beta F3 7 0.346 Alpha F4 4 0.346 Alpha F3 4 0.344 Low beta F7 4 0.344 Low beta AF3 4 0.343 Theta F3 4 0.333 Low beta F4 4 0.332 Low beta FC6 4 0.33 Low beta O1 7 0.325 Alpha P8 4 0.323 Alpha FC5 4 0.317 Low beta F3 4 0.305 Theta P7 4 0.303

70 Low beta T8 4 0.303 High beta FC6 4 0.302 Low beta AF4 4 0.299 High beta AF3 4 0.293 High beta AF4 4 0.292 Low beta FC5 4 0.292 Alpha T8 4 0.289 Alpha O2 4 0.289 High beta F3 4 0.287 Alpha O1 4 0.283 High beta O1 7 0.283 High beta F4 4 0.283 High beta F7 4 0.279 Theta O1 4 0.278 Low beta P8 4 0.268 High beta FC5 4 0.265 Theta T7 4 0.26 Low beta O2 4 0.256 High beta O2 4 0.252 Alpha F4 5 0.252 Theta O1 5 0.243 High beta T8 4 0.235 Low beta O1 4 0.226 High beta O1 4 0.206 Theta FC5 5 0.206 High beta P8 4 0.194 Alpha FC6 5 0.192 High beta T7 4 0.188 High beta T7 5 0.184 Alpha AF4 5 0.183 Theta FC6 5 0.181 Alpha AF3 5 0.18 Theta T7 5 0.176 Theta P8 5 0.175 High beta O1 5 0.172 High beta F8 5 0.167 Theta F4 5 0.166 Theta AF3 5 0.165 Theta P7 5 0.162 Theta AF4 5 0.153 Alpha F7 5 0.151 Theta F8 5 0.15 High beta F3 5 0.149

71 Theta F7 5 0.148 High beta FC6 5 0.147 Alpha F3 5 0.146 Theta O2 5 0.145 Alpha O1 5 0.144 Low beta T8 5 0.144 Alpha T8 5 0.141 High beta F7 5 0.139 Alpha O2 5 0.139 Theta F3 5 0.137 Theta T8 5 0.136 Low beta F7 5 0.136 High beta AF4 5 0.134 High beta T8 5 0.134 Alpha P8 5 0.131 Alpha FC5 5 0.13 Low beta P8 5 0.129 High beta O2 5 0.129 High beta AF3 5 0.127 Low beta F4 5 0.127 High beta FC5 5 0.125 High beta P8 5 0.123 Alpha F8 5 0.116 Low beta F3 5 0.105 Low beta FC6 5 0.099 Low beta O2 5 0.098 High beta F4 5 0.098 Low beta AF3 5 0.093 Low beta O1 5 0.086 Low beta AF4 5 0.081 Low beta FC5 5 0.076 Low beta F8 5 0.064 Theta F7 2 0.014 Theta F3 2 0.014 Theta FC5 2 0.014 Theta T7 2 0.014 Theta P7 2 0.014 Theta O2 2 0.014 Theta P8 2 0.014 Theta T8 2 0.014 Theta FC6 2 0.014 Theta F4 2 0.014 Theta F8 2 0.014

72 Theta AF4 2 0.014 Alpha AF3 2 0.014 Alpha F7 2 0.014 Alpha F3 2 0.014 Alpha FC5 2 0.014 Alpha O1 2 0.014 Alpha P8 2 0.014 Alpha T8 2 0.014 Alpha FC6 2 0.014 Alpha F4 2 0.014 Alpha F8 2 0.014 Alpha AF4 2 0.014 Low beta AF3 2 0.014 Low beta F7 2 0.014 Low beta F3 2 0.014 Low beta FC5 2 0.014 Low beta O1 2 0.014 Low beta O2 2 0.014 Low beta T8 2 0.014 Low beta FC6 2 0.014 Low beta F4 2 0.014 Low beta F8 2 0.014 Low beta AF4 2 0.014 High beta AF3 2 0.014 High beta F7 2 0.014 High beta F3 2 0.014 High beta FC5 2 0.014 High beta T7 2 0.014 High beta O1 2 0.014 High beta O2 2 0.014 High beta P8 2 0.014 High beta T8 2 0.014 High beta FC6 2 0.014 High beta F4 2 0.014 High beta F8 2 0.014 High beta AF4 2 0.014 Low beta P8 2 0.014 Theta O1 2 0.014 Alpha O2 2 0.014 Theta AF3 2 0.014 High beta T7 3 -0.129 Theta AF3 3 -0.134 High beta O1 3 -0.144

73 Theta F7 3 -0.146 Theta T7 3 -0.148 Theta FC5 3 -0.149 Theta F3 3 -0.15 Low beta FC6 3 -0.154 Alpha F4 3 -0.155 Theta F8 3 -0.155 High beta F8 3 -0.155 Low beta F8 3 -0.158 High beta AF4 3 -0.158 Alpha F3 3 -0.158 Theta FC6 3 -0.159 Theta P7 3 -0.161 High beta FC5 3 -0.162 Low beta F4 3 -0.164 High beta P8 3 -0.164 Theta O2 3 -0.166 Alpha T8 3 -0.167 High beta F7 3 -0.167 Theta O1 3 -0.168 Alpha FC6 3 -0.169 Low beta O2 3 -0.169 Alpha O2 3 -0.169 High beta F4 3 -0.173 Alpha AF3 3 -0.173 Low beta FC5 3 -0.174 High beta F3 3 -0.176 High beta FC6 3 -0.176 Alpha F8 3 -0.177 Low beta O1 3 -0.177 Theta F4 3 -0.177 Theta T8 3 -0.179 Theta AF4 3 -0.18 Alpha AF4 3 -0.181 High beta O2 3 -0.184 Alpha F7 3 -0.184 Alpha P8 3 -0.185 Theta P8 3 -0.186 High beta T8 3 -0.186 Low beta F7 3 -0.187 Low beta T8 3 -0.189 Low beta F3 3 -0.189 Low beta P8 3 -0.19

74 High beta AF3 3 -0.191 Alpha O1 3 -0.192 Low beta AF3 3 -0.198 Low beta AF4 3 -0.2 Alpha FC5 3 -0.201

Table 11: All selected features for the personalized SB 25 algorithm

Frequency band Location Feature Weight Theta T7 sd 1.0 Theta T7 dX 0.939 Theta FC5 dX 0.831 Theta P8 dX 0.818 Theta O1 dX 0.784 Theta O2 dX 0.727 Theta F4 dX 0.723 Theta T7 sdX 0.697 Theta P7 dX 0.641 High beta AF3 sd 0.538 Theta P8 sgX 0.449 Theta O1 sgX 0.436 Theta FC6 sd 0.421 Theta P7 sgX 0.398 Theta O2 sgX 0.389 Theta F4 sgX 0.374 High beta T7 sd 0.349 Theta P8 sdX 0.336 Alpha P8 sd 0.279 Theta O2 sdX 0.276 Theta O1 sdX 0.27 High beta P7 sd 0.263 Theta P7 sdX 0.262 Low beta O1 sd 0.26 Theta F4 sdX 0.258 Theta P8 sd 0.253 Theta FC5 sdX 0.249 Theta O1 sd 0.248 Theta FC5 sgX 0.245 High beta T7 dX 0.241 Alpha T8 dX 0.24 High beta AF3 dX 0.238

75 Low beta P7 sd 0.235 Alpha O1 dX 0.23 Alpha P7 dX 0.225 Theta P7 sd 0.218 Low beta P7 dX 0.207 High beta T7 sgX 0.206 Theta FC6 dX 0.2 Theta FC6 sgX 0.188 Low beta P7 sgX 0.185 Alpha O2 dX 0.176 High beta O2 sd 0.167 Theta FC5 sd 0.158 Alpha O2 sgX 0.156 Theta O2 sd 0.155 High beta T7 sdX 0.151 Theta F4 sd 0.15 Low beta O2 dX 0.143 Alpha O2 sd 0.141 Theta FC6 sdX 0.141 Low beta P7 sdX 0.136 Alpha T7 dX 0.131 Low beta FC5 dX 0.129 High beta AF3 sdX 0.126 Low beta O1 dX 0.124 Low beta F4 sd 0.123 High beta FC5 sd 0.12 Alpha O1 sgX 0.119 Alpha P7 sgX 0.117 Low beta AF4 sd 0.116 Alpha T8 sgX 0.116 Low beta T7 dX 0.114 Alpha P8 dX 0.112 High beta F3 sd 0.109 Alpha O2 sdX 0.109 Low beta O1 sgX 0.105 High beta P7 dX 0.102 Alpha P8 sgX 0.097 High beta O1 dX 0.094 Alpha O1 sdX 0.093 High beta P8 sd 0.092 High beta O1 sd 0.092 High beta AF3 sgX 0.091 High beta F3 dX 0.09

76 Theta T8 sgX 0.085 Theta T8 sd 0.085 High beta P7 sgX 0.083 Low beta O1 sdX 0.082 Theta T8 dX 0.082 Alpha P7 sdX 0.082 Low beta O2 sgX 0.079 Alpha P8 sdX 0.078 Alpha T8 sdX 0.077 Low beta FC5 sgX 0.074 Low beta FC6 sd 0.07 Alpha T7 sgX 0.069 High beta FC5 dX 0.069 Low beta T7 sd 0.068 Low beta T7 sgX 0.065 Alpha O1 sd 0.063 High beta O2 dX 0.06 Low beta FC5 sd 0.06 Theta T8 sdX 0.06 Low beta AF4 dX 0.06 High beta O1 sgX 0.06 High beta F7 sd 0.059 High beta P7 sdX 0.058 High beta F3 sgX 0.055 Low beta F7 sd 0.055 Low beta O2 sdX 0.055 Low beta O2 sd 0.054 Low beta F3 dX 0.052 Low beta P8 sd 0.052 Low beta FC5 sdX 0.052 Low beta T7 sdX 0.049 Low beta F4 dX 0.049 Low beta F7 dX 0.049 High beta FC5 sgX 0.046 Low beta F8 sd 0.045 Alpha T7 sdX 0.045 High beta P8 dX 0.045 High beta O2 sgX 0.042 High beta O1 sdX 0.042 Alpha AF4 sd 0.039 High beta F3 sdX 0.038 Low beta AF4 sgX 0.036 Alpha P7 sd 0.035

77 Alpha T8 sd 0.035 High beta F7 dX 0.035 Low beta T8 sd 0.034 Low beta F3 sgX 0.033 High beta FC5 sdX 0.032 High beta P8 sgX 0.031 Low beta F3 sd 0.031 Low beta F7 sgX 0.029 Low beta F4 sgX 0.029 Low beta FC6 dX 0.029 Alpha T7 sd 0.029 High beta O2 sdX 0.028 High beta F7 sgX 0.025 Low beta AF4 sdX 0.024 High beta P8 sdX 0.022 Low beta F7 sdX 0.022 Low beta FC6 sgX 0.021 Low beta F3 sdX 0.021 Low beta F4 sdX 0.021 Low beta P8 dX 0.02 Low beta T8 dX 0.019 Low beta F8 dX 0.018 Alpha F8 sd 0.017 High beta F7 sdX 0.017 Alpha AF4 dX 0.016 Low beta FC6 sdX 0.015 Low beta T8 sgX 0.014 Low beta F8 sgX 0.013 Low beta P8 sgX 0.011 Alpha AF4 sgX 0.011 Alpha F8 dX 0.011 Low beta T8 sdX 0.01 Low beta F8 sdX 0.009 Alpha F8 sgX 0.008 Low beta P8 sdX 0.007 Alpha AF4 sdX 0.007 Alpha F8 sdX 0.006 Alpha F3 mu 0.006 Alpha P8 mu 0.005 Theta FC5 mu 0.005 Alpha T7 mu 0.004 Theta T7 mu 0.004 Alpha F7 gX 0.003

78 Alpha F4 mu 0.002 Alpha O2 gX 0.002 Theta F3 gX 0.002 Theta F8 mu 0.002 Alpha O1 mu 0.002 Alpha FC5 gX 0.002 Alpha F8 gX 0.001 Alpha FC5 dX 0.001 Alpha T8 mu 0.001 Alpha FC5 mu 0.001 Theta FC6 mu 0.001 Theta FC5 gX 0.001 Alpha F3 gX 0.001 Low beta AF3 gX 0.001 Theta F7 mu 0.0 Theta AF4 dX 0.0 Alpha AF4 mu 0.0 Alpha P7 gX 0.0 Low beta O1 gX -0.0 Theta F3 mu -0.0 Alpha P8 gX -0.001 Alpha FC6 gX -0.001 Low beta O2 mu -0.001 Low beta FC5 mu -0.001 Alpha O2 mu -0.001 Low beta F3 gX -0.001 Alpha FC5 sdX -0.001 Theta T8 gX -0.001 Low beta T8 gX -0.001 Theta F4 mu -0.001 Alpha T7 gX -0.002 Theta AF3 gX -0.002 Alpha O1 gX -0.002 Theta F4 gX -0.002 Alpha FC5 sgX -0.002 Theta AF4 mu -0.002 Theta F7 gX -0.002 Alpha P7 mu -0.002 Low beta F7 gX -0.002 Theta T8 mu -0.002 High beta P8 gX -0.002 Low beta AF3 mu -0.002 Alpha F7 mu -0.003

79 Alpha AF3 mu -0.003 Alpha AF4 gX -0.003 Theta F8 gX -0.003 High beta T8 mu -0.003 High beta O1 gX -0.003 Low beta F7 mu -0.003 Low beta FC6 gX -0.003 Theta AF4 gX -0.003 Low beta FC6 mu -0.004 Alpha F8 mu -0.004 Theta AF4 sgX -0.004 Low beta O1 mu -0.004 Theta O1 gX -0.004 Low beta AF4 mu -0.004 Theta P8 gX -0.004 Low beta P8 gX -0.004 High beta AF3 mu -0.004 Low beta F3 mu -0.004 Low beta P7 gX -0.004 Low beta AF3 sdX -0.004 Low beta FC5 gX -0.004 Alpha AF3 gX -0.004 Low beta F4 mu -0.004 Low beta F8 gX -0.004 Theta FC6 gX -0.004 Low beta P8 mu -0.004 Low beta T7 gX -0.004 Low beta T8 mu -0.004 High beta F3 mu -0.004 Low beta F4 gX -0.004 Alpha F4 gX -0.004 Low beta P7 mu -0.005 Theta T7 gX -0.005 High beta F7 gX -0.005 High beta AF3 gX -0.005 High beta O2 mu -0.005 Low beta O2 gX -0.005 Theta O1 mu -0.005 Low beta T7 mu -0.005 Theta P8 mu -0.005 Theta O2 mu -0.005 Alpha FC5 sd -0.005 Low beta F8 mu -0.006

80 Theta P7 gX -0.006 Theta O2 gX -0.006 High beta P7 gX -0.006 High beta FC5 gX -0.006 Low beta AF4 gX -0.006 High beta F7 mu -0.006 High beta O1 mu -0.007 Alpha F4 dX -0.007 High beta O2 gX -0.007 Theta P7 mu -0.007 High beta P8 mu -0.007 High beta T7 gX -0.007 Alpha FC6 mu -0.007 Theta AF4 sdX -0.007 Alpha T8 gX -0.007 Low beta AF3 sgX -0.008 High beta P7 mu -0.008 Alpha F4 sdX -0.009 High beta T7 mu -0.01 Alpha F4 sgX -0.01 Low beta AF3 dX -0.011 High beta F3 gX -0.012 Low beta AF3 sd -0.013 High beta FC5 mu -0.014 Alpha AF3 sdX -0.018 Alpha F4 sd -0.022 Theta AF3 sdX -0.023 Alpha AF3 dX -0.024 Alpha AF3 sgX -0.024 Theta AF3 sgX -0.026 Alpha F7 sdX -0.028 Theta AF3 dX -0.03 Alpha FC6 sdX -0.034 Alpha F7 sgX -0.036 Alpha AF3 sd -0.037 Alpha F7 dX -0.041 Alpha F3 sdX -0.044 Alpha FC6 sgX -0.045 Alpha FC6 dX -0.046 Alpha F7 sd -0.05 Theta AF4 sd -0.054 Alpha F3 sgX -0.061 Alpha FC6 sd -0.065

81 Theta F8 sdX -0.073 Theta F7 sdX -0.079 Alpha F3 dX -0.082 Theta AF3 sd -0.083 Theta F8 dX -0.093 Theta F8 sgX -0.096 Theta F7 sgX -0.102 Alpha F3 sd -0.109 Theta F7 dX -0.112 Theta F3 sdX -0.117 Theta F3 sgX -0.172 Theta F7 sd -0.178 Theta F8 sd -0.178 Theta F3 dX -0.216 Theta F3 sd -0.406 Theta T7 sgX -1.039