Classify Frustration Using a Consumer Grade EEG Device For Neuromarketing Purposes Nicolette Stassen April 16, 2017 Abstract Algorithms that may classify emotions such as frustration using a con- sumer grade EEG device provide possibilities in the field of neuromarket- ing. This paper will focus on how machine learning techniques may be used to classify frustration using participants' brainwaves as input. Data has been collected using an Emotiv EPOC+ headset while 31 subjects played an adapted coin collector game. The data is analyzed using sev- eral feature extraction and machine learning techniques. Although more research is needed, reliable models have been found for both testing on an independent test set, and for testing on only a small amount of data from a subject. These models may be implemented for neuromarketing purposes. Keywords. EEG, emotion recognition, classifying frustration, ma- chine learning 1 1 Introduction Neuromarketing research is used increasingly by companies in order to ana- lyze the reactions of potential customers on their advertisements or websites. Normal marketing research uses surveys to ask customers post hoc about their experience, whereas neuromarketing research registers subconscious emotions in real time. To recognize and classify these emotions, algorithms are becom- ing increasingly important. Which emotions influence customer experience and buyer's intent are not known, but positive and negative emotions with high arousal are known to have an effect (Kim & Lennon, 2013). Within neuromar- keting research, negative emotions with high arousal, such as frustration, are used to determine deterrents in advertisements. Insights from users' affects can be used to improve the effectiveness of said adverts. Neuromarketing research focuses on measuring brain activity as determining personal affects using physiological measurements, such as skin conductance, is difficult. These measurements are multifaceted phenomenons and may show elevation in sweating during frustration, as well as in other emotional states (Figner, Murphy, et al., 2011). Research within the neuromarketing field tries to find the consumer's unconscious feelings by analyzing the reactions of the brain on several advertisements. Techniques such as functional magnetic reso- nance imaging (fMRI) and electroencephalography (EEG) may be used to mea- sure these unconscious feelings. fMRI measures changes in brain blood flow whereas EEG measures electronic activity in the brain. Both methods are used in neuromarketing research. For example, Neurensics is a neuromarketing company that mainly uses fMRI to record the human brain and how people react to several commercials 1. It does so as fMRI can measure the deep neural structures involved in emotion processing. Braingineers is a neuromarketing company mainly using EEG to record the human brain 2. EEG measures more superficial parts of the brain, but has several advantages over fMRI. Firstly, EEG hardware costs significantly less than fMRI hardware. Secondly, EEG hardware is lightweight and portable. Thirdly, fMRI exposes participants to high-intensity magnetic fields and EEG does not. Lastly, EEG has a higher temporal resolution, which makes it very suitable for neuromarketing purposes. In contrast to fMRI, it can pinpoint a moment of emotion to a much smaller time window and find the exact aspects of a certain commercial that might be frustrating to a consumer. However, the disadvantage of EEG is that it is not yet certain whether it can accurately predict emotions, as it monitors more superficial parts of the brain. Emotion recognition based on EEG is a relatively new research field in the areas of affective computing and neuromarketing. EEG devices measure brain- waves. Professional EEG devices with up to 128 channels can be used to de- termine frustration and other personal affects (Reuderink, M¨uhl,& Poel, 2013; Marox et al., 2001). However, professional EEG devices are inconvenient to 1http://www.neurensics.nl/ 2http://www.braingineers.com/ 2 use, as the electrodes have a higher chance of being placed in the wrong loca- tions and the participant's hair must have been washed right before the study. These restrictions make them unpractical for neuromarketing purposes. Con- sumer EEG devices, such as the Emotiv EPOC+ 3, consist of only 14 channels, excluding two reference channels, and are easy to set up. Until now, only few studies report being able to classify emotions using consumer EEG devices, and even fewer have successfully classified frustration (Tarnowski, Ko lodziej, Ma- jkowski, & Rak, n.d.; Garrett, Peterson, Anderson, & Thaut, 2003; F´elix,2011; Liu & Sourina, 2014; Petrantonakis & Hadjileontiadis, 2010b). Of these studies, frustration has been classified with a highest found accuracy of 63%. The Emo- tiv company provides detection algorithms to measure six different emotional states, among others frustration. They scale the output of their algorithms based on historical patterns for each individual user, therefore it takes a few hours for the system to settle down for a new subject 4. This makes the Emotiv algorithms not ideal for neuromarketing purposes. This research project will focus on whether it is possible to classify frustra- tion using a consumer grade EEG device. The Emotiv EPOC+ EEG device will be used to measure brainwaves and a machine learning algorithm will be designed to classify individual's frustration levels. Frustration will be induced by playing a simple coin collector game that is to be developed using the Unity game engine 5. This game will do so by making the user's controls unrespon- sive at fixed time intervals. Whether frustration will have been induced will be verified using pupil dilation, keyboard loggers, and behavioural analysis. The goal of this research is twofold. Firstly, it is to design an accurate algorithm that may be used to measure frustration in new subjects, without the need to refit the model first. As brainwaves differ between participants, it is not known whether it is possible to design a general model to predict frustration (Lake & Bryden, 1976; Springer & Deutsch, 1998; Bourne, 2005). Therefore, this research does not only aim to design an algorithm to measure frustration in new subjects, without the need to refit the model first, but also to design an algorithm that can classify frustration using only a few seconds of data to train on. Several challenges may be faced during this process, as classifying frustration using consumer grade EEG devices and the effects of frustrating stimuli on brainwaves are relatively unknown. Other challenges include the adaptation of feature extraction classifiers to allow for recognition of data patterns over multiple sensors and over time, as well as designing software that would connect the measurement devices in a way that does not obstruct the collection of data. These problems may be solved using serial analysis of sensor data columns over different lengths and the use of multiple computers running on synchronized software respectively. The following section will provide background information on frustration, EEG, brainwaves, and machine learning techniques. The subsequent sections 3https://www.emotiv.com/epoc/ 4https://emotiv.zendesk.com/hc/en-us/articles/200782279-How-do-you-measure-emotions-in-the-first-place-so-you-can-compare-the-outputs-and-come-up-with-a-number- 5https://unity3d.com/ 3 will discuss the experimental setup, methodology, and results from the study to collect the data for the algorithm. The following sections will explain the exploratory data analysis, pre-processing of the data, and feature extraction techniques used in this research. Afterwards, the used feature selection tech- niques and machine learning classifiers will be explained and the results of the algorithms will be presented. Finally, these results will be discussed and a con- clusion will be drawn. 4 2 Background 2.1 Frustration Frustration is an emotion that, in psychology, is defined as a response to op- position and can be defined using the valence-arousal model (Schlosberg, 1954; Lang, Greenwald, Bradley, & Hamm, 1993). Frustration is located in the model as having high arousal and negative valence. It is related to anger and dis- appointment and arises when an individual is unable to reach his or her goal (De Botton, 2001; Berkowitz, 1989). Two types of frustration exist, namely internal and external frustration (Brisset & Nowicki, 1973). Internal frustration arises when an individual lacks confidence to fulfill personal goals or desires. External frustration involves factors outside of the individual's control, such as traffic. Individuals may cope with frustration in different ways. For example, some may use violence, whereas others may use passive-aggressive behaviour. Though individuals handle frustration in different ways, most responses involve aggres- sion (Berkowitz, 1989). Frustration is difficult to measure, as individuals have different tolerances for frustration and everyone shows a different reaction to frustrating events (Szasz, Szentagotai, & Hofmann, 2011). Frustration is mea- surable for approximately nine seconds after induction (Heraz, Daouda, & Fras- son, 2008). However, as individuals react differently to frustration, this duration may differ per individual. Although measuring frustration is difficult, it is thought to be measurable physiologically (Yu, Mobbs, Seymour, Rowe, & Calder, 2014; Bradley, Miccoli, Escrig, & Lang, 2008; Feild, Allan, & Jones, 2010). Even though physiological responses of the body, such as an