Positive and Negative Emotion Classification Based on Multi-Channel

ORIGINAL RESEARCH published: 26 August 2021 doi: 10.3389/fnbeh.2021.720451

Positive and Negative Emotion Classiﬁcation Based on Multi-channel

Fangfang Long 1†, Shanguang Zhao 2†, Xin Wei 3,4*, Siew-Cheok Ng 5, Xiaoli Ni 3, Aiping Chi 6, Peng Fang 7*, Weigang Zeng 4 and Bokun Wei 8

1Department of Psychology, Nanjing University, Nanjing, China, 2Centre for Sport and Exercise Sciences, University of Malaya, Kuala Lumpur, Malaysia, 3Institute of Social Psychology, School of Humanities and Social Sciences, Xi’an Jiaotong University, Xi’an, China, 4Key & Core Technology Innovation Institute of the Greater Bay Area, Guangdong, China, 5Faculty of Engineering, University of Malaya, Kuala Lumpur, Malaysia, 6School of Sports, Shaanxi Normal University, Xi’an, China, 7Department of the Psychology of Military Medicine, Air Force Medical University, Xi’an, China, 8Xi’an Middle School of Shaanxi Province, Xi’an, China

The EEG features of different emotions were extracted based on multi-channel and forehead channels in this study. The EEG signals of 26 subjects were collected by the

Edited by: emotional video evoked method. The results show that the energy ratio and differential Eva M. Marco, entropy of the frequency band can be used to classify positive and negative emotions Complutense University of Madrid, Spain effectively, and the best effect can be achieved by using an SVM classiﬁer. When only the

Reviewed by: forehead and forehead signals are used, the highest classiﬁcation accuracy can reach Vittorio B. Lippi, 66%. When the data of all channels are used, the highest accuracy of the model can University of Freiburg Medical Center, reach 82%. After channel selection, the best model of this study can be obtained. The Germany Fatima Megala Nathan, accuracy is more than 86%. Yale-NUS College, Singapore Keywords: EEG, emotion classiﬁcation, support vector machine, decision tree, back propagation neural network, *Correspondence: k-nearest neighbor Xin Wei [email protected] Peng Fang [email protected] INTRODUCTION

† These authors have contributed Emotions play a critical role in everyday life, reflecting a person’s current physical and mental equally to this work state and significantly impacting cognition, communication, and decision-making. Emotions are Specialty section: generally considered to have two dimensions: arousal and valence, with arousal referring to the This article was submitted to Emotion Regulation and Processing, emotion’s intensity; valence referring to the specific emotional content, divided into positive, a section of the journal negative, and neutral feelings (Kim et al., 2013; Bailen et al., 2019). Positive emotions can enhance Frontiers in Behavioral Neuroscience subjective well-being and promote physical and mental health, while persistent negative emotions will affect people’s physical and mental health and work status (Gupta, 2019). Different emotions Received: 04 June 2021 arise in response to external environmental stimuli and are accompanied by changes in personal Accepted: 29 July 2021 representations and psychological reactions, measured and identified by scientific methods Published: 26 August 2021 (Wolf, 2015). Citation: Previous studies have shown that many signals enable us to identify emotions. The most intuitive Long F, Zhao S, Wei X, Ng S-C, Ni X, expression, voice, posture signals, EEG, ECG, EMG, breathing, and other physiological signals can Chi A, Fang P, Zeng W and Wei B also measure an emotion. Among many signals that can reflect emotional changes, EEG, with high (2021) Positive and Negative Emotion temporal resolution and non-artifactual characteristics, has been valued by many researchers and Classiﬁcation Based on Multi-channel. is a standard method for emotion recognition. There are some features in EEG signals, which have Front. Behav. Neurosci. 15:720451. a strong ability of emotion classification. Petrantonakis and Hadjileontiadis(2010) proposed using doi: 10.3389/fnbeh.2021.720451 higher-order crossover features to extract EEG features, tested four different classifiers, and finally

Frontiers in Behavioral Neuroscience| www.frontiersin.org 1 August 2021 | Volume 15 | Article 720451 Long et al. Positive and Negative Emotion Classiﬁcation implemented a robust emotion classification method. Chen the classification effect? Which brain region will extract the et al.(2019) verified that the feature of differential entropy features to achieve the best classification effect? Third, there significantly affects sentiment classification. In the distribution is a tendency to use as few channels as possible to identify of brain regions related to emotion classification, there are some emotions for portability. One study found that when using only differences in the response of different brain regions to different the forehead electrode (Fp1 and Fp2) and using the gradient emotions. However, most studies on emotion classification lifting Decision Tree (DT) algorithm to classify happiness and were based on multi-channel EEG signals (Gonzalez et al., sadness, its accuracy can also reach 95.78% (Al-Nafjan et al., 2019; Goshvarpour and Goshvarpour, 2019). Many valuable 2017). However, no studies have been conducted to compare the achievements have been obtained regarding the differences and effect of dual and multi-channel classification simultaneously. Is functions of different brain regions in emotion classification. For the classification effect of two-channel still better than that of example, a study shows that the prefrontal lobe and occipital lobe multi-channel under the same experimental setting? greatly contribute to emotion classification (Asghar et al., 2019). Based on the above problems in emotion recognition research, Firpi and Vogelstein(2011), in a feature selection research work, this study focused on emotions consisting of validity and it is concluded that electrodes F3, F4, T8 have the best effect on arousal when classifying emotions and focused on positive emotion classification. and negative emotions within these two categories. The EEG Although there have been many studies on emotion signals were extracted when the subjects watched positive and recognition of EEG signals, several problems are still not clear negative videos, and four classifiers were selected to identify in previous studies. First of all, most of the available data sets emotion classification. In this study, Support Vector Machine for emotion recognition use image, video, audio, and other (SVM), DT, Back Propagation Neural Network (BPNN), and ways to induce emotional changes. Some researchers used EEG k-Nearest Neighbor (kNN) algorithms were used to examine signals from subjects’ happiness, anger, sadness, and joy videos the classification effect of energy share and differential entropy to classify these four types of emotions in related studies (Calix features, the classification effect of differential entropy features et al., 2012). Other studies classify data based on emotions such in different frequency bands, and the classification effect as happiness, relaxation, grief, and fear (Alazrai et al., 2018). of differential entropy features in different brain regions. Therefore, the current criteria for emotion classification are Meanwhile, in the analysis channel, the emotion recognition varied due to the high complexity and abstractness of emotion is carried out on the multi-channel and prefrontal lobe itself. As a result, many researchers fail to reach a unified dual-channel data to explore whether there is a difference in the emotional classification standard when doing work related to classification accuracy. emotion recognition. In practice, some studies directly use the label of evoked data as the label of the final emotion classification, MATERIALS AND METHODS but it may exist under the negative emotion but does not induce the negative emotion. Therefore, there is confusion in the sample Participants classification at the beginning. Secondly, in the extraction of Twenty-six participants (age range to 18–20, M = 19, SD = 0.48; EEG signals, the previous research found the most relevant 50% female) were recruited through flyers. The subjects had features of emotion from EEG signals. At present, four kinds standard visual or corrected visual acuity, normal hearing, and of features have been used for the emotion classification of no significant emotional problems or psychiatric disorders by EEG signals: time-domain features, frequency domain features, the State-Trait Anxiety Inventory (STAI) and Beck Depression statistical features, and time-frequency domain features (Torres Inventory (BDI). No coffee or alcoholic beverages were et al., 2020). Although many of the above features have been consumed within 24 h before the start of the experiment. All reported, there is still no detailed research to show which EEG subjects signed an informed consent form and received some signal feature combinations are most significantly related to remuneration at the end of the experiment. If the subjects did emotion classification. Recently, researchers have proposed that not accept the film’s content during the experiment, they could differential entropy has a good classification effect in emotion choose not to watch the film or terminate the experiment. The classification. The characteristics of energy proportion and study protocol was approved by the Ethics Committee of the Xian differential entropy have significant effects on the two-channel Jiaotong University. emotion classification (Al-Nafjan et al., 2017). However, Duan et al.(2013) pointed out that in emotion classification, the degree Stimuli and Procedure of discrimination of signals in high-frequency bands is greater This study used videos that evoked emotion. We used the revised than that in low-frequency bands, indicating that differential dynamic video library by Deng et al.(2017), which contains eight entropy feature classification in different frequency bands is emotional states: happy, sad, and neutral. Each emotional state different still controversial. In addition, previous studies have consisted of eight video clips, a total of 64 video clips. The length shown that different brain regions choose to extract features, and of each video was 60 s. There was no significant difference in finally, there are differences in the effect of emotion classification invalidity and emotional arousal between videos of the same type. (Lin et al., 2009; Zhong et al., 2020). Since this study focused on the binary classification algorithm Therefore, a question worth studying is, when we use the for positive and negative emotions, four negative segments differential entropy extracted from different brain regions as and four positive segments were selected for the subjects to the feature of emotion classification, what is the difference in watch. During the collection, the subjects watched the video

Frontiers in Behavioral Neuroscience| www.frontiersin.org 2 August 2021 | Volume 15 | Article 720451 Long et al. Positive and Negative Emotion Classification of the corresponding emotion theme for 4 min. Each subject Feature Extraction was provided with an individual rating in the valence-arousal- The EEG signals after pre-processing were sliced into segments dominance-liking four dimensions, ranging from 1 to 9, one of 1 s length. For the whole data set, 5,456 samples can be being the smallest and nine being the largest. After watching sliced, including 2,756 positive emotions and 2,700 negative a video, the 9-point rating scale was used to evaluate their emotions. According to the features used in conventional EEG subjective emotional experience while watching the video. In signal analysis, without considering the specialization to deal this study, emotions were analyzed in terms of two dimensions: with the emotion classification problem, 59-dimensional features valence and arousal. If an individual’s score is greater than 4.5, were selected for this study, which is a total of 1888-dimensional the arousal/valence level is high; whereas if the individual’s score features for all 32 channels, and this set of features can be used as is less than 4.5, the arousal/valence level is low (Koelstra et al., the baseline features for this study. The 59-dimensional features 2012). The subjects took a 2-min break after the score was can be seen in Table 1. completed to help regain their calm. It has been shown that the energy share of the sub-band and the differential entropy feature for the emotion classification problem is more effective. The frequency band energy and its EEG Data Acquisition and Preprocessing percentage can be calculated directly by obtaining the spectrum The data set was collected in an anechoic darkroom at through the signal’s Fourier transform. Xi’an Jiaotong University and significantly reduced noise, Differential entropy is an extension of Shannon’s entropy, reverberation, and electromagnetic interference. Brain electric which is defined by the following equation (Shi et al., 2013): activity was measured from 32 channels using a modified 10- Z to 20-system electrode cap (Neuroscan Inc.). All EEGs were h(X) = − f (x) log f (x) dx continuously sampled at 1,000 Hz. An electrode was placed on X the forehead as the ground, and the nose’s tip was the recording reference. EEG was amplified using a 0.1–100 Hz bandpass. where X is the EEG signal time series and f(x) is the probability The vertical EOG was recorded with electrodes placed above density function of X. If the EEG sequence X obeys a normal and below the left eye, and the horizontal EOG was recorded distribution, then the differential entropy of the sequence is with the electrodes placed outboard of both eyes. All electrode 2 2 impedances were maintained below ten kΩ. Z 1 − (x−µ) 1 − (x−µ) h (X) = − √ e 2σ2 log √ e 2σ 2 Raw EEG data pre-processing was performed using EEGlab X 2πσ 2 2πσ 2 software (Version R2013b, San Diego, USA), an open-source 1 dx = log(2πeσ 2) toolbox running MATLAB environment (Version R2013b, 2 MathWorks, USA). Pre-processing of resting EEG data included average reference. Continuous EEG data were band-pass filtered Referring to the observation of Shi et al.(2013) on EEG between 0.5 and 45 Hz and a notch filter between 48 and signals, the EEG signals on commonly used frequency bands obey 2 52 Hz; These segments were then visually inspected to remove a normal distribution N(µ, σ ). Therefore, for a fixed frequency those with oculomotor or head movement artifacts. Data band i, the differential entropy can be calculated by the following were segmented into 2-s–epochs. Eye movement artifacts were equation: corrected using individual independent component analysis 1 h (X) = log(2πeσ 2) (ICA) by removing the corresponding components based on the i 2 i particular activation curve (Mennes et al., 2010). EEG epochs 2 contaminated by strong muscle artifacts and any EEG epochs where σi represents the variance of the EEG sequence X on with amplitude values exceeding ±80 µV at the electrodes were frequency band i. And the EEG sequence X in any frequency manually rejected. band i has no DC component, i.e., the mean value of X is 0. At this time, the variance of X can be estimated by using the following equation: Multi-channel Sentiment Classification Analysis Based on the conventional EEG data processing scheme, when N 2 1 X 2 using multi-channel EEG data for analysis, the data processing σî = xn N and analysis scheme framework in this study is shown in n = 1 Figure 1. {Xn} which is the sequence X. Also, by Parseval’s theorem, know that:

Sample Division N N In this study, the emotional state is divided according to the X 2 1 X 2 x = |Xk| = Pi 9-point rating scale, and the EEG clips that do not match i N n = 1 k = 1 the emotional state and video stimulus tags are deleted. After preprocessing, the pure EEG was divided into segments with a where {Xk} is the FFT result of {Xn}, Pi representing the length of 1 s. For the whole data set, 5,456 samples were divided, energy spectrum on frequency band i. From the above derivation, including 2,756 positive emotions and 2,700 negative emotions. it can be derived that:

Frontiers in Behavioral Neuroscience| www.frontiersin.org 3 August 2021 | Volume 15 | Article 720451 Long et al. Positive and Negative Emotion Classiﬁcation

FIGURE 1 | Flow chart of multi-channel emotion classiﬁcation algorithm.

TABLE 1 | Basic characteristics of EEG signals.

No. Features No. Features No. Features

1 Mean 21 High δ-wave energy 41 β-wave differential entropy 2 Squared difference 22 δ-wave energy 42 γ-wave differential entropy 3 Standard deviation 23 Theta wave energy 43 Total frequency band center of gravity frequency 4 Maximum value 24 Low α wave energy 44 δ-wave center of gravity frequency 5 Minimum value 25 Highαwave energy 45 θ wave center of gravity frequency 6 Median 26 αwave energy 46 α-wave center of gravity frequency 7 Skewness 27 Low β wave energy 47 β-wave center of gravity frequency 8 Kurtosis 28 Highβwave energy 48 γ-wave center of gravity frequency 9 Zero crossover value 29 βwave energy 49 Total frequency band frequency variability 10 Amplitude 30 Low γ wave energy 50 δ-wave frequency variability 11 First-order Difference 31 Highγwave energy 51 Theta wave frequency variability 12 First level difference normalization 32 γ-wave energy 52 α-wave frequency variability 13 Second level difference 33 δ-wave energy share 53 β-wave frequency variability 14 Second level difference normalization 34 θ-wave energy share 54 γ-wave frequency variability 15 Hjorth Mobility 35 α wave energy ratio 55 Sample entropy 16 Hjorth complexity 36 β-wave energy 56 Approximate entropy 17 Fractal dimension 37 γ-wave energy ratio 57 Fuzzy entropy 18 Instability index 38 δ-wave differential entropy 58 Ranking entropy 19 Total energy 39 θ-wave differential entropy 59 Spectral entropy 20 Low δ-wave energy 40 α-wave differential entropy

Classiﬁcation Algorithm In this study, four algorithms, Support Vector Machine (SVM), 1 1 1 2πe DT, BPNN, and k-Nearest Neighbor (kNN), were chosen h (X) = log(2πeσ 2) = log(P ) + log i 2 i 2 i 2 N to explore the advantages of the happy and sad emotion dichotomous classification problems. When using the SVM Since the length of all samples is 1 s, N is a constant, and algorithm, the kernel function selected in this article is a linear the latter term of the above equation is a constant, which can function, and the hyperparameters of the support vector machine be disregarded from the classification point of view. Also, the are optimized through grid search. When using the BP neural coefficient 1/2 does not affect the performance of the feature in network algorithm, the number of nodes in the input layer is classification. determined according to the number of eigenvalues of each part. After obtaining the energy spectrum of each frequency band The number of nodes in the output layer is set to two, because of the sample, the logarithm can be obtained to represent the this article mainly classifies positive and negative emotions. The differential entropy feature of the EEG signal in that frequency value of the number of hidden√ layer nodes is determined by the band. This study used this method to calculate the differential following formula: m = n1, where n is the number of input entropy and used it as a classification feature. layer nodes, l is the number of output layer nodes, and m is the A gradient boosting DT is chosen to select the most number of hidden layer nodes. When using the DT algorithm, effective channel and features from multi-channel multi- first, the classification model of the DT is generated by learning features (GBDT). The algorithm’s feature contribution rate the training set; second, the model is used to classify samples index is used as the basis for feature selection to filter of unknown types. This study used the C4.5 DT algorithm, and the features that contribute the most to the classification the segmentation index is the information gain rate. When using and simplify the model by feature selection to improve the the knn algorithm, this study used the brute method to select classification efficiency. the best k.

Frontiers in Behavioral Neuroscience| www.frontiersin.org 4 August 2021 | Volume 15 | Article 720451 Long et al. Positive and Negative Emotion Classiﬁcation

After the classifier algorithm is determined, we evaluate the find the average accuracy. In this study, all the subjects’ data were classification performance by dividing the sample data into trained and tested together as a whole, and the accuracy rates 10 equal segments that are based on 10-fold cross-validation. were calculated separately and averaged as an individual. Specifically, each round uses nine segments as the training subset and the remaining one segment as the test subset. Thus, the Classification Effects on the Baseline datasets are k non-overlapping data sets, and the datasets are Feature Set not perfectly consistent with each other. The evaluation metric In the baseline feature set of all 1,888 dimensions, the of correctness is calculated in each trial, and the generalization classification effect of each classifier is shown in Table 2. ability of the model is finally evaluated by averaging the evaluation metrics after k trials. The basic steps of fold cross- The Classification Effect of Energy validation are as follows: Occupation Ratio and Differential Entropy Feature (a) The original data set is divided into 10 subsets with as A study showed that the energy occupation ratio and differential balanced sample size as possible; entropy features have significant effects on two-channel emotion (b) The first subset is used as the test set, and the second to the dichotomous classification, so the effects of extending these two ninth subsets are combined as the training set; sets of features to multiple channels are first verified. Also, (c) Use the training set to train the model and calculate the results to test the effectiveness of linear normalization and standard of multiple evaluation metrics under the test set; normalization methods on classification, the effect of different (d) Repeat steps 2–3, and use subsets two to 10 as the test set in normalized features on these classifiers was added. The results of turn; this set of experiments are shown in Table 3. (e) Calculate the average value of each evaluation index as the From the results of experiments 2–1, it is clear that the final result. group of features, energy percentage, and differential entropy Dual-Channel EEG Emotion Classification has significantly improved the classification effect over the Process baseline system, indicating that this group of features can Considering the need for emotion classification in forehead effectively distinguish emotions. Comparing experiments 2–1, dual-channel portable devices, an algorithmic flow for emotion 2–2, and 2–3, for this group of features, neither of the two classification using only forehead electrode signals was also types of normalization can improve the classification effect, so designed in this study, shown in the figure below. normalization will not be performed in the future when using As shown in Figure 2, there are two changes compared to these two types of features. Experiments 2–4 and 2–5, on the the multichannel EEG classification process. The number of other hand, compared the effect of two features, energy share, channels is reduced, and the bad leads cannot be replaced by the and differential entropy, on emotions classification. The results peripheral leads’ interpolation of signals. The invalid signals must revealed that the energy-occupancy feature’s performance is only all be removed entirely in the stage of debugging. The number of comparable to the classification result of the full feature. In channels is too small to apply the ICA algorithm directly, so the contrast, the differential entropy feature’s classification effect Ensemble Empirical Mode Decomposition (EEMD) technique is not much different from that of the energy-occupancy + needs to be introduced to decompose each channel’s EEG signal. differential entropy feature. Thus, in terms of single metrics, EEMD is an improved algorithm of EMD that can extract the differential entropy is the feature that best captures differences eigenmode components (IMFs) of the original signal that are not in sentiment. subject to the modal mixing phenomenon (Olesen et al., 2016). Classification Effect of Differential Entropy After ICA of all IMF components, the EEG can be removed, and Features in Different Frequency Bands the subsequent process is the same as in the case of multiple Research shows that the difference between the high-frequency channels. band signals is more evident than in the low-frequency band. Therefore, experiments were designed to verify differential RESULTS entropy features of different frequency bands on emotion classification. The results are shown in the emotion dichotomous According to the above model training and testing scheme, we classification, the validity of each frequency band EEG signal arranged five sets of experiments with control variables to find is ranked as γ > β > δ > α > θ when considered from the classification algorithm to achieve the best results gradually. the perspective of differential entropy features, where γ wave, This study compares each classification model’s strengths is the best and β wave is also better. These two frequency and weaknesses using the training set accuracy and the test set bands are influential for the emotion classification problem. The accuracy. Also, to take into account the measurement of the remaining three frequency bands can be considered ineffective generalization ability of the model, two schemes were used on in distinguishing emotions compared to the baseline feature the current data collected from 12 subjects, one is to combine the set. Comparing experiments 3–1 with 3–6, when β and γ wave data from all subjects, train one model and validate its accuracy, features are incorporated simultaneously, it is better than using and the other is to train the model for each subject separately γ wave features alone. The results of this set of experiments are and validate the accuracy on their respective models, and finally shown in Table 4.

Frontiers in Behavioral Neuroscience| www.frontiersin.org 5 August 2021 | Volume 15 | Article 720451 Long et al. Positive and Negative Emotion Classiﬁcation

FIGURE 2 | Flow chart of dual-channel emotion classiﬁcation algorithm.

TABLE 2 | Classiﬁcation effect on the baseline feature set.

Classiﬁer SVM DT BP kNN

Category Training set Test set Training set Test set Training set Test set Training set Test set

Whole 0.6634 0.6362 0.9880 0.7839 0.8962 0.7949 0.8109 0.7428 Single 0.6995 0.6437 0.9313 0.8127 0.9683 0.9061 0.7996 0.7630

TABLE 3 | Effect of energy proportion and differential entropy on emotion classiﬁcation.

Classiﬁer SVM DT BP kNN

Category Training set Test set Training set Test set Training set Test set Training set Test set

Experiment 2–1 Energy ratio + Differential entropy

Whole 0.7887 0.7406 0.9765 0.7099 0.8960 0.8079 0.8652 0.8017 Single 0.7772 0.7153 0.9676 0.8219 0.9703 0.9079 0.8641 0.8088

Experiment 2–2 z-score normalization, Energy ratio + Differential entropy

Whole 0.7638 0.7093 0.9765 0.7099 0.8960 0.8079 0.8317 0.7586 Single 0.7234 0.6590 0.9676 0.8219 0.9703 0.9079 0.8293 0.7601

Experiment 2–3 Linear normalization [0, 1], Energy ratio + Differential entropy

Whole 0.7673 0.7183 0.9765 0.7099 0.8960 0.8079 0.8451 0.7615 Single 0.7305 0.6641 0.9679 0.8203 0.9666 0.9059 0.8264 0.7673

Experiment 2–4 Differential entropy

Whole 0.7912 0.7392 0.9773 0.7473 0.9052 0.8292 0.8736 0.8063 Single 0.7757 0.7232 0.9625 0.8179 0.9777 0.9216 0.8513 0.8069

Experiment 2–5 Energy ratio

Whole 0.6876 0.6353 0.9637 0.5794 0.7556 0.6928 0.7462 0.6956 Single 0.6930 0.6301 0.9558 0.7094 0.8803 0.7953 0.7869 0.7138

Classiﬁcation Effects of Differential based mainly on the differential entropy of gamma waves for Entropy Feature for Different Brain Regions channel selection. Since γ-wave features are significantly more useful than β- Based on each channel’s contribution rate and location wave features for emotion classification, channel selection was distribution, five experiments were designed to verify different performed using only the differential entropy of γ-wave to feature combinations’ effect on the classification effect. The determine the best brain regions. The channel selection was electrode combinations are shown in Table 6. The experiments performed based on a gradient boosting DT, and Table 5 were conducted according to the above five sets of electrode shows the contribution of 32 channel features to emotion combination schemes, and the results are shown in Table 7. From classification. Notably, our view is also supported by the study the above experiments, it can be seen that the electrode group of Jalilifard et al.(2016), in which the SVM was used to corresponding to experiments 5–3 has the best classification classify emotions using different rhythmic neural oscillations, effect. Combined with the electrode distribution map, it is clear and it was found that the classification of gamma rhythms that the human brain’s lateral ring region is most effective was best in the left prefrontal region of the brain (FP1), with in classifying emotions. The schematic diagram of electrode β being the second best. Therefore, this part of the results is selection corresponding to experiment 5–3 is shown in Figure 3.

Frontiers in Behavioral Neuroscience| www.frontiersin.org 6 August 2021 | Volume 15 | Article 720451 Long et al. Positive and Negative Emotion Classiﬁcation

TABLE 4 | The effect of differential entropy in different frequency bands on emotion classiﬁcation.

Classiﬁer SVM DT BP kNN

Category Training set Test set Training set Test set Training set Test set Training set Test set

Experiment 3–1 Differential entropy (γ wave)

Whole 0.8609 0.8219 0.9629 0.7408 0.8416 0.8132 0.8657 0.8308 Single 0.8599 0.8077 0.9597 0.8225 0.9342 0.8950 0.8821 0.8379

Experiment 3–2 Differential entropy (β wave)

Whole 0.8270 0.7819 0.9589 0.7069 0.8137 0.7867 0.8267 0.7912 Single 0.8245 0.7614 0.9488 0.7856 0.9180 0.8710 0.8523 0.8011

Experiment 3–3 Differential entropy (α wave)

Whole 0.6894 0.6360 0.9481 0.5915 0.7166 0.6868 0.6941 0.6578 Single 0.6661 0.6024 0.9364 0.6787 0.8175 0.7607 0.7487 0.6816

Experiment 3–4 Differential entropy (θ wave)

Whole 0.6924 0.6184 0.9461 0.5792 0.7011 0.6620 0.693 0.6131 Single 0.6347 0.5732 0.9357 0.6694 0.8098 0.7480 0.7248 0.6603

Experiment 3–5 Differential entropy (δ wave)

Whole 0.7315 0.6674 0.9482 0.5931 0.7246 0.6826 0.7429 0.6959 Single 0.6805 0.6067 0.9393 0.6798 0.8284 0.7645 0.7865 0.7186

Experiment 3–6 Differential entropy (β, γ wave)

Whole 0.8584 0.8226 0.9695 0.7568 0.8771 0.8281 0.8835 0.8422 Single 0.8703 0.8269 0.9566 0.8208 0.9580 0.9095 0.8916 0.8499

TABLE 5 | The contribution of differential entropy characteristics of each electrode to emotion classiﬁcation.

Rank Electrodes Contribution rate Rank Electrodes Contribution rate Rank Electrodes Contribution rate

1 TP9 0.1383 12 C3 0.0309 23 P7 0.0088 2 Fp2 0.1215 13 P8 0.0197 24 FC1 0.0082 3 T7 0.1013 14 F7 0.0194 25 FT9 0.0082 4 Fp1 0.0758 15 CP6 0.0194 26 Pz 0.0070 5 TP10 0.0682 16 P4 0.0179 27 Cz 0.0065 6 O1 0.0589 17 F3 0.0175 28 F4 0.0049 7 T8 0.0542 18 FC5 0.0173 29 F8 0.0038 8 CP1 0.0387 19 FT10 0.0143 30 P3 0.0037 9 O2 0.0365 20 FC2 0.0124 31 Fz 0.0019 10 Iz 0.0330 21 FC6 0.0103 32 CP2 0.0003 11 C4 0.0320 22 CP5 0.0089

TABLE 6 | The contribution of differential entropy characteristics of each (1) For DTs and BP neural networks, there is a significant electrode to emotion classiﬁcation. difference, usually about 10%, between the effect of training Experiment Electrodes the models separately for 12 subjects than the effect of training the models together for 12 subjects’ data. 4–1 Forehead Fp1, Fp2 4–2 Forehead >0.05 Fp1, Fp2, T7, T8, TP9, TP10 (2) For DTs, the test set’s accuracy is on average more than 20% 4–3 Contribution > 0.03 Fp1, Fp2, T7, T8, TP9, TP10, lower than that of the training set, which shows the severe O1, O2, Iz overfitting of DTs. 4–4 Selection of electrodes Fp1, Fp2, T7, T8, O1, O2 (3) SVM and kNN are close to each other, and there is no obvious along the head loop problem of overfitting and poor generalization performance. 4–5 Only half of the electrode Fp1, T7, O1 is selected (4) When using SVM for classification, under the premise of channel selection, its accuracy can be as high as 86%, which is better than other classifiers. Classiﬁcation Algorithm Selection Summarizing the above four sets of experiments, the following Dual-Channel Dichotomy Effect conclusions can be drawn from the four classification algorithms After several rounds of experimental comparison, in the used in this study: two-channel two-classification problem, the best feature

Frontiers in Behavioral Neuroscience| www.frontiersin.org 7 August 2021 | Volume 15 | Article 720451 Long et al. Positive and Negative Emotion Classiﬁcation

TABLE 7 | The effect of differential entropy characteristics under each electrode combination on emotion classiﬁcation.

Classiﬁer SVM DT BP kNN

Category Training set Test set Training set Test set Training set Test set Training set Test set

Experiment 5–1 Forehead

Whole 0.5143 0.5071 0.8970 0.5949 0.6372 0.6296 0.6362 0.6254 Single 0.6703 0.6038 0.9114 0.6956 0.7697 0.7461 0.7642 0.7323

Experiment 5–2 Forehead >0.05

Whole 0.8591 0.8288 0.9490 0.7460 0.7884 0.7727 0.8551 0.8332 Single 0.8895 0.8455 0.9536 0.8125 0.9046 0.8728 0.8906 0.8588

Experiment 5–3 Contribution > 0.03

Whole 0.8845 0.8618 0.9591 0.7703 0.8495 0.8332 0.8275 0.8567 Single 0.8970 0.8614 0.9560 0.8275 0.9066 0.8868 0.8924 0.8648

Experiment 5–4 Selection of electrodes along the head loop

Whole 0.8534 0.8136 0.9465 0.7187 0.8144 0.7933 0.8247 0.7982 Single 0.8598 0.8070 0.9390 0.7703 0.8775 0.8352 0.8473 0.8152

Experiment 5–5 Only half of the electrode is selected

Whole 0.7881 0.6827 0.9144 0.6428 0.7027 0.6956 0.8082 0.6957 Single 0.7669 0.6814 0.9193 0.7180 0.8041 0.7722 0.7849 0.7596

DISCUSSION

In this study, we used positive and negative emotion videos to induce emotions in subjects and extracted EEG features of different emotions based on multichannel and prefrontal channels to investigate the effect of classification of positive and negative emotions. We first obtained the classification results of the four classifiers based on a total of 1,888 features across all channels. In this part of the results, the best performing classifiers are DT and BP neural network, which can achieve an overall correct rate of 78% and 79%, while SVM and kNN only have 63% and 74% correct rates. However, since too many features can cause redundancy of information, in the second section, we use energy share and differential entropy for sentiment classification based on our previous study. Our results show that the difference between the classification effect of ‘‘energy share + differential entropy’’ and that of a single ‘‘differential entropy’’ indicator is not significant, which indicates that differential entropy is the

FIGURE 3 | Schematic diagram of the optimal electrode for multi-channel feature that best reflects the difference in sentiment if only a emotion recognition. single indicator is considered. The results are consistent with Zheng and Lu(2015); that is, the differential entropy feature is the most stable and outstanding in emotion classification. In the combination is the energy proportion and differential entropy third part of the results report, we focus on the use of differential of the δ, θ, α, β, and γ bands of the two channels, with a total of entropy as a metric to explore the classification effects at different 20-dimensional features. The best results are shown in Table 8. frequency bands. With this part of the analysis, our results show As can be seen from the above table, for the dual-channel that in terms of the feature classification effect of differential two-classification problem, the classification effects of using entropy in each frequency band, the order of the validity of SVM classifier, DT, BP, and kNN are 66%, 60%, 66%, and 64%, EEG signals is γ > β > δ > α > θ, in which the effect of γ respectively. However, considering the pursuit of the model’s wave is the best, and the effect of β wave is also better. The generalization ability in the actual classification, the overall differential entropy of these two bands is effective for emotion performance is taken as the main index. Therefore, under this classification. The result was consistent with Li et al.(2018) premise, the best classification model should still choose the using a hierarchical convolution neural network (HCNN). Their SVM classifier; its accuracy can reach 66%. study uses a HCNN to classify positive emotional state, neutral

Frontiers in Behavioral Neuroscience| www.frontiersin.org 8 August 2021 | Volume 15 | Article 720451 Long et al. Positive and Negative Emotion Classiﬁcation

TABLE 8 | The effect of dual-channel 20-dimensional features on emotion classiﬁcation.

Classiﬁer SVM DT BP kNN

Category Training set Test set Training set Test set Training set Test set Training set Test set

Whole 0.7270 0.6630 0.9365 0.6037 0.6907 0.6648 0.6894 0.6435 Single 0.7025 0.6227 0.9352 0.6886 0.8100 0.7609 0.7679 0.7035 emotional state, and negative emotional state. The differential classification effect and generalization ability. However, the KNN entropy features from different channels are organized into algorithm needs to store many training samples and is not easy to two-dimensional maps to train HCNN. The results show that implement on embedded devices. Meanwhile, the computational there is a good classification ability on Beta and Gamma waves. In complexity of prediction increases linearly with the increase the fourth part of the result report, based on the previous results, of training samples. The SVM algorithm, on the other hand, we select only the differential entropy features of the γ-band has stable computational complexity, and the training can to analyze the classification accuracy of different brain regions, be done offline. Therefore, from the perspective of practical and the results show that when electrodes are selected along the applications, SVM is superior to kNN. In the two-channel head loop, there are good results under all four classifiers, and, study, considering the model’s generalization ability, the SVM from the results of the test set, the SVM classifier has the best algorithm is more appropriate. The conclusion of this study once classification results (86.18% accuracy overall), while the DT, BP, again demonstrates the effectiveness of the SVM algorithm in and kNN root class effects were 77.03%, 83.32%, and 85.67%, emotion classification, which is consistent with the research of respectively. This is one point where this study goes beyond Nie et al.(2011). In their research, we use the multi-feature fusion previous research in that it identifies which regions have the and SVM classifier method to study the second classification greatest impact on emotion classification, given the identified of emotion, and the accuracy is 87.53%. Altogether, this study extracted features. extends the previous findings. Based on the multi-channel The study also compares the feature classification effects emotion-evoked EEG data, the two types of emotions can be of differential entropy in different brain regions. The results effectively classified using the energy proportion and differential showed that when using the differential entropy of γ wave entropy of frequency bands best effect by using an SVM classifier. for channel selection, the lateral annular region of the human When only the signals of the two channels of the forehead are brain is the most effective for emotion classification. This used, the highest classification accuracy can reach 66%. When the is a point that this study surpasses previous studies; that data of all channels are used, the highest accuracy of the model is, on the premise of determining the extraction of features, can reach 82%. After channel selection, the best model can be we point out which regions have the greatest impact on obtained in which the accuracy can reach 86%. The electrodes emotion classification. Another concern of our study is whether in the optimal channel combination scheme are all located in there is a difference in the classification effect between multi- the lateral annular region of the human brain, which provides channel and dual-channel. The discussion of this problem is a theoretical basis for the follow-up development of portable not simply to repeat the experimental process on multiple headband emotion monitoring equipment. channels but to select the best feature combination by comparing several groups of experiments. Our results show that the LIMITATIONS energy ratio and differential entropy of the δ, θ, α, β, and γ bands of the two channels have a good classification This study also has some limitations. First, the current study effect. When using SVM and BP neural network classifiers, uses the subject-dependent way for emotion recognition, even they can reach the classification accuracy of 66%. However, if a single subject’s data is used to train the emotion classifier considering the better generalization ability of SVM in previous for the subject. When the subject is changed, it is necessary to studies (Yao et al., 2019), we suggest that SVM is the best train a new classifier for the subject. There are great defects in classifier for the binary classification of positive and negative generalization ability. Second, although our research shows that emotions based on two-channel data. It’s worth noting that, better emotion classification results can be achieved by using an the classification efficiency of multi-channel feature classification SVM classifier, some studies have shown that when using a deep is better than two-channel feature classification regardless of learning model to classify emotions, its accuracy is 3.54% higher the feature classification effect due to the fact that multi- than that of the traditional SVM algorithm (Zheng et al., 2014), channel has more information and can better represent which suggests that future research can focus on deep learning the information (Lin et al., 2009; Garg and Verma, 2020). model to explore further how to achieve more efficient emotion Therefore, to pursue higher classification accuracy, as many classification results. channels as possible should be selected to explore emotion classification. DATA AVAILABILITY STATEMENT Finally, the classification effects of the four classifiers were compared. In multi-channel EEG emotion classification, SVM The raw data supporting the conclusions of this article will be and kNN are better than DT and BP neural network in made available by the authors, without undue reservation.

Frontiers in Behavioral Neuroscience| www.frontiersin.org 9 August 2021 | Volume 15 | Article 720451 Long et al. Positive and Negative Emotion Classiﬁcation

ETHICS STATEMENT the data and was involved in writing the manuscript. All authors contributed to the article and approved the submitted version. The studies involving human participants were reviewed and approved by the Ethics Committee of the Xian Jiaotong University. The patients/participants provided their written FUNDING informed consent to participate in this study. This work was supported (to PF) by Department of Military AUTHOR CONTRIBUTIONS Medical Psychology, Air Force Medical University, Xian 710032, China, the Major Project of Medicine Science and Technology XW and FL conceived the study and designed the study. SZ, of PLA (Grant No. AWS17J012), and National Natural Science S-CN, and XN collected the data. AC, PF, WZ, and BW analyzed Foundation of China (61806210).

REFERENCES IEEE Eng. Med. Biol. Soc. 2016, 845–849. doi: 10.1109/EMBC.2016. 7590833 Alazrai, R., Homoud, R., Alwanni, H., and Daoud, M. I. (2018). EEG-based Kim, M. K., Kim, M., Oh, E., and Kim, S. P. (2013). A review on the emotion recognition using quadratic time-frequency distribution. Sensors computational methods for emotional state estimation from the human (Basel) 18:2739. doi: 10.3390/s18082739 EEG. Comput. Math. Methods Med. 2013:573734. doi: 10.1155/2013/ Al-Nafjan, A., Hosny, M., Al-Ohali, Y., and Al-Wabil, A. (2017). Review 573734 and classification of emotion recognition based on EEG brain-computer Koelstra, S., Mühl, C., Soleymani, M., Lee, J. S., Yazdani, A., Ebrahimi, T., interface system research: a systematic review. Appl. Sci. 7:1239. et al. (2012). DEAP: a database for emotion analysis; using physiological doi: 10.3390/app7121239 signals. IEEE Trans. Affect. Comp. 3, 18–31. doi: 10.1109/T-AFFC. Asghar, M. A., Khan, M. J., Fawad, A. Y., Rizwan, M., Rahman, M., Badnava, S., 2011.15 et al. (2019). EEG-Based multi-modal emotion recognition using bag of Li, J., Zhang, Z., and He, H. (2018). Hierarchical convolutional neural deep features: an optimal feature selection approach. Sensors (Basel). 19:5218. networks for EEG-based emotion recognition. Cogn. Comp. 10, 368–380. doi: 10.3390/s19235218 doi: 10.1007/s12559-017-9533-x Bailen, N. H., Wu, H., and Thompson, R. J. (2019). Meta-emotions in daily life: Lin, Y. P., Wang, C. H., Wu, T. L., Jeng, S. K., and Chen, J. H. (2009). ‘‘EEG- Associations with emotional awareness and depression. Emotion 19, 776–787. based emotion recognition in music listening: a comparison of schemes for doi: 10.1037/emo0000488 multiclass support vector machine,’’ in 2009 IEEE International Conference on Calix, R. A., Javadpour, L., and Knapp, G. M. (2012). Detection of affective states Acoustics, Speech and Signal Processing (ICASSP), (Taipei, Taiwan), 489–492. from text and speech for real-time human--computer interaction. Hum. Factors doi: 10.1109/icassp.2009.4959627 54, 530–545. doi: 10.1177/0018720811425922 Mennes, M., Wouters, H., Vanrumste, B., Lagae, L., and Stiers, P. (2010). Chen, D. W., Miao, R., Yang, W. Q., Liang, Y., Chen, H. H., Huang, L., Validation of ICA as a tool to remove eye movement artifacts from EEG/ERP et al. (2019). A feature extraction method based on differential entropy and Psychophysiology 47, 1142–1150. linear discriminant analysis for emotion recognition. Sensors (Basel) 19:1631. Nie, D., Wang, X. W., Shi, L. C., and Lu, B. L. (2011). ‘‘EEG-based emotion doi: 10.3390/s19071631 recognition during watching movies,’’ in 2011 5th International IEEE/EMBS Deng, Y., Yang, M., and Zhou, R. (2017). A new standardized emotional film Conference on Neural Engineering (NER), (Cancun, Mexico), 667–670. database for asian culture. Front. Psychol. 8:1941. doi: 10.3389/fpsyg.2017. doi: 10.1109/NER.2011.5910636 01941 Olesen, A. N., Christensen, J. A., Sorensen, H. B., and Duan, R. N., Zhu, J. Y., and Lu, B. L. (2013). ‘‘Differential entropy Jennum, P. J. (2016). A noise-assisted data analysis method for feature for EEG-based emotion classification,’’ in 2013 6th International automatic EOG-based sleep stage classification using ensemble IEEE/EMBS Conference on Neural Engineering (NER), (San Diego, CA), 81–84. learning. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. 2016, doi: 10.1109/NER.2013.6695876 3769–3772. doi: 10.1109/EMBC.2016.7591548 Firpi, H. A., and Vogelstein, R. J. (2011). Particle swarm optimization- Petrantonakis, P. C., and Hadjileontiadis, L. J. (2010). Emotion based feature selection for cognitive state detection. Annu. Int. Conf. recognition from EEG using higher order crossings. IEEE Trans. IEEE. Eng. Med. Biol. Soc. 2011, 6556–6559. doi: 10.1109/IEMBS.2011.60 Inf. Technol. Biomed. 14, 186–197. doi: 10.1109/TITB.2009.203 91617 4649 Garg, D., and Verma, G. K. (2020). ‘‘Emotion recognition in valence-arousal Shi, L. C., Jiao, Y. Y., and Lu, B. L. (2013). Differential entropy space from multi-channel eeg data and wavelet based deep learning feature for EEG-based vigilance estimation. Annu. Int. Conf. IEEE framework,’’ in Procedia Computer Science, eds Sabu M. Thampi, Sanjay Eng. Med. Biol. Soc. 2013, 6627–6630. doi: 10.1109/EMBC.2013.661 Madria, Xavier Fernando, Robin Doss, Sameep Mehta, and Domenico Ciuonzo 1075 (Amsterdam, Netherlands: Elsevier B.V.), 857–867. doi: 10.1016/j.procs.2020. Torres, P. E., Torres, E. A., Hernández-Álvarez, M., and Yoo, S. G. (2020). 04.093 EEG-based BCI emotion recognition: a survey. Sensors (Basel) 20:5083. Gonzalez, H. A., Yoo, J., and Elfadel, I. M. (2019). EEG-based emotion doi: 10.3390/s20185083 detection using unsupervised transfer learning. Annu. Int. Conf. Wolf, K. (2015). Measuring facial expression of emotion. Dialogues Clin. Neurosci. IEEE. Eng. Med. Biol. Soc. 2019, 694–697. doi: 10.1109/EMBC.2019.88 17, 457–462. doi: 10.31887/DCNS.2015.17.4/kwolf 57248 Yao, J., Lu, K., and Xu, J. (2019). Classification of happiness and sadness based on Goshvarpour, A., and Goshvarpour, A. (2019). The potential of portable EEG devices. Comput. Syst. Appl. 29, 233–238. photoplethysmogram and galvanic skin response in emotion recognition Zheng, W. L., and Lu, B. L. (2015). Investigating critical frequency bands and using nonlinear features. Australas. Phys. Eng. Sci. Med. doi: 10.1007/s13246- channels for eeg-based emotion recognition with deep neural networks. IEEE 019-00825-7. [Online ahead of print]. Trans. Autonomous Mental Dev. 7, 162–175. doi: 10.1109/TAMD.2015.243 Gupta, R. (2019). Positive emotions have a unique capacity to capture 1497 attention. Prog. Brain Res. 247, 23–46. doi: 10.1016/bs.pbr.2019. Zheng, W. L., Zhu, J. Y., Peng, Y., and Lu, B. L. (2014). EEG-based emotion 02.001 classification using deep belief networks. In Proceedings - IEEE International Jalilifard, A., Brigante Pizzolato, E., and Kafiul Islam, M. (2016). Emotion Conference on Multimedia and Expo Chengdu, China IEEE Computer Society. classification using single-channel scalp-EEG recording. Annu. Int. Conf. (Vol. 2014).

Frontiers in Behavioral Neuroscience| www.frontiersin.org 10 August 2021 | Volume 15 | Article 720451 Long et al. Positive and Negative Emotion Classiﬁcation

Zhong, P., Wang, D., and Miao, C. (2020). EEG-based emotion recognition this article, or claim that may be made by its manufacturer, is not guaranteed or using regularized graph neural networks. IEEE Trans. Affect. Comput. endorsed by the publisher. doi: 10.1109/TAFFC.2020.2994159 Copyright © 2021 Long, Zhao, Wei, Ng, Ni, Chi, Fang, Zeng and Wei. Conﬂict of Interest: The authors declare that the research was conducted in the This is an open-access article distributed under the terms of the Creative absence of any commercial or financial relationships that could be construed as a Commons Attribution License (CC BY). The use, distribution or reproduction potential conflict of interest. in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this Publisher’s Note: All claims expressed in this article are solely those of the authors journal is cited, in accordance with accepted academic practice. No use, and do not necessarily represent those of their affiliated organizations, or those of distribution or reproduction is permitted which does not comply with the publisher, the editors and the reviewers. Any product that may be evaluated in these terms.

Frontiers in Behavioral Neuroscience| www.frontiersin.org 11 August 2021 | Volume 15 | Article 720451