Data Fusion and Analysis Techniques of Neuromarketing

Data fusion and analysis techniques of neuromarketing

B. Li1,Y.Wang2 &K.Wang1,3 1Shanghai Key Laboratory of Intelligent Manufacturing and Robotics, Shanghai University, China 2School of Materials, University of Manchester, UK 3Knowledge Discovery Laboratory, Department of Production and Quality Engineering, Norwegian University of Science and Technology, Norway

Abstract

The electroencephalogram (EEG) is one of the main technical means of neuroscience research. It can read the level of physiologically activity in different areas of the brain by measuring the change in electric charge on the scalp. Eye tracking instrument is one of the main equipment of cognitive psychology research; it can explore people’scognitive process. Fusing EEG and eye tracking data together integrates the consumer’s affective (emotional) and cognitive responses, giving a comprehensive understanding of the consumer’s decision-making process. At present the international research achievements of this aspect is still less. In this paper, we combined with the international latest paper published in this aspect, described the data collection, data analysis and processing, and data integration framework, expected to provide some grounding for subsequent research. Keywords: neuromarketing, EEG, eye tracking, data fusion.

1 Introduction

Neuromarketing applies the neuroimaging technology to study consumer’s behavior, explore the mechanism of consumer decision-making on neural activity level, and ﬁnd the real driving force behind the consumer’s behavior [1]. From the point of the research ﬁeld, neuromarketing mainly includes the neuron research of consumer behavior, marketing strategies, advertising and customer relationship management, etc. [2]. Neuromarketing is an emerging discipline that developed based on the cognitive science and neuroscience [3].

The development of neuroscience allow researchers to observe dynamically human brain activity from different aspects and angles, uncover the direct reasons behind human behavior. Neuroscience studies of human brain structure and function of each structure is the physiological basis of neuromarketing research [4]. Generally the electroencephalogram (EEG) devices have the characteristics of cheap, low requirement to the experimental environment and easy to popularize, so it has become one of the main technical means of neuroscience research [5]. Eye movement instrument, as an important equipment of vision research, plays an important role in cognitive science and psychology research [6]. The combination of EEG signals with eye tracking signals to study the consumer’s decision-making process has become one of the hot issues in present neural marketing research.

2 Data collection

2.1 EEG signal collection

The EEG signal is the weak electrical potential difference that measured on the scalp [7] when a large number of neurons (Mainly the cone cells) discharge at the same time in the brain neural system. The patterns of cortical activity were obtained in the ﬁve principal frequency bands, Delta (0–4 Hz), Theta (3–7 Hz), Alpha (8–12 Hz), Beta (13–30 Hz), and Gamma (30–40 Hz) [8, 9]. The EEG signals are very weak, usually determined by placing electrodes on the scalp to detect the changes of brain waves. According to the type of the electrode, and the signal acquisition method is divided into wet and dry electrode [10]. The wet electrode is one of the traditional acquisition methods, Participants must daub conductive paste when using. Dry electrode is a kind of emerging collection technology that avoid the hassles of conductive paste daub. It can reduce the subjects’ discomfort for a long time involved in the experiment and is more conducive to the practical application of EEG equipment in the future.

Figure 1: EEG data collection processing.

Khushaba et al. [11] chose the Emotiv EPOC to collect the EEG signals when researched people’s decision-making process of select preferred crackers according to their own preferences from the crackers described by different characteristics. As shown in Fig. 1, the Emotiv EPOC is a high-resolution, neuro-signal acquisition and processing wireless headset that monitors 14 channels of EEG data and has a gyroscope measure for two-dimensional control. Neuromarketing can adopt the

Emotiv EPOC to collect the EEG signals on the subjects’ scalp, and then Emotiv EPOC sends the data to a computer via Bluetooth. The Emotiv Software Develop- ment Kit (SDK) provides a packet count functionality to ensure no data is lost, a writable marker trace to ease single trial segmentation tasks, and real-time sensor contact display to ensure quality of measurements [11].

2.2 Eye tracking data collection

The data collection process using eye tracking equipment mainly includes first determining the fixation point, then distribution of the fixation point, fixation time, and fixation order [12]. The pupil diameter and blink rate can be used as indicators of cognitive processing. The data can be used to assess human psychology and behavior, which could be further utilized to get more objective results [13]. At present large manufacturers of eye tracking instrument in the world include Tobii, SMI (Sensomotic Instrument), SR Research, ASL (Applied Science Labor- atory), etc. Most product types include head-mounted (head movements freely), fixed head, and remote sensing. Sampling rate is up to 1,250 Hz (Type: iView X; Hi-Speed). System accuracy degree can reach 0.5◦ and resolution can reach 0.2◦ in a certain scope angle (Type: Tobii TX300). The scale of tens of milliseconds system error exists for all kinds of products [14]. Eye tracking instrument has a very wide range of applications, mainly used for the psychological mechanism of visual information processing, such as reading, visual search, advertising, psychology, etc. In addition, the eye tracking instrument can also be used in developmental psychology, ergonomics, traffic psychology, and sports psychology research [15]. Wei-Long Zheng et al. adopted SMI eye tracking glasses to record pupil size changes under different emotional stimuli in the aspects of research human emotional cognitive [16].

3 Data analysis

3.1 The analysis of EEG signals

3.1.1 Cleaning and denoising EEG signals EEG signals are measured by electrodes on the scalp surface, because the device circuit interference or other physiological signals change will affect the voltage ﬁeld of the electrode nearby. It is inevitably mixed with other signals in the process of measurement. As a result, the original EEG signals have low signal-to-noise ratio (SNR) and contain a lot of noise and artifact. Detecting and removing artifacts in the EEG data due to muscle activity, eye blinks, electrical noise, etc. is an important problem in EEG signal processing research [17]. The existing pretreatment method of denoising and removing artifact from EEG signals mainly includes the following four: (1) Directly delete the EEG signals fragments that contain artifact. First observe the presence of artifact in the EEG signals when speciﬁc processing, then use manual way to delete it [18]. Or set a threshold for

the power or amplitude of EEG signals. Signal more than threshold would be considered as artifact signal and be deleted automatically by software program [19]. (2) Filter the artifact in EEG signals by using the linear filter. For example, use of a high-pass filter can filter out some eye electric artifact and the low-pass filter can be used to filter out some muscle artifacts [20]. (3) Adopt the method of linear regression to filter out artifact from the EEG signals. For example, under the two kinds of waveform that electrical interference of eye presence and electrical interference of eye no presence, based on the same kind of event-related potential. Combination of eye signals recorded at the time and adopted the least squares method to calculate how much eye electrical signals being contained in the event-related potential of objects under interference. Record eye electrical signals at the same time in online processing, then minus the corresponding proportion eye electrical signals from EEG signals for correction of EEG signals [21]. (4) By using the method of signal decomposition. Using the method of signal decomposition such as singular value decomposition or independent component analysis (ICA), the original EEG signals are decomposed into multiple components, the artifact composition are identified [22], the non-artifact components to reconstruct EEG signals are reused.

3.1.2 The spectrum analysis method of EEG signals The power spectrum analysis methods is the earliest and the most commonly used in EEG spectrum analysis methods, originating from Fourier transform. In the traditional EEG power spectrum analysis method, ﬁrst a correlation function is obtained, then according to Wiener–Khinchin theorem spectrum estimation [23] is obtained. But the premise of the power spectrum analysis methods is for stationary random signals. There is a big difference in the spectral analysis results of different time for non-stationary random signal. A common improvement method is to put the signal divided into many small pieces in time domain, and viewed as quasi-steady. Fetch the square of its amplitude frequency characteristics after Fourier transform for each short signal, then multiply by the appropriate window function as the signal power spectrum estimation [11]. The main drawback of this method is that the frequency resolution is poor, and the side lobe leakage problems exist. EEG is non-stationary random signals, the correct expression and precision of the frequency domain characteristics, phase information extraction, and the transient waveform analysis is the current hot issues in the study of the EEG signal processing. However, due to the common problems of spectrum analysis method, the variance of the estimate features is bad, and the estimates values go up and down along frequency axis severely; the longer the data, the more serious is the phenomenon [24]. So the parameter model spectrum estimation method is

WIT Transactions on Engineering Sciences, Vol 113, © 2016 WIT Press www.witpress.com, ISSN 1743-3533 (on-line) 400 Advanced Manufacturing and Automation V proposed. This method of data processing can produce high-resolution spectrum analysis results. Thus it provides a new effective method for the extraction of EEG signal frequency domain characteristics. This method shows superiority, especially in the analysis of dynamic characteristics [17].

3.2 Eye tracking data analysis

The data collected by eye tracking instrument include the first fixation point, the fixation point distribution, fixation time, and fixation order and some simple de- rived data (the pupil diameter, blink rate). The corresponding software system provided by general eye tracking instrument supplier can satisfy the needs of the vast majority of customers of eye movement data analysis. For example, Tobii Stu- dio 1.3 was employed as it offers an easy-to-use solution to extract and analyze eye tracking data. The package facilitates efficient multi-person and multi-trial studies. The software combines the collection and analysis of eye gaze data with numerous other data sources, including keystrokes, external devices, video recordings, and web browser activities [11]. The experimental conditions are complex. Eye tracking instrument’sown system can’t provide all the required results. In order to achieve the desired purpose and get the needed data, users need to find another data analysis method. Wei-long Zheng et al. [16] researched the emotion recognition method, which combines EEG and pupil diameter. According to this method, a person’s pupil diameter changes under different emotional states, so a pupil diameter with the characteristics of the emotion classification is chosen. However, the pupil diameter is highly dependent on the luminance of the objects. So it couldn’t be used for emotion recognition directly. Principal component analysis (PCA) is adopted to remove the influence of luminance on pupil.

4 The framework of data fusion and analysis

An EEG machine reads the level of activity in different areas of the brain by measuring the change in electric charge at the scalp. Electrodes are strategically placed all over the scalp so that measurement can be taken from different areas of the brain. It has been found in many studies that there is a clear link between our eye movements and our cognitive processes. Eye movement monitoring is a valuable tool for capturing decision-makers’ information. Fusing EEG and eye tracking data together integrates the consumer’s affective (emotional) and cognitive responses, giving a comprehensive understanding of the consumer’s decision-making process. At present, the international research achievements of this aspect is still less. Khushaba et al. [11] investigated physiological decision processes while participants undertook a choice task designed to elicit preferences for a product. As shown in Fig. 2, they used the commercial Emotiv EPOC wireless EEG headset with 14 channels to collect EEG signals from participants and a Tobii-Studio eye tracker system to relate the EEG data to the speciﬁc choice options (crackers). First, the collected EEG signals are put into the signal preprocessing stage.

In order to ﬁlter the noise and remove the artifacts, the EEG data was processed with a bandpass ﬁlter after detrending analysis. In signal denoising stage, they used a combination of independent component analysis (ICA) and discrete wavelet transform (DWT) based denoising [25] to clean the EEG signals. In the fast Fourier transform (FFT) stage, they analyzed changes in spectral power and phase to characterize perturbations in the oscillatory dynamics of ongoing EEG. At last they provide a way to quantify the importance of different cracker features that contribute to the product design based on mutual information.

Figure 2: Block diagram of the data analysis [11].

Wei-long Zheng and Bo-Nan Dong et al. [16] combined EEG and eye tracking data to research emotion recognition method. The framework of their experiment processing is shown in Fig. 3. For EEG data, they extracted different features from ﬁve frequency bands. For eye tracking data, we extracted mean values, standard deviations, and spectral powers of frequency bands from pupil responses. They applied fusion methods of feature-level fusion and decision-level fusion combining features from EEG signals and eye tracking data.

Figure 3: The framework of research processing [16].

Throughout the process of decision–making, there is a constant cycle between the cognitive processes (information gathering and reasoning) and the emotional responses (to internal and external information). Fusing EEG and eye tracking data together can help to truly understand the consumer decision-making process.

5 Conclusion

Neuromarketing has become the research focus today. It applies the neuroscience method to study consumer’s behavior, explore the mechanism of consumer decision-making on neural activity level, and ﬁnd the real driving force behind the consumer’s behavior. Then the appropriate marketing strategies are produced. Currently the international research achievements on this aspect are still less. How to combine the EEG data and eye tracking data to study the marketing behavior will become the research hot spot of neuromarketing. In existing research case, the data from the fusion analysis method of the two kinds is very unitary, the depth of integration analysis is enough, and does not form a complete system. In this paper, we have analyzed the latest papers published on this respect and described the data collection, data analysis and processing, and data integration framework. We expect to see further research in this area in the future.

References

[1] Lee, N., A.J. Broderick, and L. Chamberlain, What is “neuromarketing”? A discussion and agenda for future research. International Journal of Psychophysiology, 63, pp. 199–204, 2007. [2] Ma Qing-guo, and Wang Xiao-yi, From neuroeconomics and neuromarketing to neuromanagement. Journal of Industrial Engineering Management, 20(3), pp. 129–132, 2006. [3] Du Jian-gang, and Wang Lin, Research on neuromarketing – Introduction of fMRI. Economic Management Journal in China, 34(3), pp. 189–199, 2012. [4] Plassmann, H., T.Z. Ramsøy, and M. Milosavljevi, Branding the brain: A critical review and outlook. Journal of Consumer Psychology, 22, pp. 18–36, 2012. [5] Kong Wan-Zeng, Xu Si-Jia, Dai Guo-Jun, and Hu San-Qing, The survey of digital evaluation of advertising with neuromarketing. Chinese Journal of Biomedical Engineering, 33(3), pp. 358–365, 2014. [6] Meng-Lung Lai, Meng-Jung Tsai, Fang-Ying Yang, Chung-Yuan Hsu, et al., A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational Research Review, 10, pp. 90–115, 2013. [7] Fu Jia-Wei, Shi Li-Chen, and Lu Bao-Liang, A survey on EEG-based vigil- ance analysis and estimation. Chinese Journal of Biomedical Engineering, 28(4), pp. 589–596, 2009. [8] Mostafa, M.M., Brain processing of vocal sounds in advertising: A functional magnetic resonance imaging (fMRI) study. Expert Systems with Applications, 39(15), pp. 12114–12122, 2012.

[9] Khushaba, R.N., L. Greenacreb, S. Kodagodaa, J. Louviereb, S. Burkeb, and G. Dissanayake, Choice modeling and the brain: A study on the electroencephalogram (EEG) of preferences. Expert Systems with Applications, 39(16), pp. 12378–12388, 2012. [10] Wang, X.W., L.C. Shi, and B.L. Lu, A survey on technology of dry electrodes for EEG recording. Chinese Journal of Biomedical Engineering, 29, pp. 777–784, 2010. [11] Khushaba, R.N., C. Wise, S. Kodagoda, et al., Consumer neuroscience: Assessing the brain response to marketing stimuli using electroencephalogram (EEG) and eye tracking. Expert Systems with Applications, 40, pp. 3803–3812, 2013. [12] Zhao Xincan, Zuo Hongfu, and Ren Yongjun, A review of eye tracker and eye tracking techniques. Computer Engineering and Application, 12, pp. 118–120, 140, 2006. [13] Tien, T., P.H. Pucher, and M.H. Sodergren, Eye tracking for skills assess- ment and training: Systematic review. Journal of Surgical Research, 191(9), pp. 169–178, 2014. [14] Zhang Chuang, Chi Jian-Nan, Zhang Zhao-Hui, et al., The research on eye tracking for gaze tracking system. Acta Automatica Sinica, 36(8), pp. 1051–1061, 2010. [15] Dan Witzner Hansen, and Qiang Ji, In the eye of the beholder: A survey of models for eyes and gaze. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(3), pp. 478–500, 2010. [16] Wei-Long Zheng, Bo-Nan Dong, and Bao-Liang Lu, Multimodal emotion recognition using EEG and eye tracking data. IEEE Engineering in Medicine and Biology Society, pp. 5040–5043, 2014. [17] Plöchl, M., J.P. Ossandón, and P. König, Combining EEG and eye tracking: Identification, characterization, and correction of eye movement artifacts in electroencephalographic data. Frontiers in Human Neuroscience, 6, 2012. [18] Gratton, G., Dealing with artifacts: The EOG contamination of the event- related brai potential. Behavior Research Methods, Instruments, & Com- puters, (30), pp. 44–53, 1998. [19] Millan, J., M. Franze, J. Mourino, et al., Relevant EEG features for the classification of spontaneous motor-related tasks. Biological Cybernetics, 86, pp. 89–95, 2002. [20] Fatourechi, M., A. Bashashati, R.K. Ward et al., EMG and EOG artifacts in brain computer interface systems: A survey. Clinical Neurophysiology, 118, pp. 480–494, 2007. [21] Croft, R., J. Chandler, R. Barry et al., EOG correction: A comparison of four methods. Psychophysiology, 42, pp. 16–24, 2005. [22] Jung, T., S. Makeig, M. Westerfield, et al., Removal of eye activity artifacts from visual event-related potentials in normal and clinical subjects. Clinical Neurophysiology, 111, pp. 1745–1758, 2000.

[23] Lin, C.T., L.W. Ko, I.F. Chung, et al., Adaptive EEG-based alertness estimation system by using ICA-based fuzzy neural networks. IEEE Transactions on Circuits and Systems I: Regular Papers, 53(11), pp. 2469–2476, 2006. [24] Levy, W.J., Effect of epoch length on power spectrum analysis of the EEG. Anesthesiology, 66, pp. 489–495, 1987. [25] Akhtar, M.T., W. Mitsuhashi, and C.J. James, Employing spatially con- strained ICA and wavelet denoising for automatic removal of artifacts from multichannel EEG data. Signal Processing, 92(2), pp. 401–416, 2012.