Physiological Data Analysis for an Emotional Provoking Exergame Larissa Muller¨ Arne Bernin Sobin Ghose Department Informatik Department Informatik and Wojtek Gozdzielewski University of Applied Sciences (HAW) University of Applied Sciences (HAW) Department Informatik Hamburg, Germany Hamburg, Germany University of Applied Sciences (HAW) School of Engineering and Computing School of Engineering and Computing Hamburg, Germany University of the West of Scotland University of the West of Scotland Email: [email protected] Paisley, Scotland Paisley, Scotland Email: [email protected] Email: [email protected] Qi Wang Christos Grecos Kai von Luck School of Engineering and Computing Computer Science Department Department Informatik University of the West of Scotland Central Washington University (CWU) University of Applied Sciences (HAW) Paisley, Scotland Ellensburg, USA Hamburg, Germany Florian Vogt Innovations Kontakt Stelle (IKS) Hamburg, Germany Abstract—In this work we enhance our previously developed each participant. Our system is designed to analyze these analysis method of provoked emotions in facial expressions individual reactions and experiences. An ambient intelligent through the analysis of physiological data. The presented work exergame application should be aware of any kind of stress the describes the integration of electrodermal activity, respiration and temperature sensors to enhance our exergaming system user might perceive, which in our application means physical for emotional provocation. The combined analysis of facial or cognitive strain as well as emotional drain. expressions and physiological data is designed to evaluate phys- During our first case study, emotional provocation of fa- ical and cognitive stress as well as emotional reactions. The cial expressions while exergaming was evaluated [5]. In this experimental setup combines a cycling game controller with a work, our previously developed event-based analysis method 3D virtual cycling game to provoke emotions. A designed data recording framework collects frontal videos and physiological is extended by the analysis of physiological data, since mental data as well as game and controller events. In this work, we strain leads to discriminative expressions in facial expres- found evidence that physiological data analysis enhances the sions [6] and self-reports [7]. This induces the necessity to previously developed analysis method. The system is able to integrate the analysis of physiological data into our system. evaluate individual differences of an entertaining and balanced Additionally, we integrate an electrodermal activity (EDA) workout program. sensor because electrodermal activity is part of the autonomous nervous system, which is known to be closely associated with I. INTRODUCTION the arousal of the participant [8] and may be applied for basic Nowadays the paradigm of ”healthy living” attracts growing emotion recognition [9]. worldwide attention. Exergames have the potential to support Measuring physiological data for tension recognition has a healthy lifestyle [1]. Healthy living combines adequate a long tradition [6] because it is an interesting topic for nutrition and a well-balanced workout program, due to an various applications, such as supporting pre-hospital care [10] increasing sedentary lifestyle [2]. The work of Biddle [3] links or driving [11]. Too much mental strain may increase negative physical activity to emotions and mood, and Sussenbach¨ [4] emotions [12]. claims sports are crucial for well-being. The work presented Our EmotionBike system is designed as a platform to in this paper encourages an ambient personal fitness program conduct behavioral analysis studies. During a case study, the accompanied by entertainment content. Entertaining aspects influences of the Big Five personality traits on specifically can be used to enhance users’ endurance and support a long- designed and crafted game elements were evaluated [5]. The lasting fitness program. The perception of entertaining values system allows a situational context awareness due to a virtual and the emotional reaction to similar game elements is highly game environment, the physical accessible exercise machine individual. This results in a different personal experience for controller, and the smart home environment [13], [14] in which the system is located. are analyzed to improve emotion recognition for frustrating game elements. II. RELATED WORK Giraud et al. [17] applied the Big Five Model [23], which is a widely used model of personality traits. It consists of There are several related works for personal fitness man- five personality traits: Extraversion (E), Agreeableness (A), agement. For instance, Shih et al. [15] developed a physical Conscientiousness (C), Neuroticism (N), and Openness (O) fitness condition measurement system. In their work, they to experience. Brouwer et al. [24] investigated correlations compare physiological data (body temperature, blood pressure between low neuroticism and high extraversion with stress and weight) with a person’s body mass index. In our work, we sensitivity and assumed that there are measurable links be- integrate the emotional state of the user, as physical aspects tween physiology and personality. Their results have shown combined with emotional states have a great potential to that skin conductance correlated negatively with neuroticism, enhance user motivation and training effects. Sussenbach¨ et which was not what they expected [24]. They induced stress 1 al. [4] use a NAO robot as a companion cycling instructor to by negative feedback while gaming and measured by baseline perform motivation research in human-robot interaction. We differences, which is an often used technique because it has do not focus on personal motivation but rather on individual been shown that there are differences in skin conductance differences in reactions to emotion provoking game elements. [6]. Our work is based on an event-based analysis method The focus of our work lies on tasks which have a high among individuals while playing games, but for the sake of physical effort, since physical activity is essential for the pre- completeness baseline differences have been evaluated as well. sented exergame. The aspect of performing physical activity Schneider et al. [25], for instance, investigated resilience by has been studied in the research by Hong et al. [16]. Our using the appraisal ratio and examining the unique influence exergame needs to sense all possible sources of stress the user of personality on stress responses across multiple stressor might perceive, including physical or cognitive strain as well outcomes, including affect and performance. As stimuli they as emotional drain. used mental arithmetic tasks. They suggested that extraversion Common stressor tasks in related research are mental arith- plays a role in the stress process because it uniquely predicted metic, loud sounds and the cold water pressor test [16]. In higher positive affect and lower negative affect in their study. our work, we analyze game elements as potential stressors. Walmik et al. [26] investigated the potential of an aug- In addition, many studies induce stress by investigating public mented helmet showing heart rate data in social context. They speaking tasks (e. g. [12], [17]–[19] ) since these tasks are studied the engagement of cycling pairs and found that it can known to be experienced as stressful [18]. Dickerson and result in a social interplay which supports engagement with Kemeny [20] have shown that social-evaluative threats are the exertion activity. Their work has shown that a biofeedback perceived as stressful. In a social-evaluative threat, the task of physiological data can support social interplay and thus performance might be perceived as being negatively judged engagement with the exertion activity. by others. Our setup includes the presence of the experimenter A form of gameplay where the current emotional state of and an independent observer, simulating an external judgment the player leads to altering the game mechanics is called to the participants. The physiological system reacts not only affective gaming [27]. In addition, Nacke et al. [28] use to changes in stress, but also to changes in physical or biofeedback in game design. Negini et al. [29] increase or mental conditions. The recognition of physiological responses decrease game difficulty by an analysis of galvanic skin during physical activity thus differs from recognition without response and Parnandi et al. [30] provides an adaptive game movement [16]. Our work is based on the fact that it requires to electrodermal activity (EDA) measurements. physical activity, thus we designed an event-based analysis of Affect recognition is an inherently multimodal task [31]. the collected data. Nasoz et al. [32] developed an affectively intelligent adaptive Previous work analyzed the measurement of physiological car interface to facilitate a natural communication with the data in natural environments. For instance, Plarre et al. [21] user. Their interface includes an affective user model created found respiration features to be highly discriminatory of for each individual driver based on physiological measure- physiological stress. ments (galvanic skin response, heart rate and temperature). An interesting research topic is the personality dimension- Vachiratamporn et al. [33] have shown that survival horror ality that provides a basis to explain individual differences.