An Adaptive Learning System Based on Real-Time User Emotions

SSStttooonnnyyy BBBrrrooooookkk UUUnnniiivvveeerrrsssiiitttyyy

The official electronic file of this thesis or dissertation is maintained by the University Libraries on behalf of The Graduate School at Stony Brook University.

©©© AAAllllll RRRiiiggghhhtttsss RRReeessseeerrrvvveeeddd bbbyyy AAAuuuttthhhooorrr... Sensemo: An Adaptive Learning System Based on Real-Time User Emotions

A Thesis presented

Karan Joshi

The Graduate School

in Partial Fulﬁllment of the

Requirements

for the Degree of

Master of Science

Computer Engineering

Stony Brook University

December 2016 Stony Brook University

The Graduate School

Karan Joshi

We, the thesis committe for the above candidate for the

Master of Science degree, hereby recommend

acceptance of this thesis

Shan Lin - Thesis Advisor Assistant Professor, Department of Electrical and Computer Engineering

Timothy J. Driscoll - Second Reader Adjunct Faculty, Department of Electrical and Computer Engineering

This thesis is accepted by the Graduate School

Charles Taber Dean of the Graduate School

ii Abstract of the Thesis

Sensemo: An Adaptive Learning System Based on Real-Time User Emotions

Karan Joshi

Master of Science

Computer Engineering

Stony Brook University

2016

Learning and education is an important part of everyone’s life. E-learning systems and smart classrooms have recently become very popular because of the continuous effort to improve learning experience of students. This work studies a new emotion sensing based real-time adaptive learning system using Microsoft Band smartwatch based sensing system. We use three bio-sensors, namely photoplethysmogram (PPG) sensor, galvanic skin response (GSR) sensor and skin temperature sensor, which are available in Microsoft Band 2 to help the system read the bio-signals from a user in real time and unobtrusively. The system is able to detect two indicators of human emotion (valence and arousal) with 79.1% - 84.7% accuracy. We present two usage scenarios for our system- online lecture system and classroom system. In order to improve the learning experience of a user, we propose two types of adaptation/suggestion methods: adapting the speed of the session and adapting the content, depending upon the emotion response of a user in real-time along with student profile. We perform experiments to find the effectiveness of the system. We test our adaptive learning system with 22 participants, and show that the autonomous feedback methods used are effective way of improving user learning experience evident from the minimal user interactions with system and higher score for quizzes after the lessons.

iii To my Mom, Dad, Sisters and Family

iv Contents

Abstract iii

List of Figures vii

List of Tables viii

Abbreviations ix

Acknowledgementsx

1 Introduction1

2 Related Work4 2.1 Emotion Recognition...... 4 2.2 Emotion based Adaptive Systems...... 5 2.3 Adaptive Tutoring Systems...... 5

3 Background6 3.1 Electrodermal Activity...... 7 3.2 Blood Volume Pulse and HRV...... 7 3.3 Skin Temperature...... 8

4 Usage Scenarios9 4.1 Online Lecture System...... 9 4.2 Classroom Lecture System...... 10

5 System Design 11 5.1 Emotion Recognition Subsystem...... 12 5.2 Feedback Control Subsystem...... 13

6 System Implementation 15 6.1 Sensor system...... 15 6.2 Feature extraction...... 16 6.3 Emotion Recognition...... 17 6.4 Feedback Control...... 17 6.4.1 PI Controllers...... 18 6.4.2 Decision Model...... 20

v 7 System Evaluation 22 7.1 Platform Evaluation...... 22 7.2 Emotion Recognition Evaluation...... 23 7.2.1 Experimental Setup...... 23 7.2.2 Accuracy...... 24 7.2.3 Reliability and Latency...... 24 7.3 Control System Evaluation...... 26 7.3.1 Response Time...... 26 7.3.2 Fitness of Model...... 26 7.3.3 System Acceptance...... 27

8 Real Deployment 28 8.1 Online Learning System...... 28 8.1.1 Experimental Setup...... 28 8.1.2 Result...... 29 8.2 Classroom System...... 29 8.2.1 Experimental Setup...... 30 8.2.2 Result...... 30

9 Conclusion 33 9.1 Future Work...... 33

A Appendix- Quiz Mergesort 35

B Appendix- Quiz Quicksort 36

C Appendix- Survey 37

D Appendix- Student Proﬁle 38

Bibliography 39

vi List of Figures

5.1 System Model...... 12 5.2 Learning Emotional Space...... 13 5.3 Feedback Control Model...... 14

6.1 System Components...... 15 6.2 Feedback Model...... 18 6.3 PI Controller...... 18

7.1 Physiological response of users to video stimuli...... 25 7.2 Skin temperature response of user to video stimuli...... 25 7.3 Model Fitness...... 27

8.1 System Comparison...... 29 8.2 Emotional response to First Session...... 31 8.3 Emotional response to second session...... 31 8.4 Quiz results for two sessions...... 32

vii List of Tables

7.1 Power usage breakdown of application running on Android device..... 23 7.2 Valence Levels...... 24 7.3 Arousal Levels...... 24

viii Abbreviations

ANS Autonomic Nervous System BVP Blood Volume Pulse EDA Electro Dermal Activity GSR Galvanic Skin Response SCL Skin Conductance Level SCR Skin Conductance Response SVM Support Vector Machine

ix Acknowledgements

This work would not have been possible without the guidance and support of the following people.

I would like to thank my adviser Dr. Shan Lin for giving me an opportunity to work with him on this exciting project. It was his continuous guidance and support that kept me motivated throughout the course of my thesis.

I would also like to thank Dr. Shahriar Nirjon for his guidance throughout this project. His continuous support has been an essential driver for this work. He and Shan helped make this diﬃcult task really easy and smooth. They have shown me how to approach a problem in a methodical way.

I would also like to thank my Thesis second reader Prof. Timothy Driscoll for his time and valuable feedback to improve this work.

Lastly but not least, I would like to take this opportunity to thank my parents for their moral support and my sisters for supporting me throughout my thesis. Special thanks to Ritika Joshi for helping with the initial system fabrication, to Arunjot Singh for moral and technical support at times and to Suhas Budhiraja for proofreading this document.

This thesis has been a great journey because of all these people and deserve to be sincerely thanked.

x Chapter 1

Introduction

E-learning systems and smart classrooms have recently become very popular. E-learning systems have provided the means for everyone to learn new things at the comfort of their homes or on the go. It has proved to be essential for people who do not have access to the classrooms or are unable to attend the classes due to their other commitments. This has led to a widespread increase in the usage of the e-learning systems. At the same time there has been increasingly growing interest in improving the current classrooms through the means of technology.

The online web based learning systems have come far from their initial conceptual versions. These systems have become more user-friendly and more user-aware as compared to the earlier versions of dictator style teaching. These systems are now more aware of the fact that if the user is being able to absorb the information or not. This is mostly done through the use of quizzes at the end of the lessons to know if the user is able to answer the questions right or not. Today’s modern learning systems have used this feature to introduce some degree of adaptivity by using the fact that if the user is not able to get the right answers to the question, that means he is not being able to understand the topic and thus the system now can adapt the level of diﬃculty of the future content accordingly.

This however, does not involve the use of the real-time feedback from the student as it is possible in real world classroom, for example, a student can ask the instructor to explain a topic using an alternative example and can give instructor a verbal/intuitive feedback for him/her to know how to proceed with the lecture. Thus, the instructor is able to see in real-time how the student is feeling about the current lecture.

This idea to make the online learning systems at par the traditional classroom systems introduces the possibility to explore use of real-time user feedback to help the system

1 adapt as a classroom instructor would. An emotion aware learning system thus can act as a real world classroom instructor and adapt based on the student’s real-time emotions. At the same time such exploration also opens the doors for introducing such feedback from students in a traditional classroom, in more collaborative way, to help the instructor make better decisions in terms of the lecture.

Emotion recognition is an emerging ﬁeld of interest. Emotion is highly subjective and fast changing. It is often accompanied by physiological changes in evoking human reactions and expressions.Emotion recognition through facial expressions [1][2], speech recognition [3][4] and gesture movements [5][6] have been proposed over the last few years where satisfactory results have been reported for speciﬁc applications. However, the results for the approaches such as facial expression recognition system are relying on the facial motion capture and such facial motions or behavioral modalities can be erro- neous or misleading. For example, anger can be masked by the happy face of a person. Also, students might tend to mask their emotions to avoid instructors’ questions. Thus, to obtain a promising result in emotion recognition, an insight into human feelings has to be considered.

In particular, little attention has been paid so far to physiological signals for emotion recognition compared to other methods. Reasons are some significant limitations resulting from the use of physiological signals for emotion recognition. The main difficulty lies in the fact that it is very hard to uniquely map physiological patterns onto specific emotion types and that physiological data are very sensitive to motion induced noise. On the other hand, physiological signals have considerable advantages. We can contin- uously gather information about the users’ emotional changes while they are connected to bio-sensors. Moreover, physiological reactions should be more robust against possible human social masking since they are directly controlled by the human autonomous nervous system. Work done in psycho physiology [7][8] provides evidence that there is a strong relationship between physiological reactions and emotional/affective states of humans. In affective computing, physiological signals have become a robust emotional data source to combat the artefacts created by human social masking [9][10].

The information about a user’s emotion is very useful particularly in human-computer interaction. A computer system which is able to understand a user’s emotional state can make desired decisions to improve his experience. This is an important aspect of adaptive systems. Online learning systems are one example where a human-in-the-loop feedback is essential for the system to be adaptive. Such human-in-the-loop system also have potential to improve traditional systems by providing essential insight into human response at a given time.

2 In this paper, we present Sensemo, a system which infers emotions of a user from physiological signals like heart rate, skin conductance and skin temperature. We use this emotion information to make learning system adaptive using real-time feedback in two ways: by adapting/suggesting the pace of the learning session and by adapting/suggesting the content. The system uses support vector machine (SVM) machine learning method to infer emotion from the multiple features derived from the physiological signals of heart rate, skin conductance and skin temperature. The adaptive system forms a human-in-the-loop feedback system using real time physiological data, and hence emotion, of user.

Contributions of this work can be summarized as follows:

• We build an eﬃcient and unobtrusive emotion detection system using sensors of Microsoft Band 2 to detect emotions speciﬁc to learning environment

• We design an adaptive learning model based on real-time emotional response of students as well as student’s learning proﬁle implementable in both online system and classroom system

• With extensive experiments on students in realistic learning environment, our system is readily accepted and improves student’s learning outcome

3 Chapter 2

Related Work

There are three categories of research topics related to our work: emotion or stress recognition system, emotion based adaptive systems and adaptive e-learning systems.

2.1 Emotion Recognition

Many researchers have attempted to build systems to detect emotions or stress. Most of these systems have been based on one of the behavioral monitoring techniques- facial expression, voice expression or other behaviors related to speciﬁc tasks like typing, mobile usage etc. Nasoz et al. [11] discuss about emotion detection using facial presence technology. Hong et al [12], discuss about voice based stress detection system with accuracy of 81% and 76% in indoor and outdoor environments. Chung [13] discuss about emotion recognition from text and Clayton [14] discusses about emotion recognition by analyzing the rhythm of user’s typing pattern on a standard keyboard. However, all these behavioral aspects are controllable by a user and thus this renders these systems unreliable. Recently, researchers have tried using the involuntary physiological signals for detecting emotions which is more reliable but challenging and involves complex data manipulations. Robert [15] discuss about emotion recognition using EEG signal and Sano et al [16] discuss about the use of wearable sensors and mobile use to detect stress, with only using the skin conductance and accelerometer as wearable sensors and with overall accuracy of 75%. Bahreini et al [17], [18] proposes a facial recognition based system and voice based system for detecting emotion in e-learning environment with an accuracy of 72% and 67% respectively.

4 2.2 Emotion based Adaptive Systems

There are a number of applications and situations where emotional information of the user can be used to improve the system. For example, lately, including emotion for the enrichment of user-computer experience has become a focus in the area of Human Com- puter Interaction (HCI). Chanel et al [19] proposes to maintain player’s engagement by adapting game difficulty according to player’s emotions assessed from physiological signals. Emotion information is the most important part of the field of affective computing.

2.3 Adaptive Tutoring Systems

Lastly, a few research works propose adaptive e-learning systems based on real-time feedback from user. Asteriadis et al [20] proposes an e-learning system based on esti- mating user behavior from tracking eye gaze and head position. Shen et al proposes a similar system for making e-learning systems adaptive through use of separate oﬀ-the- shelf bio-sensors. Craig et al discusses an Autotutor which considers emotions related to learning and provide feedback in form of an agent interacting with user.

Our system is a best-eﬀort system designed to scale to online as well as classroom system using physiological signals of users from a non-invasive compact smartwatch and providing adaptations and suggestions in respective use cases.

5 Chapter 3

Background

Every emotional response is accompanied by an autonomic nervous system response.The autonomic nervous system (ANS) is part of the peripheral nervous system in humans. It is regulated by the hypothalamus and controls our internal organs and glands, including such processes as pulse, blood pressure, breathing, and sweating in response to emotional circumstances [21]. The ANS is divided into two parts- sympathetic nervous system and parasympathetic nervous system. The sympathetic nervous system is the part of ANS which prepares the human body for an emotion-invoking event by changing the blood ﬂow and/or sweating etc. The parasympathetic nervous system acts against the sympathetic system to and works to keep energy for normal functions of body. Thus, these two systems work to maintain a balance in the human body.

The autonomic nervous system (ANS) controls heart muscles, smooth muscles and en- docrine glands [15][22]. A stressed emotion state generally accompanies sweaty palms, higher heart rate and tensed muscles. Similarly, every emotion state corresponds to a bodily change which is an involuntary response by ANS to maintain a balance in human body. Such physiological changes in response to the emotions are distinguished among diﬀerent emotions [23]. This autonomic response of body, thus, can be exploited to our advantage to infer the emotional state of the students from these physiological conditions.

These physiological conditions can be measured and monitored using speciﬁc bio-sensors. These signals are thus referred to as bio-signals. Our system uses the following set of bio-signals to get data related to emotions-

6 3.1 Electrodermal Activity

The electrodermal activity (EDA), which can also be called skin conductance, is the characteristic that defines autonomic changes in electrical properties of skin [24]. The skin conductance is measured by applying small current at two different points of skin and measuring the resulting flow of current. The EDA response includes a tonic component- skin conductance level (SCL), which is slow changing component and a phasic component- skin conductance response(SCR), which is the rapidly changing component caused by the autonomic nervous system activity. Both of these components are important measure of autonomic neural activity [25]. EDA is arguably the most useful index of changes in sympathetic arousal that are tractable to emotional and cognitive states as it is the only autonomic psychophysiological variable that is not contaminated by parasympathetic activity. EDA has been closely linked to autonomic emotional and cognitive processing, and EDA is widely used as a sensitive index of emotional processing and sympathetic activity [24].

The electrodermal activity, also called as skin conductivity response, is directly associated with the emotional response of human body. When we are stressed our palms become sweaty and this increases the skin conductivity [22]. Such changes in skin conductivity can be easily monitored with good accuracy using the galvanic skin response (GSR) sensor which is readily available in Microsoft Band 2. Such a sensor is able to give us ﬁne grain data which can detect very minor changes in skin conductivity while being highly unobtrusive.

3.2 Blood Volume Pulse and HRV

Blood Volume Pulse (BVP) is an umbrella term to refer to the phasic change in blood volume ﬂowing through vessels. The common measures derived from this physiological phenomenon are heart rate and heart rate variability (HRV). HRV is the phenomenon of time change between heartbeats. Due to sympathetic and para-sympathetic nervous system eﬀects, there always is very rapid short-term variation in heart rate which results in HRV. For example, when under a stressful condition the HRV tends to reduce as the body is focused to tackle the situation at hand. Heart rate and heart rate variability has been extensively studied with respect to emotional responding and have been found as a reliable index of emotional states [26]. Thus, blood volume pulse and heart rate variability are good indicators of event related autonomic nervous activity.

A photoplythesmograph (PPG) sensor is an unobtrusive way of measuring BVP signal and is available in the Microsoft Band 2. A PPG sensor uses optical sensors to sense 7 the change in reﬂected light from the target region which is directly related to the blood volume measure. This BVP signal is used to ﬁnd the inter-beat interval (IBI), which is the time interval between two heart beats. The Microsoft Band 2 provides direct values of IBI as a data stream as well as heart rate as a separate data stream. The HRV measure is derived from the IBI values of the user.

3.3 Skin Temperature

Skin temperature is another physiological measure which is associated with the autonomic nervous system. For example, when we get angry our skin temperature tends to rise, even if marginally. This signal is, however, also dependent on external environment and is not a fast responder. Our system explores the use of this signal into emotion recognition.

Skin temperature is an interesting physiological measure, even though it might not be a prompt event driven measure, as it can give key insight into user’s general comfort level as it has been long argued as a key factor for thermal comfort [27]. The thermal comfort and skin temperature in itself is an interesting ﬁeld of study. However, we only limit our system to use this as one parameter for our emotion recognition algorithm, which is essentially a machine learning algorithm, robust against non-useful parameters.

All of these signals are obtained from the built-in sensors of the Microsoft Band 2 and then several features are extracted further to be used into our study. We discuss these signals and features in detail in the system implementation section.

8 Chapter 4

Usage Scenarios

Our system is a best-effort system for improving the learning environment for students using the real-time sensor data of the students in every possible situations. Our system tries to improve the learning experience of the student no matter what the situation maybe, for example an ideal classroom situation with attentive students or a less ideal situation with some distractions or less attentive/ tired students. Our system does not treat different situations differently. We describe two usage scenarios for our system- the online lecture system and the classroom system.

4.1 Online Lecture System

The online lecture system scenario caters to the e-learning environment where the student only interacts with the computer for the lecture. This system involves a student, a computer based tutoring system with our adaptive algorithm, a smartwatch with the required sensors connected to the cloud through the android data acquisition application.

The user logs into the online course system in its computer, signs into our android app with the same credentials and starts using the course system. The course system begins with default settings and speed. The android app collects user’s real-time data and transfers to the course system. The course system uses the data to ﬁnd out user’s emotional response to the lecture and uses this information to automatically adjust the speed and content of the lecture to improve the learning experience of the user.

9 4.2 Classroom Lecture System

A classroom involves an instructor and multiple students. This system involves multiple students with a smartwatch each, having the required sensors, connected to their android data acquisition application which in turn connects to a central server. It also involves a desktop/web application for the instructor’s use.

Now, the instructor wants to improve the students’ learning experience by adapting his teaching style according to the students’ response. All the systems sign into Sensemo app which connects with their respective smartwatch band and starts transferring each user’s sensor data to our central server. The instructor uses our system app on his laptop which gives him students’ cummulative emotional response to the current content in real- time, like students are bored. The system also make suggestions to the instructor to adapt the speed or content, but at same time gives the freedom to instructor to adapt any way he likes.

10 Chapter 5

System Design

We present a brief description of our system design in this section. Our system is based on real-time user feedback, as emotions, collected from bio-signals from the student while he is actively involved with the learning material. This emotional feedback is then used in real-time to adapt the lecture in automatic fashion. The key components of the system are-

1. Sensor system- The Microsoft Band [28] is used to record user’s real time bio- signals, such as heart rate, skin resistance, skin temperature etc. , using the sensors available on the band.

2. Android data acquisition app- To collect data from the band, we need an android app which communicates with the band and collect sensor data. The app also processes the data for various feature extractions as discussed in following section. The app then can send the data to the cloud.

3. Cloud- The data storage is handled by the cloud. This allows easy storage for large amount of data and easy wireless access of data by the learning system.

4. Learning system- The learning system is the course teaching application which uses the real time data from the cloud to make decision for the user automatically, or the classroom system where instructor uses suggestions from the central server. This completes the loop and provides feedback to user as actions such as change in speed of slides.

The system can be further divided into logical subsections, namely emotion detection subsystem, user interaction subsystem and feedback control subsystem. This can be depicted as in the ﬁgure 5.1. These subsystems are further explained in next sections.

11 Figure 5.1: System Model

5.1 Emotion Recognition Subsystem

While the student is actively involved in the lecture and content, our smartwatch based sensor system is simultaneously recording physiological data through the connected android device. This physiological data is the first step towards the inference of emotional state of the student, which he/she is experiencing due to the ongoing lecture content. In order to make sense of this sensor data, it is very essential to first define a model for emotion states.

We define the emotions on valence-arousal space, which has been long studied and well accepted form of representation [29][30]. Valence and arousal define the axes of a two- dimensional plane where we can map emotions. Valence is an indicator of positive or negative emotion, like happy or sad, with higher valence as positive emotion and lower valence as negative emotion. Arousal is indicator of how strong the emotion is, with higher arousal indicative of strong emotions like excitement etc. Our system is trained to map the extracted feature values from the sensor system to the valence-arousal space by conducting the studies using international affective picture system (IAPS) photoset [31]. This photo-set has pictures rated for the valence and arousal values. These data points, recorded features and the rated values of arousal and valence, make the learning set for the state vector machine that is used to classify the real time data for the learning system.

We refer to the Russel [32], and Ekman and Freisen’s [33] theory of emotions which provide discrete tags to emotions, as depicted in the ﬁgure, for building our system

12 model. We also refer to Kort’s learning spiral model [34] which proposes emotions that are related to learning, such as boredom, confusion, curiosity and satisfaction. Our system maps the derived features from the sensor data onto the valence-arousal space. We divide our arousal and valence space into three levels each and these levels are mapped to discrete emotions based on study conducted by Mandryk et. al [35], which provides fuzzy rules for mapping each level to particular emotions. Our model transforms these fuzzy rules into discrete rules. The ﬁgure 5.2 shows our emotion mapping model and ﬁguratively depicts the rules used to map arousal-valence levels to emotions.

Figure 5.2: Learning Emotional Space

5.2 Feedback Control Subsystem

While involved in learning activities, a student can go through many different emotions, but these can be broadly classified into four major learning related emotions - satisfied, boredom, curiosity and confusion. Our system tries to keep the user in the optimum emotion state of curiosity, or also called as relaxed alertness [36], for optimum learning. After the emotion recognition through the sensor data, we now have access to the student’s emotional state which can help us make adaptations/suggestions to promote optimum learning.

The feedback control system, depicted in ﬁgure 5.3, can be considered as a black box which takes as input the current inferred emotions and student preferences, and provides as output the decision/suggestion to change speed or content of the slides. These decisions/suggestions are based on the current emotion state and the desired emotion

13 Figure 5.3: Feedback Control Model state, curiosity. In collaborative case of classroom with multiple students, our system infers individual emotions and profiles, and then make a suggestion to the instructor for changing speed or content based on majority voting. In case of the online learning system, the model gracefully degrades to a single user system where the system automatically adapts based on the current user’s inputs. For example, if a student is in boredom state so we can change the content of the slide to introduce more illustration based content for same topic, assuming that we have this illustrative content. Similarly, we can make decisions on speed of slides. This decision making implementation is based on a regression model which is discussed in detail in the system implementation section. The decision are at the same time also based on the student’s profile/preferences, like if he/she is a slow learner or prefers illustrations over descriptions. This student profile is generated by a questionnaire, incorporated in the android application as seen in Appendix D, before the start of the lecture, which can be easily stored online to form student profile.

14 Chapter 6

System Implementation

We describe the system implementation with respect to the sensor system, feature extraction and emotion recognition in this section. The complete system can be modelled as in ﬁgure 6.1, with each component and its functions as depicted.

Figure 6.1: System Components

6.1 Sensor system

The system hardware is based on the Microsoft Band 2 smartwatch, which has sensors such as heart rate, galvanic skin response, skin temperature, barometer, accelerometer and more. The Microsoft Band proves to be a perfect ﬁt for our system because of the sensors, particularly heart rate, galvanic skin response and skin temperature, and the ease of customization to sensor data collection. The Microsoft Band is also able to give RR intervals as one of the payloads which is essential in extracting one of the features, namely heart rate variability, which is discussed in detail later.

15 Reliable data collection, like in any sensor system, depends on the sampling frequency. The band has different sampling frequencies for different sensors. The sampling rate for different sensors of Microsoft Band 2 can be different, but default sampling rate for heart rate, GSR and skin temperature sensors is 1 Hz. Apart from the sampling rate, one other key factor for reliable data collection is the placement of sensors, which is because of the nature of sensors like photoplethysmograph and galvanic skin response. The Microsoft band 2 design takes care of this factor as it allows user with different wrist sizes to adjust the grip and get a snug fit, which allows for sensors to collect data reliably without any significant errors.

6.2 Feature extraction

The sensor system, the Microsoft Band 2, provides data from the user to the software subsystem, which is essentially an android data collection application. The android device connects with the band over BLE (Bluetooth Low Energy) and the application subscribes to the sensor data streams. Our system subscribes to ﬁve data streams from Microsoft Band, namely heart rate data stream, heart rate quality stream, GSR data stream, RR interval data stream and skin temperature data stream. All this raw data is processed by the android app to derive features for machine learning process.

Our system derives features using basic statistical methods-

• Running Mean- The running mean feature computes a sample of mean values over time for the input signal using a rectangular window that is shifted over the time. The size of rectangular window depends on the input signal.

• Running deviation- This feature computes the local standard deviation using a rectangular window over time.

Our system computes computes and collects these six diﬀerent features, namely heart rate, skin conductance level (SCL), skin conductance response (SCR), skin temperature level (STL), skin temperature response (STR) and heart rate variability (HRV). The features of heart rate, STR and SCL are the running mean features computed from the collected data over window of 50 samples, while the features of HRV, SCR and STR are running deviation features over window of 50 samples. The HRV is extracted from the RR-interval data stream using the RMSSD (Root mean square of successive diﬀerences) method.

16 The android app, after collecting the sensor data and computing all the features, then stores the data to a cloud database which then can be easily accessed by the e-learning application.

6.3 Emotion Recognition

Having recognized the features and signals to be used for the system, it is essential to devise a technique to infer emotion from these signals and features. There has been limited research in ﬁeld of emotion recognition using bio-signals, unlike other methods such as face recognition, and thus there is limited knowledge about what levels or values of bio-signals accompany which emotion. There is an inherent problem of determining the ground truth.

Emotion recognition is done in the desktop learning application. The data stored in the cloud in last step is fetched by the application. The inference of emotion from the sensor data and extracted features uses machine learning. We test two diﬀerent machine learning algorithms- k-nearest neighbour algorithm and multi-class state vector machine. These algorithms both are capable of classifying multi-dimensional data into multiple classes.

Both of these algorithms are supervised and thus need training set. We conduct preliminary studies to collect data and create training set. These preliminary studies are explained in detail in evaluation section to follow. This training set is then used to classify the real time data using machine learning algorithm. Our studies show that SVM gives better accuracy than kNN which is supported by many other studies [37][38]. The SVM gives an accuracy of 83% and 85% for valence and arousal respectively, while kNN gives 75% and 79% accuracy for valence and arousal respectively. Thus, we use SVM as our machine learning algorithm for the system.

6.4 Feedback Control

As discussed before, out feedback control system involves making decisions of change of speed or content based on the current emotion state and profile of the student. Our system tries to bring the student to the optimal emotion state. The feedback control model can be logically divided into two subsections- emotion transition informer and feedback decision maker, which has been depicted in figure 3. We depict the detailed feedback control in figure 6.2, which shows two PI controllers and a regression model block. These are further explained in following sections.

17 Figure 6.2: Feedback Model

6.4.1 PI Controllers

We propose a PI controller based stage to find the optimum transition between current state to the optimum emotion state. We define emotions on a 2D plane-valence and arousal. Thus, we use two PI controllers, one each for valence and arousal, to find the emotion transition information in terms of error between current state and optimum state. Both PI controllers have the same system, decision maker, being controlled. The collective output from both arousal PI and valence PI stage provides the information about distance to optimum emotion state. A PI controller is a good fit for our system, because it has a slower response than a Proportional controller which is ideal for emotion study as these are gradual in change. Also, PI controller considers the historic values which is desired as a person’s current emotions are influenced by past emotions. We develop our PI controller model based on control theory [39] and inspired by results of another bio-feedback based study [40].

Figure 6.3: PI Controller

Figure 6 shows the basic PI controller where process is the emotion control process through system adaptation, having the transfer function G(z). The input to controller

18 R(z) representing the desired optimal emotion (arousal and valence), Y(z) representing the current emotion (arousal and valence), U(z) representing the suggested emotion change and E(z) representing the diﬀerence between current and optimum emotion. The transfer function of the PI controller is given as:

U(z) (K + K )z − K = P I P (6.1) E(z) z − 1

The transfer function of the feedback loop [39] is given as:

[(KP + KI )z − KP ]Gz Fr(z) = (6.2) (z − 1) + [(KP + KI )z − KP ]Gz

The important step is to identify the system being controlled. We assume a ﬁrst order system [39], based on regression model as discussed in next section, with the transfer function as: a G(z) = (6.3) z − b Taking the inverse z-transform of the above equation (3), we get the time-domain response as: y(k − 1) = au(k) + by(k) (6.4)

The term u is the decision model equation which is discussed in more detail in the next section. The a and b variables are student speciﬁc and are based on linear regression on historical data of y(k-1), u(k) and y(k) [40].

Now according to control theory, there are four design goals that need to be deﬁned for any controller- stability, steady-state error, settling time and maximum overshoot. The

stability is deﬁned when poles of Fr are inside unit circle [39], [40] and we tolerate

maximum steady state error of 0.5 valence/arousal. The settling time (Ks) is 1 unit

which is one lecture slide and allowing maximum overshoot (Mp) of 10%.

Now, we ﬁnd the desired roots, re±jθ of the characteristic polynomial according to our design goals. According to control theory, we get:

r = e−4/ks = 0.018 (6.5)

πln(r) θ = = 18.19 (6.6) ln(Mp) The characteristic equation with these poles is given as:

(z − re+jθ)(z − re−jθ) = z2 − 0.036z + 0.00032 (6.7)

19 We get the modelled characterstic equation by replacing G(z) in equation 2 and taking denominator of transfer function:

2 z + [a(KP + KI − (1 + B)]z + (B − AKP ) (6.8)

Comparing the desired and modelled characteristic equation, we solve for KP and KI and get: 0.964 b − 0.00032 K = ,K = (6.9) I a P a

6.4.2 Decision Model

The decision model takes as input the arousal and valence delta functions and student preference information fed to student. Based on all these inputs it decides how much speed is to be changed and if content presentation change is required or not. There have been some studies [41] which have tried implementing such a model in learning environment to suggest changes but these rely mostly on intuitive solution rather than a mathematical model.

We base our decision model on multiple parameters. Firstly, student’s current emotional state and distance from the optimal emotional state in terms of arousal and valence delta functions. Secondly, student specific data- student profile, which tells the model about preferences such as fast learner/ slow learner and visual learner/ verbal learner, and rate of information, which is system derived student specific parameter to know how fast the student is learning the slides. The rate of information is an important parameter which can be considered as controlled variable as the speed of slides and content defines the rate of information. This variable tells us about the learning rate at which student is settling and thus is an important decision making point.

The relation between these input variables and the decisions- how much speed and illustrative/ narrated content, is modelled using multivariate regression model. A regression based model for learning is Special preliminary experiment is performed to form the data set for the regression study where the students are freely allowed to control the speed of the slides and type of slides. These choices along with the emotional data is recorded, which forms the data set for modelling. A multivariate regression model equation is given as:

Y = βo + β1 ∗ X1 + β2 ∗ X2 + ... + βi ∗ Xi (6.10)

Y is the dependent variable, speed and content, which depends on independent variables X. The X variables include the output of the PI stages representing suggested change

20 in emotion, the student’s objective preferences in terms of content and the rate of information. The beta coeﬃcients are obtained from the regression modelling and are calculated every time the student starts using the system. The historical data is stored for X variables along with the student preferences.

21 Chapter 7

System Evaluation

We present description for three diﬀerent experiments in this section. First, we evaluate Sensemo sensor platform which includes the Microsoft band and android application. Then we conduct an empirical study to collect data which is used in evaluating emotion recognition subsystem. Then we conduct another set of empirical experiments to evaluate the feedback control system.

7.1 Platform Evaluation

We evaluate the Sensemo platform on two parameters- Energy Consumption and Mem- ory Footprint. The energy consumption is an important factor because of the fact that the Android device and Microsoft Band both are battery powered and it involves heavy data collection. Similarly the memory footprint is an important factor to ensure that the data collection app on the android device runs smoothly without lag for real time data collection.

The battery consumption for android device and Microsoft band were monitored through native system application. The estimated battery-life for the Microsoft Band with the Sensemo app connected is 5 hours, which is well beyond a typical class session. The estimated battery-life of an android device, with 2600 mAh battery, running the Sensemo application is 18 hours as the Sensemo app uses just 340 mW power additionally on an android device.

The android application is a small memory package. It only takes 3.28 MB of memory space in internal memory. The application, on average, uses just 215 KB of run time memory space, which is really low.

22 State Screen Bluetooth Band Power Standby Off Off Disconnected ≤ 22mW App Off On Disconnected 273 mW App Off On Connected 340 mW

Table 7.1: Power usage breakdown of application running on Android device

7.2 Emotion Recognition Evaluation

We evaluate the emotion recognition subsystem on two parameters - accuracy and latency. The emotion feedback data is the bases of our system and thus accuracy of this data and quick response of emotion recognition system is essential.

7.2.1 Experimental Setup

For data collection for diﬀerent emotion states it is important to determine a way to provide stimuli for such emotions. There may be multiple ways of producing the stimuli but the method needs to be practical and executable in a laboratory setting for experiments and at the same time be ethical. There are many diﬀerent types of stimuli possible, such as audio, video or pictures etc. The consistency and reliability of such stimuli is essential to the experiment.

The international affective picture system (IAPS) photoset provides solution to all these problems. The IAPS photoset has a collection of 1182 pictures which are rated by several researchers for the valence and arousal values. These values provide the suitable target valence and arousal values for the sensor readings and proves to be a good starting point for mapping and classification of sensor readings to the valence-arousal space. The IAPS photoset has advantages that it is reliable and very well suitable for a laboratory setting for experiments. We define a procedure of using these pictures. We first give user 5 mins to rest and then show him 5 low arousal, medium valence pictures for 15 mins. Then we conduct the experiment by showing the desired pictures. We again show low arousal, medium valence pictures between two experiments to get the user to the neutral emotion state. It is observed that high arousal, low valence pictures have strong negative content like mutilation, which long lasting effect on user and can effect subsequent readings for other pictures. So, these types of picture experiments are conducted at the end of the duration of session.

23 7.2.2 Accuracy

The accuracy of emotion recognition is important towards the goal of the system. The IAPS photo-set helps in evaluating the accuracy of the emotion recognition subsystem, in addition to forming the training set. The above mentioned experiment setup is used to record data for each picture from the set. To form the learning set, we choose pictures from all the diﬀerent ranges of valence and arousal ranging from very low to very high. We collect data for 200 pictures from the set with 100 pictures overlapping among the volunteers and 100 distinct pictures to cover diﬀerent range of valence and arousal. All the volunteers were graduate students and followed the above mentioned procedure. This forms the training set with the recorded sensor data and with rated arousal and valence values (which are available from IAPS) as classes. We made sure to balance the number of data points for each range category of arousal and valence- low, mid and high, so that each range had equal number of data points.

We then use support vector machine (SVM) for supervised classification. We train and test our SVM classifier using weka machine learning tool [42]. We perform 10-fold cross validation to test for accuracy of classification. The confusion matrix shows the results of the experiment for valence and arousal in table 7.2 and table 7.3 respectively. The experiments show that SVM has overall accuracy of 79.1% and 84.7% for valence and arousal respectively.

Low Mid High Low Mid High Low 80.0% 2.8% 17.2% Low 77.7% 22.2% 0% Mid 8.5% 68.5% 22.8% Mid 3.8% 85.8% 1.0% High 2.9% 13.0% 84.1% High 1.6% 11.8% 86.4%

Table 7.2: Valence Levels Table 7.3: Arousal Levels

7.2.3 Reliability and Latency

The response latency and reliability of emotion recognition is also important towards the goals of the system. It is important to devise a method to test that the system is able to give reliable data for instantaneous stimuli. The method itself should be reliable so that the results are reliable. Many research works [43] and [44] discuss about the eﬀect of video and music clips on emotion of the users. The emotional responses to video and music eﬀects are strong to be easily observed.

We conduct another set of experiments where the users are made subject to video stimuli. The videos are music videos chosen to make sure there is certain element of strong emotional stimuli intuitively at certain point of time. The aim of the experiment is to

24 (a) GSR Response (b) HRV Response

Figure 7.1: Physiological response of users to video stimuli

find out if our system records similar pattern of physiological signals from different users for same stimuli which has been proposed by other studies as discussed earlier. We show different sensor data streams in response to same music video for three users among the study group.

Figure 7.2: Skin temperature response of user to video stimuli

The plots show how the response patterns of GSR and HRV are similar for different users, which signifies that, firstly indeed we can rely on physiological signal patterns for inferring emotions and secondly, our system is well capable of recording such data reliably. We can also visually notice that GSR response shows exactly same pattern at around 50 sec mark, which is due to a strong visual stimuli and for all three users it is under 3 sec range. Also, we can see from figure 7.2, that skin temperature change does not have unique patterns with short term emotions and just rises over the time. We also find that correlation coefficient of skin temperature to valence and arousal came out to be significantly lower than other features. However, keeping it as a feature in the training set did not reduce the accuracy of classifier.

25 7.3 Control System Evaluation

The feedback control system is responsible for adapting the system based on the current emotions of the student. For the correct and eﬃcient working of the system, it is important that the feedback control provides adaptations for the right moment of emotion detected. In other words, it should be able to respond fast enough to cater to the right moment. Thus, response time of feedback control system and, in eﬀect, of the whole system should be low. Another important parameter is the acceptance of the feedback system by students.

7.3.1 Response Time

We test the feedback system for its response time with respect to the inferred emotion data to the adaptation decision. The key factor affecting this delay is the stability time of the PI stage. We also include the time taken by the emotion classifier into account. Thus, this response time analysis helps understand the time taken by the system to provide a useful suggestion/adaptation once the user’s data is made available. We find this response time through software timers. These experiments are conducted by using real lecture slides obtained from the MIT open courseware where no adaptation is provided to users but only recorded for purpose of analysis. The average response time of the feedback system comes out to be 3 sec which is significantly less than the duration of any single lecture slide.

7.3.2 Fitness of Model

We model the adaptation/suggestion maker system using multiple linear regression model, which can seem overly simplified for a system modelling physiological response of human body to a wide range of stimuli. However, our system involves incremental learning through continuous recording of data from users while they are involved in interacting with the system. Thus, statistically the model fitness should improve over the number of slides viewed by the user. We evaluate this fitness of model by computing the R2, which is the correlation between the observed responses of system and predicted responses of system. The closer the value to 1, the better the model fitness is. It is observed that the fitness of model improves for 150 or more slides.

26 Figure 7.3: Model Fitness

7.3.3 System Acceptance

The adaptation/suggestion is meant to facilitate student’s learning experience with his/her minimum eﬀort. We create another parameter of evaluation as the number of interventions by the student to adjust the speed or slide of the lecture as test for feedback acceptance for the online learning system. The less the number of interventions are the better the performance of feedback system. We note these interventions through software counter. We ﬁnd that on average student interventions are 2 per minute, which is acceptable. We also conduct user surveys for classroom system to get user’s response about their experience, which is discussed more in next chapter.

27 Chapter 8

Real Deployment

We deploy our system for learning purpose with 22 diﬀerent graduate students. The system deployment is done in two diﬀerent setups separately- the online learning prototype system and the classroom system.

8.1 Online Learning System

The students are allowed to take any lecture slide, but we make sure that the students are new to the lecture topic for the sake of evaluating. The lectures are of diﬀerent lengths for diﬀerent topics. The topics varied from biology to geography.

8.1.1 Experimental Setup

We implement our real-time emotion recognition system with our Java based learning system and adapting content from MIT open courseware [32]. We make sure that none of the students participating in the experiment has any prior knowledge of the content. We, thus, use the topic of biological anthropology from MIT open courseware as all the students participating in the experiment have an engineering background.

We experiment with twenty-two students. All the students who participated were graduate students and unaware of the topic of lecture. We use the feedback methods on half of the participants and no feedback for the other half. However, we use the same system and also make all the users wear the emotion recognition system to ensure similar environments. We then show each user the lecture slides and also record the emotional data during the course of experiments. Each experiment duration ranges between 30-50

28 minutes. The experiment ends with a small quiz with 10 simple questions on the topic, with 1 credit for each right answer

8.1.2 Result

We compare the performance of students based on quiz. We assume that better learning experience helps students answer more questions correctly. We also compare the data on emotional regions in which the students stay during the duration of experiment. We evaluate our system based on how long does it keep the user in the desired emotional state, curiosity, compared to the same setup without using the adaptive feature. We conduct these experiments with two different groups of 11 students each. The groups were formed randomly. We conduct these experiments with 4 different lectures. The figure 8.1 shows that our system is able to keep the students in the curiosity emotional state for longer duration of time compared to non-adaptive system. While it is possible that other factors can influence this result, like student’s focus level etc., our system still makes best-effort to improve the learning experience and arguably provides better result in all of the lecture experiments.

Figure 8.1: System Comparison

We ﬁnd that the average score for participants without adaptive learning system is 5 while it is 8 for participants who used adaptive learning system. The emotional data also shows that the participants who used adaptive learning system stayed in the “optimum emotion state” for longer duration during the lecture.

8.2 Classroom System

We deploy our system for classroom based lecture, where an instructor is teaching a class of multiple students.

29 8.2.1 Experimental Setup

We deployed our real-time emotion sensing system with an informational interface for the instructor, which provides suggestions to the instructor based on the student’s data. This deployment experiment involved 10 computer engineering graduate students. The experiment was divided into two parts- ﬁrst session without the real-time feedback to instructor, and second session with the real-time feedback to professor based on student’s data. For keeping the two sessions exactly the same and to be able to collect data from students in both sessions, students were made to wear the smartwatches in both the sessions. The android applications then collect the data from their respective Microsoft Band, and send it to central server. The major diﬀerence between these two sessions being the feedback to the instructor. The instructor was provided with the real-time suggestions during the second session to help him adapt his lecture. During the lecture experiment, due to low network availability, data from one user did not register to the server and data from another user only registered during the duration of second session.

The two lecture sessions involved topics of mergesort algorithm and quicksort algorithm respectively, which are relevant to the computer engineering students and so the students were expected to be interested in the lecture. These two algorithms were considered because of their similar diﬃculty and applications. The mergesort algorithm was explained without using our system and quicksort algorithm while using the system. The sessions started with the student signing into the android application and answering the questions to build their student proﬁle/preferences. Each session ends with short quiz, as in Appendix A and B, with questions related to the respective topics. At the end of experiment, a short survey, as in Appendix C, is taken by students with questions about system acceptability. It was made sure that instructor had extra animations and images for second session which he would only use if suggested by the feedback.

8.2.2 Result

We compare our real-time feedback system with the traditional system, based on two parameters- the quiz results and the ability of our system to keep the students in optimum emotional state of curiosity. The first session involved the merge sort algorithm. Since, the topic is considered relevant to the computer engineering students, the response is expected to be positive. The data collected from this session is summarized in the figure 8.2, where we can see that the students are mostly seen in satisfaction emotional state, with the curiosity being the least significant emotion. Even though satisfaction can be considered a positive state, but constructive learning happens mostly in curiosity state as mentioned earlier.

30 Figure 8.2: Emotional response to First Session

Similarly we collect data for session two, which is summarized in figure 8.3. We can easily visualize that students experienced curiosity more than first session. Thus, while using our system it tends to promote the curiosity emotional state, although it also depends on instructor as we give him/her freedom to take appropriate action. Also, the count of boredom emotional state reduced significantly. One important outcome was that we were able to finish second session in significantly less time than first session, even with same amount of information and still got improved results. At the end of both sessions we conduct quiz and compare the results between two sessions, in figure 8.4. The quiz involved 5 questions each of comparable difficulty, with 1 credit for each question. We can see that the students performed significantly well on average in second session.

Figure 8.3: Emotional response to second session

Also, based on the survey results, the students did not find using smartwatch as distracting or inconvenient during the lecture. Some students did find that they experienced better learning in second session compared to first, even though first session lasted longer for exactly similar amount of information.

31 Figure 8.4: Quiz results for two sessions

32 Chapter 9

Conclusion

Learning and education is one important phase of life. Now, with advent of online learning environments and smart classes, there are new possibilities of improving these systems. With the increasing popularity of wearable technologies with sensor capabili- ties, we now have greater access to personalized data from users. A system which can use this personal data to improve user’s learning experience can tremendously help improve the learning curve of students and help boost their interest and curiosity into the subject matter.

Sensemo, is such system which uses the real-time data from bio-sensors available in commercial smartwatches and infers user’s emotions with accuracy of 79.1%-84.7% for valence-arousal respectively, and provides personalized adaptive feature for present systems. We evaluate the use of sensemo system in two environments- online learning system and classroom system. The online learning system is single user system where Sensemo automatically adapts the lecture speed and content based on user’s emotions, while also allowing user to manually move back and forth. The classroom system is multi-user based system where the Sensemo provides suggestion to instructor for changing speed or content based on collaborative data of students. We evaluate our system for its ability to keep user in optimal emotional state and through student survey feedback, and ﬁnd that our system provides improvement over the non-adaptive system by keeping the student in curious state of emotion and is also unobtrusive and non-distracting for students to use based on the survey results.

9.1 Future Work

An adaptive system for learning environment based on user’s real time response has great potential to improve the learning experience as proved by our results. However, such 33 a human-in-the-loop system at the same time has many possibilities for improvement. Possibility to include more sensors or diﬀerent modalities can be explored to improve emotion inference and current state of students. Since our models use machine learning and statistical regressions, the system accuracy can always be improved by collecting more data from more users. The content suggestions can also be improved by devising a means of classifying each slide for the amount and type of information it contains, so that our system can better adapt the content. Overall, the current system has proved to be promising to improve the learning experience of students and can be further improved through ﬁne tuning the sub parts of the system.

34 Appendix A

Appendix- Quiz Mergesort

Quiz- Merge Sort

1. Is Merge Sort an in-place algorithm?

2. What is the time-complexity of the divide stage of merge sort?

3. What is the time-complexity of the merge stage of the merge sort?

4. What is the best-case overall time complexity of the merge sort?

5. What is the worst-case overall time complexity of the merge sort?

35 Appendix B

Appendix- Quiz Quicksort

Quiz- Quick Sort

1. Is Quick Sort an in-place algorithm?

2. What is the time-complexity of the divide stage of quicksort?

3. What is the time-complexity of the merge stage of the quicksort?

4. What is the best-case overall time complexity of the quicksort?

5. What is the worst-case overall time complexity of the quicksort?

36 Appendix C

Appendix- Survey

Survey

1. Did you ﬁnd any diﬀerence in the lecture sessions of mergesort and quicksort?

2. Did you ﬁnd any inconvenience wearing smartwatch during the lecture? If so, please explain.

3. Did you ﬁnd it distracting to wear the smartwatch during the lecture?

4. Which one do you think you understood more- mergesort or quicksort?

37 Appendix D

Appendix- Student Proﬁle

38 Bibliography

[1] Abhinav Dhall, Akshay Asthana, Roland Goecke, and Tom Gedeon. Emotion recognition using phog and lpq features. In Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on, pages 878–883. IEEE, 2011.

[2] Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza Bulut, Chul Min Lee, Abe Kazemzadeh, Sungbok Lee, Ulrich Neumann, and Shrikanth Narayanan. Analysis of emotion recognition using facial expressions, speech and multimodal information. In Proceedings of the 6th international conference on Multimodal interfaces, pages 205–211. ACM, 2004.

[3] Akshay S Utane and SL Nalbalwar. Emotion recognition through speech. Interna- tional Journal of Applied Information Systems (IJAIS), pages 5–8, 2013.

[4] Yixiong Pan, Peipei Shen, and Liping Shen. Speech emotion recognition using support vector machine. International Journal of Smart Home, 6(2):101–108, 2012.

[5] Saurabh Singh, Vishal Sharma, Kriti Jain, and Ravi Bhall. Edbl-algorithm for detection and analysis of emotion using body language. In Next Generation Com- puting Technologies (NGCT), 2015 1st International Conference on, pages 820–823. IEEE, 2015.

[6] Stefano Piana, Alessandra Stagliano, Francesca Odone, Alessandro Verri, and An- tonio Camurri. Real-time automatic emotion recognition from body gestures. arXiv preprint arXiv:1402.5047, 2014.

[7] John L Andreassi. Psychophysiology: Human behavior & physiological response. Psychology Press, 2013.

[8] Robert W Levenson. The autonomic nervous system and emotion. Emotion Review, 6(2):100–112, 2014.

[9] Rui Guo, Shuangjiang Li, Li He, Wei Gao, Hairong Qi, and Gina Owens. Perva- sive and unobtrusive emotion sensing for human mental health. In Proceedings of

39 the 7th International Conference on Pervasive Computing Technologies for Health- care, pages 436–439. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2013.

[10] NK Suryadevara, T Quazi, and Subhas C Mukhopadhyay. Smart sensing system for human emotion and behaviour recognition. In Perception and Machine Intelligence, pages 11–22. Springer, 2012.

[11] Fatma Nasoz, Kaye Alvarez, Christine L Lisetti, and Neal Finkelstein. Emotion recognition from physiological signals using wireless sensors for presence technologies. Cognition, Technology & Work, 6(1):4–14, 2004.

[12] Hong Lu, Denise Frauendorfer, Mashﬁqui Rabbi, Marianne Schmid Mast, Gokul T Chittaranjan, Andrew T Campbell, Daniel Gatica-Perez, and Tanzeem Choudhury. Stresssense: Detecting stress in unconstrained acoustic environments using smart- phones. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 351–360. ACM, 2012.

[13] Chung-Hsien Wu, Ze-Jing Chuang, and Yu-Chung Lin. Emotion recognition from text using semantic labels and separable mixture models. ACM transactions on Asian language information processing (TALIP), 5(2):165–183, 2006.

[14] Clayton Epp, Michael Lippold, and Regan L Mandryk. Identifying emotional states using keystroke dynamics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 715–724. ACM, 2011.

[15] Robert Horlings, Dragos Datcu, and Leon JM Rothkrantz. Emotion recognition using brain activity. In Proceedings of the 9th international conference on computer systems and technologies and workshop for PhD students in computing, page 6. ACM, 2008.

[16] Akane Sano and Rosalind W Picard. Stress recognition using wearable sensors and mobile phones. In Aﬀective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on, pages 671–676. IEEE, 2013.

[17] Kiavash Bahreini, Rob Nadolski, and Wim Westera. Towards multimodal emotion recognition in e-learning environments. Interactive Learning Environments, 24(3): 590–605, 2016.

[18] Kiavash Bahreini, Rob Nadolski, and Wim Westera. Towards real-time speech emotion recognition for aﬀective e-learning. Education and Information Technologies, pages 1–20, 2015.

40 [19] Guillaume Chanel, Cyril Rebetez, Mireille B´etrancourt,and Thierry Pun. Emo- tion assessment from physiological signals for adaptation of game diﬃculty. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 41 (6):1052–1063, 2011.

[20] Stylianos Asteriadis, Paraskevi Tzouveli, Kostas Karpouzis, and Stefanos Kollias. Estimation of behavioral user state based on eye gaze and head pose—application in an e-learning environment. Multimedia Tools and Applications, 41(3):469–493, 2009.

[21] Boundless. Boundless Psychology. Aug 2016. URL https://www.boundless. com/psychology/textbooks/boundless-psychology-textbook/emotion-13/ biology-of-emotion-68/biology-of-emotion-261-12796/.

[22] Andreas Haag, Silke Goronzy, Peter Schaich, and Jason Williams. Emotion recognition using bio-sensors: First steps towards an automatic system. In Tutorial and research workshop on aﬀective dialogue systems, pages 36–48. Springer, 2004.

[23] Paul Ekman, Robert W Levenson, and Wallace V Friesen. Autonomic nervous system activity distinguishes among emotions. Science, 221(4616):1208–1210, 1983.

[24] Jason J Braithwaite, Derrick G Watson, Robert Jones, and Mickey Rowe. A guide for analysing electrodermal activity (eda) & skin conductance responses (scrs) for psychological experiments. Psychophysiology, 49:1017–1034, 2013.

[25] ME Dawson, AM Schell, and DL Filion. The electrodermal system. cacioppo, jt, tassinary, lg, & berntson, gb (eds), handbook of psychophysiology (pp. 200–223), 2000.

[26] Bradley M Appelhans and Linda J Luecken. Heart rate variability as an index of regulated emotional responding. Review of general psychology, 10(3):229, 2006.

[27] Helmut Mayer and Peter H¨oppe. Thermal comfort of man in diﬀerent urban environments. Theoretical and applied climatology, 38(1):43–49, 1987.

[28] Microsoft. Microsoft band documentation. https://developer.microsoftband.com/Content/docs.

[29] Lisa Feldman Barrett. Discrete emotions or dimensions? the role of valence focus and arousal focus. Cognition & Emotion, 12(4):579–599, 1998.

[30] Elizabeth A Kensinger. Remembering emotional experiences: The contribution of valence and arousal. Reviews in the Neurosciences, 15(4):241–252, 2004.

[31] Peter J Lang, Margaret M Bradley, and Bruce N Cuthbert. International affective picture system (iaps): Technical manual and affective ratings. Gainesville, FL: The Center for Research in Psychophysiology, University of Florida, 1999. 41 [32] Jonathan Posner, James A Russell, and Bradley S Peterson. The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology, 17(03):715–734, 2005.

[33] Paul Ekman, Wallace V Friesen, Maureen O’Sullivan, Anthony Chan, Irene Diacoyanni-Tarlatzis, Karl Heider, Rainer Krause, William Ayhan LeCompte, Tom Pitcairn, Pio E Ricci-Bitti, et al. Universals and cultural diﬀerences in the judg- ments of facial expressions of emotion. Journal of personality and social psychology, 53(4):712, 1987.

[34] Barry Kort, Rob Reilly, and Rosalind W Picard. An aﬀective model of interplay between emotions and learning: Reengineering educational pedagogy-building a learning companion. In icalt, volume 1, pages 43–47, 2001.

[35] Regan L Mandryk and M Stella Atkins. A fuzzy physiological approach for contin- uously modeling emotion during interaction with play technologies. International journal of human-computer studies, 65(4):329–347, 2007.

[36] Renate Nummela Caine, Geoﬀrey Caine, Carol McClintic, and Karl J Klimek. 12 Brain/Mind Learning Principles in Action: Teach for the Development of Higher- Order Thinking and Executive Function. Corwin Press, 2015.

[37] Panagiotis C Petrantonakis and Leontios J Hadjileontiadis. Emotion recognition from eeg using higher order crossings. IEEE Transactions on Information Technol- ogy in Biomedicine, 14(2):186–197, 2010.

[38] Kyung Hwan Kim, SW Bang, and SR Kim. Emotion recognition system using short-term monitoring of physiological signals. Medical and biological engineering and computing, 42(3):419–427, 2004.

[39] Joseph L Hellerstein, Yixin Diao, Sujay Parekh, and Dawn M Tilbury. Feedback control of computing systems. John Wiley & Sons, 2004.

[40] Shahriar Nirjon, Robert F Dickerson, Qiang Li, Philip Asare, John A Stankovic, Dezhi Hong, Ben Zhang, Xiaofan Jiang, Guobin Shen, and Feng Zhao. Musicalheart: A hearty way of listening to music. In Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems, pages 43–56. ACM, 2012.

[41] Liping Shen, Minjuan Wang, and Ruimin Shen. Aﬀective e-learning: Using” emotional” data to improve learning in pervasive learning environment. Educational Technology & Society, 12(2):176–189, 2009.

[42] Mark Hall, Eibe Frank, Geoﬀrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. The weka data mining software: an update. ACM SIGKDD explorations newsletter, 11(1):10–18, 2009. 42 [43] Alan Hanjalic and Li-Qun Xu. Aﬀective video content representation and modeling. IEEE transactions on multimedia, 7(1):143–154, 2005.

[44] Fernando Silveira, Brian Eriksson, Anmol Sheth, and Adam Sheppard. Predict- ing audience responses to movie content from electro-dermal activity signals. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 707–716. ACM, 2013.