BandReader – A Mobile Application for Data Acquisition from Wearable Devices in Affective Computing Experiments∗ Krzysztof Kutt AGH University of Science and Technology [email protected] Grzegorz J. Nalepa AGH University of Science and Technology [email protected] Barbara Gizycka˙ AGH University of Science and Technology [email protected] Paweł Jemio{o AGH University of Science and Technology [email protected] Marcin Adamczyk AGH University of Science and Technology Abstract As technology becomes more ubiquitous and pervasive, special attention should be given to human-computer interaction, especially to the aspect related to the emotional states of the user. However, this approach assumes very specific mode of data collection and storage. This data is used in the affective computing experiments for human emotion recognition. In the paper we describe a new software solution for mobile devices that allows for data acquisition from wristbands. The application reads physiological signals from wristbands and supports multiple recent devices. In our work we focus on the Heart Rate (HR) and Galvanic Skin Response (GSR) readings. The recorded data is conveniently stored in CSV files, ready for further interpretation. We provide the evaluation of our application with several experiments. The results indicate that the BandReader is a reliable software for data acquisition in affective computing scenarios. I. IntroductionDRAFTeach user, mobile systems offer various meth- ods of personalization and customization of eople nowadays are getting more and these devices. However, as they become more more accustomed to technology that is and more miniature, they can be fit to pieces Ppervasive and ubiquitous. Besides using of hardware of a size of an accessory. Today’s computers at work and at home, many own at wearable technologies, such as wristbands, pro- least one mobile device, be it a smartphone or vide a number of advanced sensors to moni- a tablet. To address individual preferences of tor human activity and health condition. Fur- thermore, these devices could be used for de- ∗The paper is supported by the AGH University Grant. 1 BandReader livering data on human physiological states choice of words for slogans, titles, etc. How- for affective computing applications. Affec- ever, as technology is becoming more advanced tive Computing (AfC) [1] is a new computing and pervasive, a prescribed design begins to paradigm that has been gaining a lot of inter- lag behind the possibilities brought by the po- est recently. Affective computing systems can tential modern conveniences. An important use a range of signals for the classification of aspect of AfC systems is their ability to gather a person’s emotional state. Physiological signal data, process it and generate an answer dynam- monitoring is one of the commonly considered ically, and in real time – a mechanism widely cases. For many AfC systems it is important known as an affective loop. This basic concept to use non-intrusive and pervasive technolo- of AfC applications refers to the continuous gies. Therefore, wearable devices, including chain of interactions between the user and the wristbands, seem a promising solution. software. As a response to the user’s oper- Emotional aspect of human-computer inter- ations in the software, the system generates action is crucial for the process to have a nat- new content or modifies its characteristics, by ural and easy flow [2]. A number of studies means of colors, animations, or more subtle on usability1 has proven that careful system changes in the parameters. Then, the user can design, mindful of user’s emotional states and react to these new conditions, and so on. In responses has a great impact on the overall order to implement this in the application, how- user’s performance with the application. Our ever, one needs to adapt a specific approach tool can simplify the implementation of practi- and be aware of several issues that arise. cal AfC software on mobile devices, while also One difficulty in this context regards proper taking the affective aspect of interaction into data acquisition. Oftentimes, AfC systems use consideration. multiple sensors or complex platforms con- The rest of the paper is organized as fol- sisting of various devices. To ensure reliable lows: In Section II we stress the affective loop’s data interpretation, it is reasonable to con- role in HCI, In Section III we describe selected sider signals from more than one modality – hardware. Then in Section IV we specify our by which we mean both audiovisual data and application. In Section V we provide an experi- other user’s behavior metrics. Another prob- mental evaluation of our solution and situate lem is related to the temporal resolution of our approach in the context of other studies in data handling. Collecting affective information Section VI. Finally, in Section VII we summa- from the environment in asynchronous man- rize the paper and give directions for future ner and processing it offline is not sufficient work. for applying the affective loop [3] in the in- teraction. Synchronization and integration of II. Affective Loop in multimodal and multichannel data is therefore Human-Computer Interaction a grave issue. From the practical point of view, in order Recent methods in HCI emphasize the use of to create AfC systems that support wristbands information regarding human emotion on the as input devices, specific software solutions interface and interactionDRAFT level. The so-called need to be provided. In our work, we are in- emotive interfaces aim at such an interface de- terested in ubiquitous solutions that integrate sign that addresses the user’s affective state wearable sensors with mobile technologies. To directly, with use of various emotion inducing acquire and process data from wristbands, ded- techniques such as colors, photographs of hu- icated mobile applications have to be devel- man facial expressions of emotions, and careful oped. These applications need to address sev- eral of important requirements, such as oper- 1Usability refers to the characteristics of software that determine the degree of ease and efficiency with which ating in real time, or the support for different the application is used. hardware solutions. In this paper we present 2 BandReader such an application that we developed and ments are not supported by any satisfactory successfully evaluated in a number of AfC ex- software enabling to collect, store and mean- periments. BandReader is an important step in ingfully interpret data. satisfying these concerns, as it offers software Xiaomi Mi Band 2 is a simple band aimed at support for wearable devices that provide phys- fitness and sleep tracking. It is equipped with, iological data, which is necessary to develop among others, an optical HR sensor and a step the affective loop. counter. Xiaomi Mi Band 2 allows to check one’s HR at a certain point of time and shows III. Selected Wristbands for it on its display. In order to track it continu- ously, though, the device has to maintain the Affective Computing connection with the official smartphone soft- Analysis of current market state led us to selec- ware, Mi Fit. The full functionality is brought tion of bands that could be useful for presented by using the band with one or both of the devel- applications: Empatica E4, Microsoft Band 2, oper’s applications: Mi Band Notify & Fitness Xiaomi Mi Band 2, Apple Watch 3 and Scosche and Mi Band Tools. Rhythm+. All of the devices listed here pro- Apple Watch 3 is a brand new technology for vide at least a certain mode of HR monitoring, smartwatches. This wristband is essentially the which forms a basic condition from pervasive only smartwatch we included in our analysis. AfC development point of view. Considering The integrated components include accelerom- fitness tracking appliances developed by Fit- eter, barometric altimeter, and ambient light, bit, Garmin, Polar, Xiaomi or Apple, contin- gyro and HR sensors. With the preloaded Ap- uous HR measurement is provided in each ple watchOS operation system and Siri soft- case. Some companies, such as Scosche, de- ware, unlike Mi Band 2, Rhythm+ or E4, it is cided to focus just on rough cardiovascular meant to be a fully-developed smartwatch that activity monitoring. Few developers, namely supports SMS messaging, email, push mail and Empatica E4 and Microsoft Band 2, enhanced IM. their wristbands with more complex biomet- Scosche Rhythm+ is a HR sensor for exercise rics tracking. Both devices include Electroder- intensity measurement. The distinctive feature mal Activity (EDA) or GSR sensors, which aid for this device, as claimed by the developer, the user’s feedback with more comprehensive is its unmatching accuracy provided with the physiological body state overview. Wristbands application of Green/Yellow Optical Sensors are a solution that provides a compromise be- technology. Another notable feature is its dual- tween convenience and capability. This is not, mode processor, which allows simultaneous however, sufficient for achieving decent level data transmission between the Rhythm+, mul- of affect recognition and enhancing interaction. tiple ANT+ displays (such as sport equipment Easy and user-friendly body activity tracking at fitness centers) and the smarthphone app comes with numerous disadvantages regard- (paired via Bluetooth SMART) as well. The ing unusual use cases. Collected data is of- device, however is not equipped with any sort ten preprocessed before being presented to the of display or screen, so any real-time data mon- user. Therefore, the informationDRAFT that the user itoring has to be performed with external soft- acquires is already interpreted in a specific, ware. undocumented way. This is unfortunate for re- Empatica E4 is an advanced sensory wrist- searchers who would prefer access to raw data. band based on the technologies previously de- Information processing should be fully control- veloped in the Affective Computing division lable in order to apply best models for further of MIT Media Lab.