Visualizing Brain Patterns Information Systems and Computer Engineering

NeuroExplore - Visualizing Brain Patterns

A Physiological Computing InfoVis

Daniel Jose´ dos Santos Rocha

Thesis to obtain the Master of Science Degree in Information Systems and Computer Engineering

Supervisors: Prof. Sandra Pereira Gama Prof. Hugo Alexandre Teixeira Duarte Ferreira

Examination Committee

Chairperson: Prof. Lu´ıs Manuel Antunes Veiga Supervisor: Prof. Sandra Pereira Gama Members of the Committee: Prof. Joao˜ Antonio´ Madeiras Pereira

November 2017

Acknowledgments

I would like to thank my parents for their friendship, encouragement and caring over all these years, for always being there for me through thick and thin and without whom this project would not be possible. I would also like to thank my sibblings, grandparents, aunts, uncles and cousins for their understanding and support throughout all these years. I would also like to acknowledge my dissertation supervisors Prof. Sandra Gama and Prof. Hugo Fer- reira for their constant technical insight, motivational support and overall enthusiastic sharing of knowledge that has made this Thesis possible. Last but not least, to all my friends and colleagues - particularly database participants: Antonio´ Amorim, Pedro Costa and Jose´ Fernandes - that helped me grow as a person and were always there for me during the good and bad times in my life. Thank you. To each and every one of you – Thank you.

Abstract

The objective of this project is the creation of a Physiological Computing Information Visualization (InfoVis) interface through which, interactively, users can visually decipher one’s intricate emotions and complex mind-state. To this end, we cooperated closely with Neuroscience experts from Instituto de Biof´ısica e Engen- haria Biomedica´ (IBEB) throughout our work. Consequently, we assembled a Brain Computer Inter- face (BCI), from a Bitalino Do It Yourself (DIY) hardware kit, for retrospective and real-time biosignals visualization alike. The resulting wearable biosensor was successfully deployed in an extensive Database (DB) acquisition process, consisting of activities with concrete, studied brain-pattern correlations. This big-data InfoVis foundation’s magnitude and its, at times saturated, physical signal accredited the development of a data-processing pipeline. Indeed, our solution - entitled NeuroExplore - converts and presents this large number of digitalized, raw biosignal items into more recognizable visual idioms. The system interaction was intentionally designed in order to augment users’ discoveries and rea- soning regarding visually recognizable metrics, as well as subsequently derived trends, outliers and other brain patters. Strengthening this intent, we adopted an iterative development process in which, recurrently, expert needs and user suggestions were equated as orienting guidelines. This all culminated in a ﬁnal version we deemed worthy of extensive functional and utility user testing and expert validation. In the end, our project achieved both excellent user usability scores as well as expert interest, some already relying on our solution for their own research.

Keywords

InfoVis, Physiological Computing, Affective Computing, Brain-Computer Interface, biosignals, qEEG

iii

Resumo

O objectivo deste projecto e´ a criaçao˜ de uma interface InfoVis para Computaçao˜ Fisiologica,´ atraves´ da qual, interactivamente, os utilizadores consigam decifrar visualmente, emoçoes˜ intr´ınsecas e com- plexos estado de mente. Para este fim, trabalhamos consistentemente em proximidade com especialistas em Neuroscien- cia do IBEB. Consequentemente, constru´ımos uma Brain Computer Interface (BCI), usando um kit de hardware DIY Bitalino, para a visualizaçao˜ retrospectiva ou em tempo-real de biosinais. O biosensor wearable resultante foi empregue, com sucesso, num extenso processo de aquisiçao˜ de base de dados, dividido em actividades concretas cujas correlaçoes˜ com padroes˜ cerebrais foram estudadas. A magnitude destes dados - fundaçao˜ do nosso InfoVis - associada a` poss´ıvel saturaçao˜ destes jus- tifica a implementaçao˜ de preprocessamento. De facto, a nossa soluçao˜ - NeuroExplore - filtra, deriva e apresenta este enorme numero´ de dados fisiologicos´ em idiomas visuais mais facilmente reconhec´ıveis. A interacçao˜ com o sistema foi desenhada, intencionalmente, para aumentar as descobertas e racioc´ınio do utilizador - explicitamente - ao reconhecimento visual de metricas,´ bem como outros padroes˜ - subsequentemente - visualmente derivaveis.´ Fortalecendo esta intençao,˜ adoptamos um metodo´ iterativo de desenvolvimento onde, recorrentemente, equacionamos as necessidades de experts bem como sugestoes˜ de utilizadores como linhas mestras. Tudo isto culminou num prototipo,´ considerado apto para uma extensa validaçao˜ de funcionalidade e utilidade por parte de utilizadores e experts. No final, alcançamos´ notas de usabilidade excelentes bem como interesse por parte de especialistas, alguns dos quais ja´ utilizam a nossa soluçao˜ para a sua propria´ investigaçao.˜

Palavras Chave

Visualizaçao˜ de Informaçao,˜ Computaçao˜ Fisiologica,´ Computaçao˜ Afectiva, Interface Cerebro´ Com- putador, Biosinais, Eletroencefalografia Quantitativa

Contents

1 Introduction 1 1.1 Objectives...... 5 1.2 Contributions...... 5 1.2.1 KickUP Sports Accelerator...... 6 1.2.2 GrowUP Gaming recording sessions...... 6 1.2.3 Instituto de Apoio as` Pequenas e Medias´ Empresas e a` Inovac¸ao˜ (IAPMEI) Schol- arship...... 7 1.3 Document Structure...... 7

2 Background 9 2.1 Sentiment Analysis...... 11 2.2 Affective Computing...... 11 2.2.1 Affect Visualization...... 12 2.3 Physiological Computing...... 12 2.4 Brain Computer Interfaces (BCI)...... 14

3 Related Work 17 3.1 Emotion’s Visual Representation...... 19 3.2 Affective Computing...... 20 3.2.1 Text Analytics...... 20 3.2.2 Video Content Analysis...... 26 3.2.3 Physiological Computing...... 30 3.3 Discussion...... 35

4 Proposed Solution 37 4.1 Biosignal Input Solution...... 39 4.1.1 Device Conﬁguration...... 41 4.2 Architecture...... 42 4.2.1 Users...... 43 4.3 Development Process...... 44

vii 4.3.1 Biosignal Dataset Development...... 44 4.3.1.A International Affective Picture System (IADS)...... 47 4.3.2 Data preprocessing...... 47 4.3.2.A Saturation Filtering...... 48 4.3.2.B EEG Power Spectrum Calculus...... 49 4.3.2.C Photopletismography (PPG) Peak-Finding Algorithm...... 49 4.3.3 Derived Data...... 50 4.3.3.A Brain-wave extraction...... 50 4.3.3.B Heart-Rate metric...... 51 4.3.3.C Emotional-Valence metric...... 51 4.3.3.D Meditation metric...... 51 4.3.3.E Engagement metric...... 52 4.3.4 InfoVis Development...... 52 4.3.4.A Low Fidelity Prototype...... 52 4.3.4.B High Fidelity Prototype...... 53 4.3.4.C β Version InfoVis...... 54 4.3.4.D Final InfoVis...... 55 4.4 Discussion...... 57

5 Evaluation 59 5.1 Context: European Investigators’ Night 2017...... 61 5.2 Usability Tests...... 62 5.2.1 Methodology...... 62 5.2.2 Results...... 64 5.2.2.A Task Completion Times...... 65 5.2.2.B Errors...... 67 5.2.2.C System Usability Scale (System Usability Scale (SUS))...... 68 5.3 Case Studies...... 68 5.3.1 Methodology...... 70 5.3.2 Results...... 70 5.3.2.A User 1...... 71 5.3.2.B User 2...... 72 5.4 Discussion...... 72

6 Conclusion and Future Work 75 6.1 Future Work...... 77

A Code of Project 87

viii B NeuroTech Artifacts 89 B.1 NeuroTech Business Model Canvas...... 89 B.2 Gaming Tournament Conversation Feedback...... 89

ix x List of Figures

1.1 Cognitive Science’s comprising ﬁelds of study. Each line joining two disciplines represents interdisciplinary synergy [1]...... 3

2.1 Paul Ekman’s 6 categorical emotions: Happy, Sad, Fear, Anger, Suprised, Disgust.[2].. 12 2.2 Valence/Arousal Emotional Space [3]...... 12 2.3 International 10–20 system. An internationally recognized method to describe and apply the location of scalp electrodes in the context of an EEG test or experiment.A = Ear lobe, C = central, F = frontal, FP = frontal-polar, O = occipital, P = parietal, T = temporal [4].. 13

3.1 Thayer’s Arousal/Valence model [5]...... 19 3.2 Lee et al. newly proposed visualization [5]...... 19 3.3 Sanchez et al. proposal for emotion’s representation [6]...... 20 3.4 Sanchez et al. newly proposed visualization [6]...... 20 3.5 Staked Area Chart for each of Ekman emotions’ intensity over time [7]...... 21 3.6 Visualization (VIS) screenshot displaying the following idioms: Line chart, Pie chart, Radar chart [8]...... 22 3.7 Screenshots of four distinct text-extracted emotion VIS: a) avatars; b) emoticons; c) Hooloovoo; d) Synemania. [9]...... 22 3.8 On the left: Sadness over age Area chart; On the right: keyword’s usage during valentine’s day Line Chart [10]...... 23 3.9 InfoVis Interface: input blog (left), document structure and recognized emotions (middle) and the sentence’s emotion table (right) [11]...... 24 3.10 Chicago geo-tagged emotional Twitter messages VIS[3]...... 24 3.11 USA aggregated emotion tweets VIS[3]...... 24 3.12 Sentiment before Sep 2013 Autralian Election InfoVis[12]...... 25 3.13 Sentiment after Sep 2013 Autralian Election InfoVis[12]...... 25 3.14 Emotional Heat Map InfoVis[13]...... 26

xi 3.15 Emotional Saccade MapInfoVis[13]...... 26

3.16 Emotient InfoVis displaying Emotional Engagement current value with an angular meter idiom and evolution with a line chart (left) as well as Ekman’s emotions in a retrospective stacked area chart and its overall values in pie charts (right)1...... 27

3.17 EmoVu InfoVis usage of a area-chart which represent each of Ekman’s emotions with a distinctive color3...... 27

3.18 Affdex InfoVis comparing a user’s line chart for Expressiveness with other users according to age groups4...... 28

3.19 Kairos InfoVis presenting a subjects’ facial recording side by side with multiple retrospective area charts, one for each of Ekman’s emotions5...... 29

3.20 Nviso’s InfoVis usage of an area chart to represent each of Ekman’s emotions detected over time as a result of facial expressions’ detection (left)6...... 29

3.21 EEG’s Power Spectrum...... 30

3.22 Volumetric bull’s eye idiom [14]...... 30

3.23 Gavrilescu [14] proposed Electroencephalography (EEG) InfoVis...... 30

3.24 Valence/Arousal space-visualization mapping [15]...... 31

3.25 Emotional heat map; Valence/Arousal bar-chart [15]...... 31

3.26 Screenshot of AffectAura’s InfoVis[16]...... 32

3.27 Museum’s top view heat map [17]...... 32

3.28 Emotional Intensity bar charts and scale under painting [17]...... 32

3.29 Screenshot examples of iMotions’ InfoVis for distinct emotions7...... 33

3.30 Screenshot of Neurosky’s MindWave InfoVis8...... 34

3.31 Screenshot of bitalino’s OpenSignalsInfoVis. In it we can observe a line-chart for each of bitalino’s channel raw data over time [18]...... 34

4.1 Bitalino board with Electromyography (EMG)[18]...... 40

4.2 BrainBIT prototype Outside and Inside headband view detailing: (1) The Power Block; (2) Battery; (3) PPG; (4) Electrodes...... 40

4.3 BrainBIT BCI ﬁnal prototype used in this project...... 41

xii 4.4 System architecture. Different, arrow colors and directions represent different system interactions: In Black (a), we see the users’s (1) electrical and chemical biosignals (EEG and PPG) and the physical accelerometer input. In Red (b), BrainBit (2) listens for requests - sent over Bluetooth by ClientBIT (3) - which sends a raw, discrete domain, digitalized version of the user’s biosignals, accordingly. This version is then sent to Server- BIT (4) for post-processing. Indeed, in Green (c), we see the the Python Server (4), JSON-formatted, preprocessed data answer the request it received (in identical fashion) from ClientBIT (3) - a web-based, Javascript/HTML InfoVis interface.In Yellow, the input means of exploring our InfoVis - typically a computer mouse - is depicted. Finally, a large blue Blue, one-directional arrow, represents the optical NeuroExplore (5) InfoVis output, through which the user can visually analyze their own, previously collected, or real-time, BrainBIT Psycho-Physiological data. Finally, in Yellow, the input means of exploring our InfoVis - typically a computer mouse - is depicted...... 43 4.5 Recording Comma Separated Value (CSV) file screenshot snip depicting a six-line long, respective to one second, selection (a) with one thousand CSV each (b)...... 46 4.6 Raw biosignals (left) versus Preprocessed (right) database sizes, in GigaByte (GB).... 47 4.7 Initial NeuroExplore conceptual paper prototype artifact, dating February 2017...... 53 4.8 Initial NeuroExplore software prototype displaying live data acquisition, April 4th, 2017... 54 4.9 NeuroExplore retrospective mode beta-version InfoVis, featuring the VIS of post-processed, derived physiological metric’s charts. On the left we can see the maximazed menu. On top, its show/hide interface next to the file selection interface. On the bottom, the previous recording time slider with respective play/pause buttons is visible. The plotted idioms y-axis log-scale usage is also visible...... 55 4.10 NeuroExplore retrospective mode final-version InfoVis, introducing a broader VIS space due to input file-selection menu integration and mouse-over interaction,...... 56

5.1 IBEB’s Physiological Computing table at European Investigators’ Night 2017. On the bottom-right we can see a participant ﬁlling the SUS...... 61 5.2 Box Plot for user task completion times...... 66 5.3 Box Plot for number of user errors in each Task completion...... 67 5.4 Our project’s ﬁnal system usability score presented in Bangors’ Adjective Rating Scale [19] 69

B.1 NeuroTech Business Model Canvas (BMC)...... 91

xiii xiv List of Tables

2.1 Brain-wave frequency bands...... 14

4.1 Bitalino’s Technical Speciﬁcations [20]...... 40 4.2 Depiction of our Bitalino Channel-Sensor conﬁguration...... 41 4.3 Depiction of each session’s activities, besides meditation...... 45

5.1 Planned User Tasks Description...... 63 5.2 Final User Tasks...... 65 5.3 Task Completion Times...... 66 5.4 User Errors Detailed...... 67 5.5 SUS scores from each user...... 69

xv xvi List of Algorithms

4.1 EEG Raw Signal Saturation Detection for each second of input data...... 48

xvii xviii Acronyms

AC Alternate Current

AI Artiﬁcial Intelligence

BMC Business Model Canvas

BCI Brain Computer Interface

BMI Brain Machine Interface bpm Beats per minute

CIO Chief Information Ofﬁcer

CNS Central Nervous System

CSEA Center for the Study of Emotion and Attention

CSV Comma Separated Value

D3.js Data Driven Documents

DB Database

DIY Do It Yourself

EDA Electrodermal Activity

EEG Electroencephalography

ECoG Electrocorticography

ECG Electrocardiography

EMG Electromyography

FP1 Left Frontal Polar

xix FP2 Right Frontal Polar

FPS First Person Shooter

FFT Fast Fourier Transform fMRI Functional Magnetic Resonance Imaging fNIRS Functional Near-infrared Spectroscopy

GB GigaByte

GUI Graphical User Interface

GSR Galvanic Skin Response

HCI Human Computer Interfaces

IBEB Instituto de Biof´ısica e Engenharia Biomedica´

IADS International Affective Picture System

IAPMEI Instituto de Apoio as` Pequenas e Medias´ Empresas e a` Inovac¸ao˜

IQ Intelligence Quotient

InfoVis Information Visualization

ICE Intra-cortical Electrodes

MCU Micro-Controller Unit

MEG Magnetoencephalography

MIT Massachusetts Institute of Technology

PPG Photopletismography

SUS System Usability Scale

SVG Scalable Vector Graphics

UBA User Behavior Analytics

UI User Interface

VIS Visualization qEEG Quantitative Electroencephalogram

xx 1 Introduction

Contents

1.1 Objectives...... 5

1.2 Contributions...... 5

1.3 Document Structure...... 7

1 2 Despite the advent of Computer Science, historically, Humanity has consistently relied on a more organic form of processing power: Our brain! This computer’s intricate psycho-physiological inner workings seem ever elusive. Contemporaneously, the quest to study and better understand our mind and its processes is the purpose [1] of the interdisciplinary scientiﬁc ﬁeld titled Cognitive Science (Fig. 1.1).

Figure 1.1: Cognitive Science’s comprising ﬁelds of study. Each line joining two disciplines represents interdisciplinary synergy [1].

In this project, we are predominantly interested in two of its sub-disciplines: Computer Science and Neuroscience - the scientiﬁc study of the nervous system. Speciﬁcally, we are concerned with:

1. The relation between measurable user Physiological data, typically from the Central Nervous System (CNS), and Emotions. Neuroscientists have documented and exempliﬁed such rela- tions typically via the usage of an EEG and its spectral analysis - Quantitative Electroencephalo- gram (qEEG).

2. How to interactively, intuitively and graphically present an InfoVis of the subsequent, derived Psycho-Physiological data.

An emotion is a mental state and an affective reaction towards an event based on subjective experience [21]. Emotions are a quintessential part of the Human nature. Crucial to communication between humans, they also play a fundamental role in rational and intelligent behavior [22]. Their omnipresent nature influences and determines many aspects of our daily lives. In many situations, emotional intelligence is more important than Intelligence Quotient (IQ) for successful interaction [23]. There is also significant evidence that rational learning in humans is dependent on emotions [24]. Thus, the study of emotions and their recognition is well justified as evident by psychology research dating back to the 19th century [25]. We live in an age where collecting personal user data is ever increasingly valued. The Web is evolving toward an era where virtually represented communities will define future products and services [26].

3 Additionally, awareness of and reflection on emotional states has proven particularly important in collaborative contexts as “group self-awareness, of emotional states, strengthens and weakens, modes of interaction, and task processes is a critical part of group emotional intelligence that facilitates group efficacy” [27]. Furthermore, awareness of emotional states inside a group has been correlated with improved communication, decision support and mitigation of group conflicts, while offering an additional level of rich contextual information and also supporting collaborative learning. At the same time, lower values of emotional intelligence and negative emotion contagion inside groups have been linked to declining intra-group communication and collaboration, suggesting the need for emotion awareness solutions [15]. Finally, the study of people’s affective and emotional reaction to an advertisement is booming and increasingly trending towards simple, user-friendly software applications from complex, expensive laboratory user studies [28]. In sum, user data is relevant not exclusively for commercial purposes: it can improve corporate communication and collaboration [27]; personalize Human Computer Interfaces (HCI) interactions; and improve government intelligence applications [23] - able to monitor increases in hostile communications or model cyber-issue diffusion. As seen in these examples, the case for acquisition and analysis of user Psycho-Physiological data in an attempt to determine one’s mental and emotional information spans several areas. This is un- derstandable due to the contemporary high value of user information for Big Data User Behavior Ana- lytics (UBA)[29]. The study of these phenomena can be achieved through different approaches, most notably:

Natural Language Processing The computational analysis of texts and linguistics [3,7–12, 30].

Facial Recognition Systems Video analysis of user’s facial expressions [13, 15, 16, 31, 32]

Physiological Computing Usage of physiological data from humans as input to a computer system [14–18].

Some solutions draw from more than one of these approaches, integrating multiple data sources in a process known as Data Fusion. This multi-modal, multi-sensor approach is usually combined with machine learning Artificial Intelligence (AI) techniques, intended to increase the accuracy of the measurements by training a program to better identify an emotion according to its past experiences. Unfortunately, the consequences of these efficacy increases are: the scatter of different data in distinct devices; the overall tremendous amount of data to take into account; and the difficulty to cognitively interpret the data, typically abstract by itself, specifically in pattern recognition cases. Additionally, noisy data filtering also needs to be taken into account.

4 In order to overcome this heavy cognitive load, InfoVis practices can be implemented. InfoVis is the ﬁeld of visually and interactively representing otherwise abstract data in order to amplify cognition [33]. It enables compact graphical presentations for manipulating large numbers of items, possibly ﬁltered from larger datasets. Additionally, it allows users to interactively discover, decide and explain different and previously concealed patterns such as: trends, clusters, gaps, outliers and more.

1.1 Objectives

The goal of this work is the development of an interactive visualization so that users better understand one’s mental state and emotions via Psycho-Physiological input data. Throughout this project, cooperation and mentorship from IBEB faculty members, actively participating as Neuroscience experts, is paramount to the understanding of Physiological Computing concepts, necessities and applications. In other words, IBEB contributions were indispensable in our journey to visually present the uncensored and inherently complex insight of our mental inner workings. Thus, a InfoVis assisted analysis, search and comparison of current and previous results could potentially identify new trends, outliers and features. In order to achieve our goal we must:

1. Research and choose a methodology to acquire a Physiological data collection for further analysis.

2. Research and develop algorithms to derive Psycho-Physiological information from the original raw data

3. Develop an interactive visual interface to examine such data through the display of:

(a) Real-Time Information

(b) Retrospective Information

4. Validate our work through both user and expert feedback and its statistical analysis.

1.2 Contributions

While the events described in this document took place, IBEB colleagues showed interest in utiliz- ing our project. Speciﬁcally, our hardware - BrainBIT - and software -NeuroExplore - have been used in hands-on demos and pitches. This was intended to exemplify an early version of a product of an hypothetical start-up company project - entitled NeuroTech - which won a KickUp Sports Accelerator Program1. This event took place at Estadio´ da Luz ofﬁces and stages2.

1http://kickupsports.eu/accelerator/ 2http://portugalstartups.com/2017/02/benﬁca-partners-kickup-launch-sports-accelerator/

5 1.2.1 KickUP Sports Accelerator

For an equity stake of 8%, KICKUP Sports Accelerator is an international acceleration program that aims to scout, accelerate and invest in early stage sports business startups, focused on sports performance, lifestyle activities, entertainment and others. This information and further details are accessible online.3. In practice, the not-so hypothetical business case for a Mindfulness Coaching, professional sports, BCI assisted company was put to the test in a twelve week long incubation, mentoring event - starting 17th of April, 2017. This took form in several distinct processes, such as;

Business Model Debating Several workshops and conferences presenting success cases and dis- cussing topics such as brand recognition took place. From this resulted a BMC detailing Costumer Segments and Relationships, key Parters, Activities and Resources as well Value Propositions, Cost Structure and Revenue Streams

Client Meetings On May third, we met with the responsible of an e-sports club association 4. We prepared and successfully executed a live demo for our guest. Upon seeing its functionality, and as a performance training device, the client showed interest in partnering with the project.

Stakeholders Networking Finally, the event allowed us to collect valuable insight by meeting people with experience actually manufacturing a Physiological Computing device as well as ergonomics and product design experts.

In the end the whole event was very productive. Much to our delight, the NeuroExplore live demo was presented ﬂawlessly and led to the scheduling of a gamer tournament recoding session with one of our interested clients - using our system solution.

1.2.2 GrowUP Gaming recording sessions

On June 18th, a meeting followed by live recordings took place in a video-game tournament lounge. We interviewed participants in an attempt to probe product adoption, as seen in AppendixB, section B.2. Then, using NeuroExplore, we captured the corresponding Psycho-Physiological data. Our test subject was a 23 year-old gamer participating in a fantasy card-game tournament. We helped him correctly place his BrainBIT and, keeping it on, progressed through three matches and their three respective recordings. In an unfortunate feat, our tester lost all matches. Regardless, for future referencing, we saved its raw biosignal data and shared it with fellow NeuroTech project colleagues and fellow researchers alike5.

3http://kickupsports.eu/ 4http://growupesports.com/about/ 5https://drive.google.com/drive/folders/0B3AeqOjiwGU3QmhhRm9Ra1g3R2c

6 1.2.3 IAPMEI Scholarship

At the end of June, we were contacted to be part of another IBEB Physiological Computing project: EmotAI. The project applied to funding via a Competitiveness and Inovation agency - IAPMEI. This program is entitled StartUp Voucher scholarship; more details can be found on its web-site6. Due to unrelated events, one of the two scholarship applicants could not accept the challenge. As the project was in its infancy, it was proposed we ﬁlled in the vacant position. This entailed documentation being gathered and sent. However, upon receiving mixed-messages, it was conﬁrmed to us that, for regulatory reasons, the scholarship recipient could not be changed at this phase. As such, for now, our participation in this project is of a voluntary nature. In this fashion, we have contributed to the project via the the development of NeuroExplore as well as by recording and submitting pitch videos demonstrating the technology7.

1.3 Document Structure

This document abides by the ensuing structural arrangement:

Chapter2 presents an expansive number of research ﬁelds, within this projects’ context, whose achieve- ments are of fundamental interest for our project’s existence.

Chapter3 extensively illustrates the distinct industry and academy Psycho-Physiological VIS examples. Every example encompasses one or more categories deﬁned in the Background (Chapter2).

Chapter4 meticulously reports all the project developments. Specifically, we present our wearable BCI prototype - BrainBIT; we detail our project’s architecture; finally, we account each phase of the iterative development process - from data acquisition to final version.

Chapter5 entails the project’s evaluation methodology followed by its results’ scrutiny. Explicitly, these assessments consisted of usability and functionality user tests and case studies.

Chapter6 concludes our chronicle by retrospectively summing up this projects results. Additionally, future work, usages and possibilities for NeuroExplore are examined - including a simultaneous start-up company project of which we are members.

6https://www.iapmei.pt/PRODUTOS-E-SERVICOS/Empreendedorismo-Inovacao/Empreendedorismo/Startup-Voucher.aspx 7https://photos.app.goo.gl/fEt2NH2U2U4SIVls1

7 8 2 Background

Contents

2.1 Sentiment Analysis...... 11

2.2 Affective Computing...... 11

2.3 Physiological Computing...... 12

2.4 Brain Computer Interfaces (BCI)...... 14

9 10 In 1872 [34], Charles Darwin wrote: “facial expressions of emotions are universal, not learned differ- ently in each culture”. There have been arguments both in favor and against this view ever since [35,36]. In recent years, different technologies have been developed whose purpose is the capture and interpretation of the various emotions users might experience. This is accomplished using facial expression recognition - a clear nod towards Darwin - but also written text, speech and intonation, electro-dermal response, brain-wave signals and other physiological signals [15]. In this section we introduce related IT ﬁelds of study - within this context - whose concept’s compre- hension is of underlying interest for our project.

2.1 Sentiment Analysis

”An important part of our information-gathering behavior has always been to find out what other people think”[37]. Sentiment Analysis, also known as opinion mining, is the field of study that analyzes people’s opin- ions, sentiments, evaluations, attitudes, and emotions. It has evolved from basic polarity classification based on thumbs up/down to natural language processing, text analysis and computational linguistics in which keyword spotting is used to retireve term frequencies and their presence [38, 39] Meanwhile, a broader approach entitled Multimodal Sentiment Analysis also takes into account audio content, typically converting it to text first, as well as audiovisual contents. Numerous new technologies have potential usages in opinion mining, such as: facial expression, body movement, or a video blogger’s choice of music or color filters [40].

2.2 Affective Computing

Traditionally, “affect” was seldom linked to lifeless machines and was normally studied by psycholo- gists. Currently, Affective Computing is a branch of Computer Science concerned with the theory and construction of machines which can recognize, interpret and process human emotional states - the focus of our work - as well as simulate them [41]. Additionally, it is an important topic for harmonious human- computer interaction, by increasing the quality of human-computer communication and improving the intelligence of the computer [42]. It was only in recent years that affect features were captured and processed by computers. An “affect model” is built based on the various sensors - typically cameras - which acquire information and build a personalized computing system with the capability of perception and interpretation of human feelings as well as giving us intelligent, sensitive and friendly responses [42]. Common examples for this include applications that react to emotions extracted from facial expressions or speech interactions as well as

11 systems that change the style of a responsive artistic visualization based on the machine’s perceived emotional state of the viewer. This contrasts with Sentiment Analysis’ approach of trying to ﬁnd speciﬁc emotional hints or keywords. Affective computing focuses on a broader set of emotions or the detection and estimation of continuous emotion primitives [38].

2.2.1 Affect Visualization

The majority of contemporary works use Ekman’s discrete model to represent emotion [43]. The model uses six basic emotion labels that are universal among different cultures. The basic emotions are anger, disgust, fear, happiness, sadness and surprise (see Figure 2.1). There is an ongoing controversy in the ﬁeld of psychology about how to adequately discrete labels categorize emotion [44].

Figure 2.1: Paul Ekman’s 6 categorical emotions: Happy, Sad, Fear, Anger, Suprised, Disgust.[2] Figure 2.2: Valence/Arousal Emotional Space [3].

As an alternative (see Figure 2.2, dimensional models of emotions have also been proposed. In these, an emotion is expressed by a normalized number scale for each dimension, typically two: Valence (pleasantness of emotion) and Arousal (intensity of the emotion). Using dimensional models to represent emotions is less “black and white” than using coarse emotion labels as it allows blends of different emotional states, the natural inclusion of the neutral emotional state (zeroes in all dimensions) and any value between neutral and full emotion [3].

2.3 Physiological Computing

”The human body is chemical, electrical, mechanical, thermal and magnetic in nature” [45]. Physio- logical sensors can be used to detect body information such as heart rate and brain-wave signals that

12 reﬂect changes in a user’s mood and/or environment.

EEG recording devices are much less expensive and portable than other brain activity recording techniques such as Magnetoencephalography (MEG),Functional Magnetic Resonance Imaging (fMRI) and Functional Near-infrared Spectroscopy (fNIRS), they are still much too expensive for a daily use from a customer point of view, especially for disabled people. This is why plenty of commercial EEG devices are now available such as Neurosky, Mindﬂex, Emotiv Epoc, etc [6]. According to [6], the best low-cost EEG device in terms of usability is the Emotiv Epoc headset.

Physiological computing represents a mode of human–computer interaction where the computer monitors, analyzes and responds to the user’s psycho-physiological activity in real-time [46]. The measurement of physiological signals can take form in a multitude of distinct techniques. These techniques usually fall under one of two categories: 1) nervous system activity analysis; 2) cardiovascular system activity analysis [47]. The former encompasses methods such as: Galvanic Skin Response (GSR), EEG, Electrocorticography (ECoG), MEG, Intra-cortical Electrodes (ICE), fMRI, fNIRS; the latter, typically Electrocardiography (ECG) or PPG.[48]. Additionally, these can be grouped in two probing categories: invasive and non-invasive. Invasive approaches such as ECoG and ICE need electrodes to be surgically implanted in the cerebral cortex. In contrast, non-invasive techniques require no surgery. [49].

Most notably, EEG detects voltages, in volts, resulting from ionic density ﬂuctuation within the neu- rons of the brain. These measurements occur in different scalp locations, where an electrode is placed. (Fig. 2.3). Despite poor spatial resolution, EEG’s relative affordability and small, non-invasive sensors paired with very good temporal resolution crest this technique as one of the most widely used in contemporary BCI[50]. Indeed, EEG signals and their spectrum have been previously documented in what is known as qEEG: the ﬁeld concerned with the numerical analysis of this data and its associated behavioral correlations [51, 52].

Figure 2.3: International 10–20 system. An internationally recognized method to describe and apply the location of scalp electrodes in the context of an EEG test or experiment.A = Ear lobe, C = central, F = frontal, FP = frontal-polar, O = occipital, P = parietal, T = temporal [4]

13 2.4 Brain Computer Interfaces (BCI)

Electronics’ ever improving processing power and decreasing hardware sensor sizes [53] allow for contemporary wearable computers to achieve what was previously only possible with expensive scientific equipment in a medical laboratory. Wearable body monitoring is a story about data and data analysis, as much as it is a story about HCI form factors and size reduction [54]. A BCI is a complex application of signal processing and neuroscience. The system uses mental activity, involuntarily produced by the user, to control a computer or an embedded system typically via EEG signals which allow communication or interaction with the surrounding environment [55]. BCI technologies can be a used to decipher the user’s state of mind the state of mind of the user (arousal, emotional valence, attention, meditation, among others) [50]. Alternatively known as Brain Machine Interface (BMI), these systems also enable humans to interact with their surroundings via control signals generated from EEG activity, without the intervention of peripheral nerves and muscles [55]. These systems permit the sole encephalic activity to control external devices such as computers, speech synthesizers, assistive appliances, and neural prostheses. This can be particularly relevant to severely disabled people who are totally paralyzed or ‘locked in’ by neurological/neuromuscular disorders, such as amyotrophic lateral sclerosis, brain stem stroke, or spinal cord injury. Such systems would improve their quality of life and simultaneously reduce the cost of intensive care [55]. Typically, a BCI is an artificial intelligence based system that can recognize a certain set of patterns in brain EEG signals via a number of consecutive phases, namely: (1) signal acquisition for brain signals capturing; (2) preprocessing or signal enhancement for preparing the signals in a suitable form for further processing, typically resulting in a power spectrum; (3) feature extraction for identifying discriminative information in the brain signals that have been recorded, generally an analysis of the different frequency bands in the power spectrum known as Brain Waves (See Table 2.1); (4) classification for organizing the signals based on the extracted feature vectors; and finally (5) the control interface phases for translating the classified signals into meaningful commands for any connected device, such as a wheelchair or a computer [56].

Table 2.1: Brain-wave frequency bands

Brain Wave Frequency (Hz) Delta (δ) 1-4 Theta (θ) 4-8 Low Beta (β) 12-15 Mid Beta (β) 15-20 High Beta (β) 20-30 a Gamma (γ) 30-100

Concerning brain-waves’ activity - which are of particular interest to our project - neuroscientists have

14 found correlations between these and mental-state metrics such as: Emotional Valence, Attention and Meditation. For example, Trimble et al [?] states that ”there is a clear asymmetry in activity between the left and right hemispheres of the brain” and that “alpha frequencies in the left prefrontal cortex (PFC) indicate positive emotion and in the right PFC indicates negative emotion”. Other studies noticed similar correlations [28, 57–59]. In recent times, the commercial computer industry has come up with a tantalizing solution to these problems: wireless EEG systems (such as Emotiv EPOC, Imec’s wireless EEG headset, NeuroFocus MyndTM, Neurokeeper’s headset, NeuroSky Mindwave, Plux’s Bitalino). These BCI examples require no speciﬁc medical training or laboratory equipment, as they are intended for use by the general public. Most of these solutions are “gaming EEG systems”, analyzing EEG activity to control the movement of characters or objects in games via headsets that comprise a small array of electrode sensors. Other advantages are listed below:

1. Wirelessly connected to software that runs on a laptop (so no need for an expensive laboratory);

2. Requires little adjustment of electrodes (so no need for long electrode placement procedures)

3. Either use a small cotton pad that are soaked in saline solution to connect each electrode to the scalp (so no need for messy gel, and hence no need for head washing) or nothing at all.

. Finally, Chapter3 presents multiple BCI studies and industry solutions, often comprising several areas of research mentioned in this chapter. Due to the abstract nature of their raw data, these often relied on an InfoVis for Psycho-Physiological datum presentation. Incrementing upon this background overview, it is our intention to present concrete and practical, state of the art examples.

15 16 3 Related Work

Contents

3.1 Emotion’s Visual Representation...... 19

3.2 Affective Computing...... 20

3.3 Discussion...... 35

17 18 As previously stated in chapter2, there is a broad interest in user Psycho-Physiological data. Through different acquisition techniques alike, the resulting data by itself is uninterpretable. Thus, it is always necessary to derive these values into more intuitive visual representations. As such, studying the best way to visualize and interact with our data can be help improve existing offers. We have documented our ﬁndings and grouped them in the following fashion: a) Emotion’s Visual Representation; and b) Affective Computing, with the following sub-sections: (i) Text Analytics; (ii) Video Content Analysis; (iii) Physiological Computing

3.1 Emotion’s Visual Representation

Figure 3.1: Thayer’s Arousal/Valence model [5] Figure 3.2: Lee et al. newly proposed visualization [5]

Lee et al. [5] presented a color study for emotion visualization. Explicitly, the research attempted to measure emotions using different colors, thus proposing a correlation between color and emotion and ultimately allowing to “identify user’s emotion with different colors”. The researchers used Thayer’s arousal-valence polar coordinates model (Fig. 3.1) for emotional cat- egory identification. The RGB accumulative color system in which red, green and blue light are added together was used. Additionally, each of its components was subject to machine learning techniques comparing results with other samples in order to improve their overall accuracy. Working with 240 samples, participants were asked to associate a color with each of the following for coordinates: (1,0), (1,1), (-1,0), (-1, -1), corresponding to Pleased, Aroused, Miserable and Calm, respectively. A consequent color gradient between the polar coordinates of each of these four points, ranging from the color of one point to the other, can be seen in the final visualization (Fig. 3.4). In the end, the authors validated their results using volunteers and achieved good intervals of confi-

19 dence. In spite of this, subjective and cultural differences were still identified as being influential in certain emotion-color correlations, specifically in the Miserable-Black instance, where most disagreements were identified. Sanchez et al. [6] propose an inventive new way to visually represent emotions. Essentially, the [-1,1] 3D vector-space representation of emotion is used, matching each axis to Affinity, Pleasure and Arousal. It was identified as a common, useful way to measure emotion. However, its mathematical visual representation was considered rational and emotionally unappealing. For that reason, a visual correspondence was proposed between Pleasure and Shape, Arousal and Brightness and Affinity and Size of a geometric figure (Figures. 3.3 and 3.4). In this geometric figure, the Japanese representation of displeasure – X – and pleasure – O – were used as guidelines for the shape’s appearance. On the other hand, brightness increases according to arousal and size increases correspond to affinity. In the end, a user study was successfully conducted to validate results. Participants were presented seven geometric figures and had to match them with each of Ekman’s six emotions plus one: Neutral. 83,93% matching accuracy was identified for non-experts, and 90,46% for experts.

Figure 3.3: Sanchez et al. proposal for emotion’s representation [6] Figure 3.4: Sanchez et al. newly proposed visualization [6]

3.2 Affective Computing

In this section we present state of the art examples of Affective Computing InfoVis.

3.2.1 Text Analytics

Torkildson et al. [7] used machine learningAI techniques for sentiment analysis, emotion classification and corresponding visualization purposes. Specifically, emotional classifiers are defined and a

20 collaborative visualization is proposed after analyzing twitter posts data preceding, during and after the 2010 Gulf Oil Spill Crisis. Categorical Attributes of Emotion were deﬁned based on Ekman’s six basic emotions: joy, anger, fear, sadness, surprise and disgust. Sentiment attributes: negative, neutral and positive, were designated and intended to be mutually exclusive. Afterward, the purpose of the pretended visualization was set: “support the analysis of the emotional impact of events”. Finally, the authors decided to graphically present the information using a stacked area chart (Fig. 3.5). This idiom allows “easy comparison of values, facilitated by the colored bands. The top 25% of values, the time instances with the highest emotion frequency, have the highest color saturation. The coloring makes these peaks easily distinguishable.” Ultimately, this paper provides an interactive VIS for emotional analysis which allows for tasks such as: “What emotional response did the public manifest as a consequence of the POTUS speech?” - to be accomplished.

Figure 3.5: Staked Area Chart for each of Ekman emotions’ intensity over time [7]

Scharl et al. [8] achieved emotion detection through machine learning techniques using social media platforms for data collection. Furthermore, several consequent visualizations are proposed. The object of study is a popular television work of fiction. Using SenticNet 3 [60], sentiment annotations and affective knowledge is defined according to the following attributes: pleasantness, aptitude, attention, and sensitivity. Additionally, these attributes can reflect sentiment information by means of color coding, ranging from red (negative) to gray (neutral) and green (positive). Sentiment is shown with variable saturation, depending on the degree of polarity. The resulting VIS resorted to several idioms such as: Line charts, for effective comparison of average sentiment of each character as well as the level of disagreement ( derived data corresponding to

21 the standard deviation of sentiment which reveals polarization); Pie charts for comparison of overall positive/neutral/negative results; Radar charts, “to depict affective knowledge along emotional categories”, by proﬁling across several emotional categories, as seen on Figure 3.6.

Figure 3.6: VIS screenshot displaying the following idioms: Line chart, Pie chart, Radar chart [8]

Figure 3.7: Screenshots of four distinct text-extracted emotion VIS: a) avatars; b) emoticons; c) Hooloovoo; d) Synemania. [9]

Krcadinac et al. [9] intended to “introduce a novel textual emotion visualization” named Synemania. This VIS relies on theSynesketch software library to parse text and extract emotional tags from it. Its purpose is to facilitate: emotion communication; emotion evocation; and overall user enjoyment. In order to achieve this, Ekman’s emotion classiﬁcation is used. This is achieved through an exhaustive user study. In it, chat conversations’ visualizations are eval- uated using four different ways to visualize emotion: avatars; emoticons; and two new VIS: Hooloovoo and Synemania. The former focuses on color as a representation of emotion, and displays a matrix of squares colored accordingly. The latter goes further and uses both color and abstract animation

22 to evoke emotions. See Figure 3.7 for a visual representation of these four distinct alternatives. In the end, results proved that despite the participants inability to successfully recognize the “exact emotional types”, the generative art content of Hooloovoo and Synemania was “enjoyable and effective in communicating emotions”. In fact, “Synemania proved to be better (on a statistically significant level, p < 0.001) than other visualizations in evoking emotions.” Interestingly, if an EEG was added to this VIS as a physiological data input, this could lead to a more accurate emotional classification [61]. Kamvar and Harris [10] created an emotional search engine together with a web-based visualization. This was achieved extracting sentences from social media that include the words “I feel” or “I am feeling”, as well as the corresponding author’s sex, age, location and time of occurrence. Combining these metrics allows for several distinct visualizations such as: bar-charts with age breakdown of feelings from people in the last few hours; world-maps characterized by a geographic breakdown of feelings from people in the last few hours; line-charts relating sentiments over time such as stress and relaxation over the week or love and loneliness in the week of Valentine’s Day (Fig. 3.8); stacked area charts relating a specific feeling’s frequency over the aging of a human. These results were based on an immense amount of different emotions or sentiments possible thanks to the applied text extraction technique. Combining these techniques with physiological input data could potentially yield even more accurate results. The overall result exposes the diverse capabilities of interfaces that allow for item-level exploration of sentiment data.

Figure 3.8: On the left: Sadness over age Area chart; On the right: keyword’s usage during valentine’s day Line Chart [10]

Quan and Ren [11] presented an emotional VIS intended for bloggers’ usage. In it, an intuitive display of emotions in blogs is presented which can be used to capture emotional change rapidly. This is achieved through machine learningAI techniques applied to emotion recognition at different textual levels, thus recognizing multi-label emotions. The emotion visualization interface uses eight different emoticons to represent the following emotions: expectation, joy, love, surprise, anxiety, sorrow, angry and hate. Multiple emotions are recognized in each word, each sentence and overall document, as depicted in Figure 3.9. The left section is the input blog. The central area shows the document structure and recognized emotions for this blog. The

23 rightmost part shows a table containing sentence emotions present in each sentence. The develop- ers approach of using emoticons instead of colors for emotional representation was fully intentional, designed to avoid ambiguity.

Figure 3.9: InfoVis Interface: input blog (left), document structure and recognized emotions (middle) and the sentence’s emotion table (right) [11]

Figure 3.10: Chicago geo-tagged emotional Twitter mes- Figure 3.11: USA aggregated emotion tweets VIS[3] sages VIS[3]

Guthier et al. [3] proposed a visualization of the affective states of citizens. These emotions are detected from geo-tagged posts on Twitter using machine learning techniques. The detected emotions are aggregated over space and time and, ﬁnally, visualized using a city map idiom. Emotions are represented as four-dimensional PADU vectors (pleasantness, arousal, dominance and unpredictability). Unlike the categorical approach, this dimensional representation is attractive because it provides an algebra to describe and relate an inﬁnite number of emotional states and intensities. However, given its continuous nature, the main drawback of this approach is that it does not offer an intuitive understanding of affective information, since people are used to reporting emotions by means of

24 words. The authors decided to visualize the 4D emotion vectors as four concentric disks. The radius of these is proportional to the amount of active tweets. The order of the disks was chosen based on their importance, and a color model was created to represent each dimension: Pleasentness is gray, Arousal is green, Dominance is red and Unpredictability is blue. Increasing brightness of these colors was then chosen as a representation of intensity. Each emotional tweet is taken into account for 30 minutes, averaging its value with its geographical neighbors. As we can see in Figures 3.10 and 3.11, a geographical map then presented the ﬁnal result.

Figure 3.12: Sentiment before Sep 2013 Autralian Elec-Figure 3.13: Sentiment after Sep 2013 Autralian Election tion InfoVis[12] InfoVis[12]

Wang et al. [12] set out to visually represent emotions over time. The authors created a new visualization named SentiCompass. Intended to compare the sentiments of time-varying Twitter data, its purpose is the improvement of affective analysis of sentiments. SentiCompass uses a two-dimensional polar space with the horizontal axis being valence (the level of pleasantness) and the vertical axis being arousal (the level of activation). These coordinates are then converted into emotional categories. Polar coordinates representation is used in order to better discern the cyclic behavior of sentiment data. This is combined with a perspective representation of Time Tunnel, thus incorporating the temporal dimension and, again, facilitating the comparison of cyclic behaviors. These rings vary in size according to their time interval correspondence: the smaller they are, the more remote the time. Finally, color coding is used to represent the two dimensions of sentiment: Valence (green to red) and Arousal (blue to yellow). The ﬁnal result can be observed in Figures 3.12 and 3.13. This new idiom proved effective in the two case studies conducted by the researchers. Its intuitiveness, however, can be questioned due to the co-

25 centric circumference time interval representation of emotions which results in difﬁcult value comparison of distinct emotions.

3.2.2 Video Content Analysis

Hupont et al [13] created an emotional visualization – EMOTRACKER - employing not only facial analysis for emotion recognition purposes as well as eye-tracking for gaze analysis. After noting several industries’ increasing desire for objective measurements of engagement with content, sparked by brands increasingly striving to build emotional connections with consumers, the authors note there is a lack of tools for achieving these aims. “In fact, the important question of how to efficiently visualize the extracted effective information to make it useful for content designers has been scarcely studied”. The resulting VIS was composed of two modes: “emotional heat map”, with selectable emotional layers, and “emotional saccade map”, with a dynamic representation that shows the path formed by the user fixation points (points the user had been looking at for a minimum configurable time, in millisec- onds). In both modes, the users could also see their current emotional state via an emoticon as well as Eckman’s six emotions, plus neutral, as seen on Figures 3.14 and 3.15. Finally, it would be of scientific interest to analyze the impact in this VIS’s accuracy if an EEG was added as input as done by Soleymani [61].

Figure 3.14: Emotional Heat Map InfoVis[13] Figure 3.15: Emotional Saccade MapInfoVis[13]

Emotient Inc. was a startup focused on video emotional analysis and its consequent visualization. Recently acquired by Apple Inc.1, the company’s main product was an API called Emotient Analytics 2. This software provided facial expression detection and frame-by-frame measurement of seven key emotions, as well as intention, engagement and positive or negative consumer sentiment. All of this

1 http://blogs.wsj.com/digits/2015/12/11/silicon-valley-kingpins-commit-1-billion-to-create-artiﬁcial-intelligence-without-proﬁt- motive/ 2https://web.archive.org/web/20151219062637/http://www.emotient.com/products

26 was then incorporated in a visualization (Fig. 3.16). In it we can see line charts for Emotional Engage- ment, Attention and Sentiment as well as the average of each of these metrics. Furthermore, stacked area chart displayed each emotion over time as well as pie charts for the video’s average, emotional engagement and participants’ gender. The insight gained from quantifying emotions allowed companies to pinpoint and ﬁx issues as well as improve their marketing performance. Additionally, all of this was accessible independently of platform, via a web browser.

Figure 3.16: Emotient InfoVis displaying Emotional Engagement current value with an angular meter idiom and evolution with a line chart (left) as well as Ekman’s emotions in a retrospective stacked area chart and its overall values in pie charts (right)1.

Figure 3.17: EmoVu InfoVis usage of a area-chart which represent each of Ekman’s emotions with a distinctive color3

Produced by Eyeris Inc, EmoVu3 is a self-serve cloud-based web platform that “allows video content creators to accurately measure their content’s emotional engagement and effectiveness of their target audience”. The creatures argued that establishing an emotional connection with the audience creates a

3http://emovu.com/e/video-analytics/emovu-cloudsync/

27 more affective tie the creators assert that emotional campaigns are likely to generate larger proﬁt gains than rational ones. Emotional acquisition techniques are employed, such as: face-tracking; head pose detection; facial recognition; multi-face detection; gender and age group recognition; and eye tracking and openness. These were then subject to machine learning techniques to reach acceptable accuracies. This emotional analysis is then encoded using seven emotions (Ekman’s six plus neutral), ﬁve Engagement metrics, three Mood indicators, four Age groups and gender. Finally, an interactive visualization (Fig. 3.17) consisting of a layered area chart representing the different emotions detected over a commercial’s duration is presented. Adjusting it using metrics such as Age, Gender and Location is also possible. A free demo is offered upon account creation.

Figure 3.18: Affdex InfoVis comparing a user’s line chart for Expressiveness with other users according to age groups4.

With millions of faces analyzed to date and one-third of the Global Fortune 100’s companies using their technology, MIT’s Media Lab’s offspring Affectiva Inc has developed a video analysis emotion detection and visualization API – Affdex4. The argument that Emotions are the number one inﬂuencer of attention perception, memory, human behavior and decision making, the API is capable of detecting Ekman’s six emotions plus neutral, 15 nuanced facial expressions and even heart-rate by evaluating color changes in a person’s face, which pulse each time the heart beats. The resulting visualization consists of a retrospective line chart, generated after analyzing via web- cam the user’s reaction to an ad. This line chart compares different age groups reactions as well as the

4 http://www.affectiva.com/solutions/affdex/

28 user’s, along the ad’s duration. The measured reactions are: Surprise; Smile; Concentration; Dislike; Valence; Attention; and Expressiveness (Fig. 3.18). This demo is publicly available to test, online.

Figure 3.19: Kairos InfoVis presenting a subjects’ facial recording side by side with multiple retrospective area charts, one for each of Ekman’s emotions5.

Another thriving company in the emotional research scene is Kairos 5. Offering an Emotion Analysis API powered by IBM’s supercomputer Watson. Using machine learningAI techniques, their product examines people’s reactions to ads for neuromarketing purposes. These responses are broken down and interpreted according to Ekman’s six basic emotions as well as the overall sentiment, which can range in a scale from Very Negative (-100) to Very Positive (+100). Users can submit videos containing people’s faces by visiting Kairos’ website. The video is then processed and the results, presented to users. The visualization consists of an interactive interface alongside the submitted video. An area chart is presented for each emotion, displaying its intensity over time (Fig. 3.19). A free demo is available online.

Figure 3.20: Nviso’s InfoVis usage of an area chart to represent each of Ekman’s emotions detected over time as a result of facial expressions’ detection (left)6.

Nviso 6 is a swiss company based at the Swiss Federal Institute of Technology in Lausanne which specializes in emotion video analytics. Based on Ekman’s Facial Action Coding System, an artiﬁcial

5 https://www.kairos.com/ 6http://www.nviso.ch/

29 intelligence algorithm captures and measures the response to the main facial muscles involved in the expression of emotion in real time. Proprietary deep learning 3D Facial Imaging systems decode facial movements and underlying expressed emotions. Speciﬁcally, this is achieved scrutinizing facial expressions and eye movements by tracking 43 facial muscles and hundreds of muscle movements using ordinary webcams. Intended for real-time consumer behavior analysis, the results are presented through an on-line VIS tool. In it we can identify the intensity of each fundamental emotion at any given moment of the submitted video by examining a stacked area chart (Fig. 3.29).

3.2.3 Physiological Computing

Figure 3.21: EEG’s Power Spectrum Figure 3.22: Volumetric bull’s eye idiom [14]. Figure 3.23: Gavrilescu [14] proposed EEG InfoVis

Gavrilescu and Ungureanu [14] investigated contemporary methods to display EEG data and subsequently proposed a new VIS. After mentioning the several useful usages of EEG, such as “the assessment of a user’s emotional state”, the authors note that the nature of EEG signals, lacking any intrinsic visual data, causes multiple challenges regarding their graphical representation. Particularly “for data spanning over frequency bands and extended durations”. The current, most common ways, for EEG visual representation are then dissected:

Power Spectrum graphs are identiﬁed as being time-consuming and tedious to compare (Fig 3.21). This issue is particularly severe when representing raw data for a large number of electrodes and for various brainwaves.

Volumetric bull’s eye plot - a 2D top-down disc-view of the cranium (Fig. 3.22) - is identiﬁed as an effective way of relating desired information for a single sample across all electrodes positions.

30 However this idiom is unable to effectively represent complex, multivariate data concurrently. This is due to the difﬁculty to “represent multi-value data across multiple ranges and time phases in a single image”.

Finally, a new VIS (Fig. 3.23, aiming to “provide an intuitive means of representing the data over multiple sample phases and for all available frequency bands” is presented. Using color spots varying in size and color according to the brainwave frequency and voltage, respectively; and glyphs varying in volume depending on the variation of the data between consecutive time phases (Fig. 1).

Figure 3.25: Emotional heat map; Valence/Arousal bar- Figure 3.24: Valence/Arousal space-visualization map- chart [15]. ping [15].

Cernea et al. [15] identified emotions with the purpose of enhancing the experience of multi-touch interfaces’ users. A VIS is proposed for this context and this approach is verified conducting an EEG user study. The researcher’s goal is to improve the user’s emotional self-awareness; the awareness of other users emotion in collaborative and competitive scenarios; and the evaluation of touch systems and user experience through the visualization of emotions. This was achieved using Russel’s model [62] encoding a variety of emotions in terms of affective valence (pleasant or unpleasant) and arousal (excited or calm) (Fig. 3.24). Using EEG, facial expressions and GSR readings as input, the VIS would then be presented around the user fingertip’s touch-screen outline. It would vary from a sharp to a curvy outline according to displeasure or pleasure and from blue (slow pulsation) to red (intense pulsation) according to low/high arousal. Additionally, for user retrospec- tion purposes, a bar chart representing arousal-valence values over time could be posteriorly analyzed. Finally, a blue-violet-red heat map was used in order to indicate average arousal around a football field for a specific multi-touch video game application (Fig. 3.25). McDuff et al. [16] presented AffectAura: a “multimodal sensor set-up for continuous logging of audio, visual, physiological and contextual data” intended for emotional memory improvement purposes. Their broad approach for emotional input data was subject to filtering and machine learning techniques in order

31 to improve affect recognition. Consequently, a “classification scheme for predicting user affective state and an interface for user reflection” was proposed and successfully tested. The visualization consisted of a retrospective log of emotional activity. The system classified emotions using a categorical version of the three-dimension valence-arousal- engagement model. Valence could be either negative, neutral or positive and arousal and engagement could each be either low or high. Visually, a circle geometry would change color, shape, opacity and size according to valence, arousal, engagement and overall activity respectively (Fig. 3.26). In the end, users could successfully leverage cues from AffectAura to construct stories about their days, even after they had forgotten these particular incidents or their related emotional tones. Most participants (83%) thought that this emotional analysis would be useful for reflecting and self-improvement.

Figure 3.26: Screenshot of AffectAura’s InfoVis[16]

Figure 3.28: Emotional Intensity bar charts and scale un- Figure 3.27: Museum’s top view heat map [17]. der painting [17].

Du et al [17] conducted an emotional analysis study in an art museum. Using a GSR physiological sensor to compute “the individual and average affective responses and provide the “emotion proﬁle” of a painting as well as the “emotional map” of the museum”. Museum visitors were simply asked to visit the museum while wearing the GSR as no other source of

32 emotional feedback was targeted. This was intentional, as the participant’s nervous system responses were preferred in order to avoid culturally and psychologically biased responses. The visualization encoded emotion using a color scale ranging from blue to yellow to red as the emotional intensity increased. The VIS consisted of a 3D virtual museum. The ﬂoor was used as a heat map, changing color according to the overall emotional reaction to nearby paintings. Additionally, bar charts were presented under each painting, displaying each of the participant’s emotional reaction in the y-axis, for retrospective analysis purposes (Figures 3.27 and 3.28).

Figure 3.29: Screenshot examples of iMotions’ InfoVis for distinct emotions7

iMotions 7 is a biometrics research platform that provides software and hardware for many types of bodily cues intended for emotional identiﬁcation. They provide an API which supports more than 50 hardware devices responsible for emotional acquisition techniques such as: eye tracking, facial expressions, EEG, GSR, ECG and EMG. This physiological data is converted to emotional insight using a deluge of metrics such as: positive, negative and neutral valence; 19 action units; Ekman’s six basic emotions plus neutral and two advanced emotions – confusion and frustration. This data is then visualized, in real time, showing an area chart for each emotion, over the duration of a commercial which is displayed alongside the subject’s recorded reaction video. If used, eye tracked areas of interest are also displayed on the ad. Similar area charts are also available to other metrics. Interestingly the technology was used by the US Air Force and was featured in television shows. NeuroSky’s MindWave is a BCI headset catering directly to the consumer market 8. The hardware consists three electrodes: a reference and ground electrodes located on a ear clip and one EEG electrode on the forehead above the eye (FP1 position, see ﬁgure 2.3)9. MindWave’s also supports previous recording data display in the format of an InfoVis (Fig. 3.30). In it we can visualize brain-wave bands’ intensities (counts) in a radar chart (left) as well as a colored bar

7https://imotions.com/ 8 http://neurosky.com/biosensors/eeg-sensor/biosensors/ 9http://neurosky.com/biosensors/eeg-sensor/ultimate-guide-to-eeg/

33 Figure 3.30: Screenshot of Neurosky’s MindWave InfoVis8

chart (right). The raw EEG signal is displayed above the bar chart. Additionally, two circular meters display the normalized (1-100) Attention and Meditation metrics. This angular display of numbers is a bad practice as it complicates user comparison. An alternative, linear representation would ﬁx this.

Finally, signal quality is also displayed so the user can better identify issues such as signal saturation.

Figure 3.31: Screenshot of bitalino’s OpenSignalsInfoVis. In it we can observe a line-chart for each of bitalino’s channel raw data over time [18].

34 Bitalino is a low-cost toolkit to learn and prototype applications using physiological signals10. It supports a wide range of sensors, although not all simultaneously, due to bandwith limitations [18]. Namely: an Accelerometer, PPG, ECG and Electrodermal Activity (EDA)/GSR. Additionally, a User Interface (UI) and VIS software - OpenSignals11 - can be installed for a real-time or retrospective analysis as well as loading of pre-recorded signals (Fig. 3.31). On it we can see line charts displaying each sensor’s signal. OpenSignals main limitations are its lack of derived metrics, the VIS shows RAW data only; its absent of any post processing which can result in saturated signal visualization; its connectivity can be somewhat lacking and frequent crashes have been reported.

3.3 Discussion

In this section, we have extensively investigated several distinct InfoVis examples of emotional, affective and sentimental metrics such as Emotional Valence, Arousal, Meditation and Attention. We started by investigating two scientific literature papers studying how to represent an emotion visually as well as their success rates. Moving on to Affective Computing InfoVis examples, we found: seven emotional analysis InfoVis from extracted text, six from video camera recordings and six from physiological computing, some even using hybrid, data fusion techniques. As far as we could tell, there is only one clear example of an InfoVis dedicated to EEG[14]. However, its lack of emotional representation as well as the ”unjustified usage of 3D” in the form of a Human’s head - a typically prohibitive practice in InfoVis design [63] - proved disappointing. It is noteworthy to point out that academic literature focused mostly on emotions extracted from text and, recently, on fusion approaches. Moreover, with the exception of EMOTRACKER, all video based emotional analysis InfoVis were proprietary in nature. Additionally, combining different emotional input such as physiological data and camera facial recognition also proved a popular approach with supplementary, insightful information as a consequence of new metrics as well as increased complexity and efficacy. Noteworthy accuracy increasing techniques for emotional acquisition goals were also identified.AI’s machine learning approaches were common and advantageous as programs could be trained to better identify emotions according to numerous past examples. After this state of the art review, it became clear that there are several distinct approaches and fields of research analyzing and scrutinizing one’s mental state and emotions. Despite their accompanying UI, a lacking amount of InfoVis specific Physiological Computing studies was, regrettably, registered.

10http://bitalino.com/en/ 11http://bitalino.com/en/software

35 With the exception of Gavrilescu’s [14] work which featured ”unjustified 3D interaction” (see 3.23), we could not find any other EEG InfoVis. Here lies our opportunity to expands the borders of scientific knowledge. Further motivating our work, we propose the development of an intractable and intuitive Psycho-Physiological visual medium. Through which, users will be able to better understand their emotional and mental inner-workings.

36 4 Proposed Solution

Contents

4.1 Biosignal Input Solution...... 39

4.2 Architecture...... 42

4.3 Development Process...... 44

4.4 Discussion...... 57

37 38 This chapter portrays all the work which led into the most recent version of our proposed InfoVis solution1. In it we will document the various stages of this process. We start with presenting and justi- fying our chosen device and method of affective data aquisition: a bitalino BCI physiological computing wearable device. Subsequently we will detail the physiologicalDB acquisition methodology. Finally, we shall present our InfoVis system alongside its development process, detailing its iterative nature as well as its underlying algorithms.

4.1 Biosignal Input Solution

Efficient Neuro-Physiological acquisition is key for any biosignal database upon which we can develop and InfoVis, particularly so in real-time scenarios. This is comprehensible as the database itself is the foundation of any InfoVis. As such, we must discuss the means through which we visually and interactively amplify cognition of this otherwise incomprehensible data. During initial IBEB meetings, we learned about the opportunity of using a BCI as input to a Physiological Computing InfoVis database. We were introduced to Bitalino2 - a DIY BCI biosignals hardware kit - which we ended up using as our affective data acquisition device. The hardware was introduced to this project thanks to IBEB. Furthermore, in January 2017 we participated in a case study meeting for Bitalino at IBEB. In there, we had the opportunity to meet one enthusiastic and helpful supporter: Bitalino’s parent company Chief Information Officer (CIO), Ph.D Hugo Silva 3 - with whom we kept contact. Indeed, we found one bug in Bitalino’s bare-bones python API and were credited for debugging it4. Intended to be a Physiological Computing equivalent of the traditional Arduino platform, the hardware consists of a lowcost, modular wireless biosignal acquisition system, with a credit card-sized form factor that integrates multiple measurement sensors for bioelectrical and biomechanical data. A particularly noteworthy attribute is this BCI’s battery longevity: it runs up to 10 hours using Bluetooth and two sensors with the standard battery. Additionally it can be upgraded to achieve more than 40 hours of battery life in continuous operation [20]. Bitalino supports many distinct sensors, yet its analog ports are limited. For this and other features such as weight in Table 4.1. We equipped ours (Table. 4.2) with an accelerometer, four EEG electrodes and one PPG ear-clip sensor. The rationale behind this multi-sensor approach is straightforward. The accelerometer is used to detect user movement and the resultant signal saturation. Through EEG analysis, we can extract CNS information such as Emotional-Valence. The PPG allows us to estimate the user’s heart rate and thus better understand his arousal levels. Technical specifications of the compo-

1https://github.com/chipimix/thesis 2http://bitalino.com/ 3https://www.linkedin.com/in/hugo-silva-0661763/ 4https://github.com/BITalinoWorld/python-serverbit/commit/151cfbb0a80efd360c4dc917b141dfb9118a79ff

39 Figure 4.2: BrainBIT prototype Outside and Inside head- Figure 4.1: Bitalino board with EMG[18] band view detailing: (1) The Power Block; (2) Battery; (3) PPG; (4) Electrodes. nents can be found at Bitalino’s website5, including hardware data-sheets6.

Table 4.1: Bitalino’s Technical Speciﬁcations [20]

Speciﬁcations Sampling rate Conﬁgurable to 1, 10, 100, or 1,000 Hz Analog ports 4 input (10 bit) + 2 input (6 bit) Digital ports 4 input (1 bit) + 4 output (1 bit) Data link Class II Bluetooth v2.0 (∼10-m range) Actuators LED Sensors EMG, ECG, EDA, Accelerometer , and PPG Weight 30 g Size 100 × 60 mm Battery 3.7-V LiPo Consumption ∼65 mAh (with all peripherals active)

The next step was the assemblage of the Bitalino. The pre-ordered Plugged Kit consists of electronics (cables, sensors, the processing unit and other blocks) but no supporting apparatus as seen in Figure 4.1. As such, a considerate amount of thought went into the ergonomics and the resistance to wear and transport of our device. Our solution, created by IBEB colleague Mariana Antas de Barros, consists of two velcro strips glued to each-other with the hardware mostly between them. The only visible electronic components on the inside of the strip are the electrodes. The PPG sensor, which hangs on the ear-side, the Power Block with its indicate LED and power switch and the battery are visible on the outer side of the strip. All of this is detailed and can be seen in Figure 4.2. Finally, our headband had improved usability substantially over its original components. Its wearable nature allowed it to be used without cables or any other support. Ergonomically, the soft component of velcro was placed in the inner side of the headband so that our prototype was comfortable and ﬁtted

5http://bitalino.com/en/ 6bitalino.com/datasheets

40 users regardless of head size. This prototype’s uniqueness entitled it its own name: BrainBIT. The ﬁnal result can be seen in Figure 4.3.

Figure 4.3: BrainBIT BCI ﬁnal prototype used in this project.

4.1.1 Device Conﬁguration

Bitalino is well documented7, which facilitates its configuration. When powered on, its firmware is prepared to receive a set of commands that can be used to control the state of the device. Namely, Idle - with the LED blinking at 0.5Hz, biosignal streaming is off - and Live - activating each sensor channel stream while the LED blinks at 1Hz. Additionally, Data Packet configuration, totaling a maximum of six channels, up to four with a resolution of 1024 bits two limited by a 64bit resolution, is presented in Table 4.2 - as previewed in the beginning of section 4.1.

Table 4.2: Depiction of our Bitalino Channel-Sensor conﬁguration

Analog Channel Correspoding Sensor Sensor Resolution A0 Left Electrode 1024 bits A1 Right Electrode 1024 bits A2 Photoplethysmogram 1024 bits A3 Accelerometer x-axis 1024 bits A4 Accelerometer y-axis 64 bits A5 Accelerometer z-axis 64 bits

Finally, one last imperative setting is set to be conﬁgured: the Micro-Controller Unit (MCU)’s sample rate. It is conﬁgurable to 1, 10, 100 or 1000Hz.

7http://bitalino.com/en/learn/documentation

41 Since the brainwave frequencies we intend to analyze lie in the zero to 50 Hz range (Table. 2.1), the device’s sample rate should be of at least 100 Hz, as stated by the Nyquist–Shannon Sampling Theorem [64]. Nonetheless, we chose a sample rate of 1000 Hz as this a more scientific approach with the added benefit of supporting the identification of harmonics (distortion) in waves of higher frequencies- up to 500Hz.

4.2 Architecture

System architecture’s can be divided in three main pillars: the hardware - which has been extensively presented in section 4.1; the users - whose perception we are attempting to improve, as explicit in sub- section 4.2.1; last but not least, the software - responsible for filtering, processing and the derived data InfoVis. Figure 4.4 portrays the behavior and structure of these comprising elements in a detailed, conceptual system architecture. This comprehensive schematic displays system modulus using numbers - from (1) to (5). Additionally, lettered, colored arrows - from (a) to (e) - display the data pipeline transitions from chemical and electrical biosignals to filtered, processed, derived psycho-physiological data. Upon scrutinizing our figure we can better understand our system components, their functions and how they Interact with each other. Particularly, its data abides by the following color scheme, according to its stage:

Black stands for the analog biosignals input, from a current or previous subject.

Red represents the U digitalized biosignal. These vary in resolution from 64 to 1024 bits as stated in Table4.2;

Green depicts the python output preprocessed data which features saturation removal, peak-ﬁnding and the raw EEG singal’s resulting Power Spectrum calculus.

Yellow illustrates the positional coordinates input system used to interact with the InfoVis

Blue portrays the output of visual data.

These categories help us understand how the system’s software processes and transforms previously incomprehensible Physiological Data, into a visually recognizable depiction of data. To do so, we rely on two main software components:

ServerBIT Python back-end application, based on the Twisted8 event-driven networking engine. This architecture is based on an asynchronous message passing protocol, in which the server and

8http://twistedmatrix.com/

42 Figure 4.4: System architecture. Different, arrow colors and directions represent different system interactions: In Black (a), we see the users’s (1) electrical and chemical biosignals (EEG and PPG) and the physical accelerometer input. In Red (b), BrainBit (2) listens for requests - sent over Bluetooth by ClientBIT (3) - which sends a raw, discrete domain, digitalized version of the user’s biosignals, accordingly. This version is then sent to ServerBIT (4) for post-processing. Indeed, in Green (c), we see the the Python Server (4), JSON-formatted, preprocessed data answer the request it received (in identical fashion) from ClientBIT (3) - a web-based, Javascript/HTML InfoVis interface.In Yellow, the input means of exploring our InfoVis - typically a computer mouse - is depicted. Finally, a large blue Blue, one-directional arrow, represents the optical NeuroExplore (5) InfoVis output, through which the user can visually analyze their own, previously collected, or real-time, BrainBIT Psycho-Physiological data. Finally, in Yellow, the input means of exploring our InfoVis - typically a computer mouse - is depicted.

the client communicate using JSON-formatted strings. ServerBIT receives Python instructions as string-based requests from a client, evaluates them, and replies with the same instruction having the evaluation results as input arguments.

ClientBIT is the HTML/Javascript/CSS front-end that connects to ServerBIT and opens a speciﬁed BITalino device to request biosensor data Using the Data Driven Documents (D3.js) javascript library 9. D3.js helps us bring data to life using HTML, SVG and CSS. D3’s emphasis on web standards allowed us to build our own framework, combining powerful visualization idioms and a data-driven approach to DOM manipulation.

4.2.1 Users

It is important to understand that our system is focused on the visualization of brain patters and other Physiological signals. As such, for design and validation purposes, it is relevant to deﬁne our user as the ﬁnal InfoVis data visualizer. This distinction is relevant: the wearable user is not necessarily the user visualizing and interacting

9https://d3js.org/

43 with its resulting data. Indeed the wearable user can be a mere test subject, whose aggregate data can be of interest for an external entity. Notwithstanding, the user can interact with our browser-based InfoVis in one of two ways: using just a screen and a mouse, or a touch-based interface, to visualize previous recordings; or, adding BrainBIT into the equation, visualize their mental inner-workings in real time.

4.3 Development Process

With our hardware means of raw, biosignal data acquisition established, we now present all stages of development which led into our ﬁnal, interactive InfoVis solution: NeuroExplore.

4.3.1 Biosignal Dataset Development

The foundation for any InfoVis system is the data whose cognition, perception and insight we are attempting to revolutionize. We aim to transform and represent data in a form that allows human interaction such as exploration and newfound knowledge as a result. This is what differentiates an InfoVis from a more traditional and elementary Graphical User Interface (GUI). Accordingly, we set out to collect a physiological, big data sample using BrainBIT. As we will detail in sub-sections 4.3.2 and 4.3.3, several neuroscientific studies have documented correlation between specific brain-wave frequencies’ power changes, according to their respective electrode scalp location in the International 10-20 system (Figute 2.3), and certain activities such as Meditation [65, 66], mental Engagement (or Focus) [67–69], and positive or negative emotional-valence [70–72]. Originally, these studies were conducted with traditional EEG lab equipment. Contemporaneusly, commercial BMI EEG solutions have shown similar results [28]. Indeed Work by Thie [73] and Debener [74] suggests that these wearable systems are valid tools for measuring visual evoked potentials and EEG activity when walking outside. Accordingly, we devised a set of tasks in which, in accordance to our goals (Chapter1, the mentioned, mental activities, emotions and varying degrees of focus are stimulated. The exercises’ nature was designed according to previous investigators work in the field of Neuroscience and [! ([!)qEEG], as suggested by IBEB colleagues. Particularly, these were designed to stimulate specific the mental action or processes in which correlation has been found by analyzing different scalp location qEEGs. The chosen tasks The final result can be seen in Table 4.3. As a result, and considering the fact that there is no impediment in measuring the acceleration and PPG in the ensuing conditions, the chosen activities for our test-subjects to complete were designated as follows:

44 Meditation Users were asked to complete 12 minute’s meditation sessions using Headspace. Headspace is a Mindfulness meditation app with scientiﬁcally proven beneﬁts [75–77]. More information regarding this app is found on its web-site10.

Puzzle Solving Users’ signals were recorded while playing free ﬂash puzzle games for ﬁve to six minutes.

Music Listening A wide range of music videoclips were showed (visual and audio) to our users. Addi- tionally, IADS were used in two, closed-eyes sessions (see sub-section 4.3.1.A for more details). This usually took four to six minutes.

Video Watching Small emotional stories or movie clips as well as fail videos were showed to the users for approximately ﬁve minutes.

Table 4.3: Depiction of each session’s activities, besides meditation

Puzzle Game Song Author and Title Video clip URL Title S01 James Brown - I Feel Good Bloxorz youtube.com/watch?v=GLq Vp5z9D4 S02 Animals - House of the Rising Sun Bloxorz youtube.com/watch?v=f8OmSWxF6h8 S03 The Verve - Bitter Sweet Symphony Flow Free youtube.com/watch?v=etKoVSPfBJ0 S04 Radiohead - Like Spinning Plates Flow Free youtube.com/watch?v=OnMG8G0NG U S05 Gorillaz - Clint Eastwood Isoball 3 youtube.com/watch?v=F2bk 9T482g S06 Bethoven - Moonlight Sonata Isoball 3 youtube.com/watch?v=-rgDvP39Lqw S07 Mogwai - Auto Rock Redremover youtube.com/watch?v=GV4DCfZt-Qo S08 Crazy Astronaut - State Redremover youtube.com/watch?v=GV4DCfZt-Qo v2 S09 Cartoons - Witch Doctor SuperStacker2 youtube.com/watch?v=zCsmBDI8vNA S10 Police -Walking on the Moon SuperStacker2 youtube.com/watch?v=a9mJQiJnheo S11 Siavash Amini - The Wind Blosics 2 youtube.com/watch?v=9UchIJV9jGM S12 Bonobo - Silver Blosics 2 youtube.com/watch?v=9UchIJV9jGM3t=6m S13 Mariah Carrey - Without You Klocki youtube.com/watch?v=9UchIJV9jGM3t=11m S14 Pirates of the Caribbean Main Theme Klocki youtube.com/watch?v=9UchIJV9jGM3t=17m S15 Pokemon Season One Main Theme Electric Box 2 youtube.com/watch?v=7grc 6cDYd4 S16 Bonobo - Stay the Same Electric Box 2 youtube.com/watch?v=7SmN1D5eS4w S17 Peer Gynt - Morning Mood Color Switch youtube.com/watch?v=-cBgb8IOO2U S18 Los del Rio - Macarena Color Switch youtube.com/watch?v=sfzsznPP590 S19 IADS Audio Clips 4 In A Row youtube.com/watch?v=sfzsznPP590#t=6m S20 IADS Audio Clips 4 In A Row youtube.com/watch?v=sfzsznPP590#t=12m

BrainBIT captured the user’s physiological signals in each session. Explicitly, each second of a recording contains 6000 data values: 2000 for each of the two EEG with a sampling resolution of 1024; 1000 for the PPG, also with a resolution of 1024bits; and ﬁnally, 3000 for the accelerometer with a resolution of 1024 bits for one of its channels and 64 bits for the other two due to technical limitations as seen in Table 4.2.

10https://www.headspace.com/science/meditation-research

45 A total of twenty sessions per user was achieved. Each session consisted of four activities starting with meditation followed by puzzle solving, music listening and then video watching. Typically this would take around 35 minutes with small breaks between each activity of around 30 to 120 seconds. This pause was intentional as it allowed the users to provide feedback as well as to decompress.

As a consequence, each session contained between 7 920 000 to 10 440 000 data values as its duration varied from 21 to 29 minutes according to movie and music lengths. A total of 23 sessions were completed between April and June 2017. Three 24 year old male participants took part in this endeavor.

The ﬁrst three sessions were excluded because we wrongly saved processed data instead of RAW data. The remaining 20 sessions were divided by tester (three) and activity types (four) which resulted in a total of 250 CSV ﬁles.

Figure 4.5: Recording CSV ﬁle screenshot snip depicting a six-line long, respective to one second, selection (a) with one thousand CSV each (b).

Finally, each CSV ﬁle was formatted as visible in Figure 4.5 so that each six lines corresponded to one second of recording. Each one of these lines stores 1000 comma separated data values speciﬁc to each channel. Indeed, each of these lines corresponds, accordingly, to each Bitalino Channel, from A0 to A5, as seen in Table. 4.2. This channel format is respected both in Real-Time as well as in Retrospective VIS

The full extent of data preprocessing filtering, subsequently described in subsection 4.3.2, resulted on each recording’s respective CSV, database file decrease of approximately 37%, as visible in figure 4.6.

In the end, the database decrease from approximately 7,97 to its final size of 5,81GB. The entire gather process took three months, starting March 27th, and features 250 CSV files spanning four activity categories correlated with specific, documented, brain patterns.

46 Figure 4.6: Raw biosignals (left) versus Preprocessed (right) database sizes, inGB.

4.3.1.A IADS

The International Affective Digitized Sound system (IADS) provides a set of acoustic emotional stimuli for experimental investigations of emotion and attention.

This set of 167 naturally occurring standardized, emotionally-evocative, internationally accessible sound stimuli is widely used in the study of emotion. Its includes contents across a wide range of semantic categories.

It is part of a system for emotional assessment and was developed by the Center for the Study of Emotion and Attention (CSEA) at the University of Florida. Each sound-clip lasts six seconds long and consists of varied examples such as: bees buzzing, a baby laughing, a female sighing or trafﬁc noises.

Finally, as seen in Table 4.3, we played these sound in a closed-eyes recording session.

4.3.2 Data preprocessing

The massive nature of collected, enigmatic, raw biosensor data added to its susceptibility to real- time recording condition deterioration demanded preprocessing, for feature extraction and ﬁltering purposes. As such, we started by building a data cleaner Python script, which we used for the retrospective database and extended it into the real-time back-end preprocessing development. NeuroExplore data preprocessing is implemented by the Python server - serverBIT. Receiving as input the unprocessed biosensor signal, this back-end component is tasked with the following successive data transforma- tions:(1) Filtering Signal Saturation; (2) Calculating the EEG Power Spectrum; (3) PPG blood volume pressure Peak-Finding.

47 4.3.2.A Saturation Filtering

As we began acquiring data, we noticed BrainBIT was susceptible to external, electro-magnetic noise.

Visible examples featured the interference from the Alternate Current (AC) power supply - were discernible in the InfoVis Power Spectrum at 50Hz. In addition, other coherent noises [78] were visible in the raw EEG line-chart. In one instance, this coherent noise was the clear outcome of a nearby cooling fan electric motor. This correlation was visible in NeuroExplore as we moved closer of away from the fan.

As stated in section 4.1, we intended to ﬁlter saturation using our physical sensor - the Accelerometer.

In practice, any unprocessed EEG recorded saturation lasting more than 10ms meant it is respective 1000-values communication had to be removed. The justiﬁcation for this is two-fold: Firstly, 10ms correspond to the period of a 100Hz wave, and thus, the resulting noise could compromise the Power Spectrum; Secondly, we could not simply remove these values and keep the rest of the recording because, further in the data transformation pipeline, Fast Fourier Transform (FFT) requires a temporally equidistant input.

The resulting EEG-Saturation ﬁltered data included all data instances in which Accelerometer-detected abrupt changes. In other words, we found that recorded velocity differences were always accompanied by detrimental EEG saturation. As a consequence, we ended up discarding the Accelerometer data from the preprocessing procedure.

The ﬁnal, detailed Algorithm 4.1 depicts how we detected raw signal saturation.

Algorithm 4.1: EEG Raw Signal Saturation Detection for each second of input data input : raw EEG1 and raw EEG2, 1000 long input Arrays with Left Frontal Polar (FP1) and Right Frontal Polar (FP2) raw EEG signal output: Boolean relative to Saturation presence in this second of recording count raw1 = 0 count raw2 = 0 for curr ms ← 0 to l000 do read current; if raw EEG1[curr ms] ≥ 1018 or raw EEG1[curr ms]=0) then count raw1++ if raw EEG2[curr ms] ≥ 1018 or raw EEG2[curr ms]=0) then count raw2++ if count raw1 ≥ 10 or count raw2 ≥10 then return true else return false

48 4.3.2.B EEG Power Spectrum Calculus

In order to derive the EEG’s Power Spectrum, we used the NumPy library11. This Python library implemented a function - rFFT12 - which computes the one-dimensional discrete Fourier Transform for real input. This means the function does not compute the negative frequency terms, and the length of the transformed axis of the output is therefore n/2 + 1., which resulted in 501 values (from 0 to 500Hz) as n = sampling rate = 1000 Hz. In our case, this real input is the digitalized, electrode captured, 1000 electric potential difference values - each associated with a time interval of one millisecond. This input was received every one- second, from BrainBIT. Thus, this function computed the one-dimensional 501-point discrete Fourier Transform (DFT) of a real-valued 1000-point array by means of an efﬁcient algorithm called the Fast Fourier Transform (FFT) [79]. The resulting formula is visible in equation (4.1).

PS = EEG Power Spectrum, ranging from 0 to 500Hz rFFT = One-dimensional Discrete Fourier Transform for real input. raw = One second respective 1000 values of unprocessed EEG signal

PS = | rF F T ( raw) |2 (4.1)

4.3.2.C PPG Peak-Finding Algorithm

The PeakUtils13 Python package ascribed an Massachusetts Institute of Technology (MIT) license for this undertaking. As the name implies, it implements methods for detection of peaks in one-dimensional data (e.g. BrainBIT output signals). This is achieved by ﬁnding the index of the local maxima, depending on the passed threshold and minimum distance. We chose it because of its efﬁciency and simplicity, its a Python implementation of Matlab’s Find Peaks function, and because it also supports methods for baseline correction, Gaussian adjustment and others. Considering the PPG’s blood volume values rested at half on the sensor’s resolution (512), through a trial and error approach, we set the normalized threshold as seen in Equation (4.2). prev ppg = up to 10 seconds (10 000 values) of previously preprocessed PPG sensor data. It is not normalized and, thus, must be divided by its maximum resolution (1023).

11https://docs.scipy.org/doc/numpy/user/index.html 12https://docs.scipy.org/doc/numpy/reference/generated/numpy.fft.rfft.html#numpy.fft.rfft 13https://pythonhosted.org/PeakUtils/

49 max( 0.5, 0.8 ∗ (prev ppg)/1023) (4.2)

Additionally, we set the minimum distance between consecutive peaks as 600 ms, as this value corresponds to the heart-rate of 100Beats per minute (bpm), which, in turn, stands at the limit of the typical adult’s resting heart-rate spectrum - [60 − 100] bpm.

4.3.3 Derived Data

NeuroExplore relies on attributes derived from the ServerBIT output of preprocessed data. In this subsection we will document how the front-end (ClientBIT) derived each of the following metrics: Theta (θ), Alpha (α) and Beta (β) brain oscillations; Heart-rate; Emotional Valence; Engagement; and Medita- tion. These last three metrics in particular: Emotional-Valence, Engagement and Meditation, are deter- mined exclusively for each second of non-saturated communication. This was intended as it is futile to derived metrics from a saturated signal. In these missing value situations, we opted to assign the previously, adequately derived, metric.

4.3.3.A Brain-wave extraction

As we will see further ahead, for our derived metrics we will need to extract information from these specific brain waves: Tetha (θ), Alpha (α) and Beta (β). There as several possible protocols to extract brainwaves, as studies have shown correlation with the sum of a specific wave’s spectrum, the value of its peak or its location in the mentioned spectra [66]. This lack of a flawless protocol is candid example of Neurosciences’ scientific research knowledge- frontier exploration and discovery. Enthusiastically, we propose and test two techniques. Firstly, we used the maximum value of a specific neural oscillation as its representative metric. This can be seen in equations (4.3), (4.4) and (4.5).

PS = EEG Power Spectrum, ranging from 0 to 500Hz h = speciﬁc frequency, in hertz

θ = max( PS(h)17≤h≤20 ) (4.3) α = max( PS(h)17≤h≤20 ) (4.4) β = max( PS(h)17≤h≤20 ) (4.5)

Secondly, we propose the calculus of the mean value of each wave’s speciﬁc frequencies, as seen in equations (4.6), (4.7) and (4.8):

θ = mean( PS(h)17≤h≤20 ) (4.6) α = mean( PS(h)17≤h≤20 ) (4.7) β = mean( PS(h)17≤h≤20 ) (4.8)

In the end, we opted for this second option as we observed less erratic oscillations, over time, in the

50 consequently derived metrics. This, in turn, resulted in signiﬁcant less outliers for the derivable metrics in succession.

4.3.3.B Heart-Rate metric

Heart rate is the speed of the heartbeat measured by the number of contractions of the heart per minute (bpm). Our PPG signal represents such contractions via the acquisition of more or less light in accordance to the ﬂuctuating blood volume pressure. As described in sub-section 4.3.2.C, when the PPG data reaches the front-end application, it has been preprocessed and features a sentinel value (1023) for each heart-beat corresponding peak. As such, we keep in a queue the nearest 10 seconds respective to each detected and recorded spikes. Every time we a receive a new message, we update the queues values, adding one second to previous ones as well as the new one. Finally, we deﬁne the heart rate as the average time difference between each two consecutive queue entries. The javascript function responsible for this - estimateHR() - is detailed on AnnexA, with respective project code comments for added insight.

4.3.3.C Emotional-Valence metric

Neuroscientists have correlated several emotional states (such as depression [80]), as well as the Emotional-Valence dimensional space, with EEG FP1 and FP2 alpha-wave asymmetry [70–72]. Accord- ingly, we deﬁned our Emotional-Valence metric as seen in Equation (4.9):

EmotionalV alence = αFP 1 − αFP 2 (4.9)

These metric values are then subtracted from one another, according to their FP1 and FP2 electrode scalp location. This frontal, alpha-wave asymmetry is intended to facilitate the users’ current emotional attraction vs rejection, thus translating into ordinary Human language the originally complex, localized electrode’s electric potential readings.

4.3.3.D Meditation metric

We have previously justified our motivation for measuring emotional-valence and engagement, respectively, for their neuromarketing and high-performance training capabilities. Meditation has also been for subject of neuroscientist’s EEG analysis for some time [65]. Correspondingly, we believe that it is of interest to be able to monitor inter-session meditation, as well as their respective evolution. Indeed, scientific studies have proved the daily-life health benefits

51 of this practice, decreasing stress levels which, chronically, have been linked to the development and progression of this broad spectrum of disorders [66]. We based our formula on review the work by Fingelkurts [66] which features a state of the art table listing 35 studies correlating meditation with alpha-power increases (versus 3 against) as well as theta power surge (27 studies).

θ + θ + α + α Meditation = FP 1 FP 2 FP 1 FP 2 (4.10) 4

4.3.3.E Engagement metric

Neuroscientists have correlated Task Engagement, Mental Workload in Vigilance, Learning, and Memory Tasks with EEG[68]. Different studies have attempted to evaluate player task engagement and arousal. Evaluating player task engagement [67, 69] using the following equation (4.11):

β + β Engagement = FP 1 FP 2 (4.11) αFP 1 + αFP 2 + θFP 1 + θFP 2

We also adopted this algorithm, in particular due to its player engagement measuring correlation, as one of our presented database’s tasks is playing puzzle games.

4.3.4 InfoVis Development

In this subsection we detail the various iterative development process stages which led into the current version of NeuroExplore, a Physiological Computing InfoVis through which users can better decipher their brain and body’s Psycho-Physiological state. Thoroughly, we detail how NeuroExplore handled speciﬁc data structures to achieve a fast action/feedback loop required by dynamic queries, through vision and biosignals alike. These components are integrated into a coherent framework that simpliﬁes the management of so- phisticated physiological data structures through the BrainBIT BCI or our collected database, the Server- Bit Python back-end, and the InfoVis Javascript implementation, ClientBIT.

4.3.4.A Low Fidelity Prototype

Initial NeuroExplore development began after several meetings and correspondence with IBEB faculty in which we probed whether hypothetical design solutions were viable from a Neuroscientiﬁc, often biosignal analysis, standpoint. Furthermore, our bioengineering colleagues provided inﬂuencing feedback, as we recurrently aimed to better understand and meet their expectations. Upon learning important Physiological Computing and qEEG concepts we started testing Bitalino Prototypes and their respective OpenSignals VIS solution as well as API development protocols and

52 examples14, we started tinkering with possible InfoVis solutions. These were usually discussed and validated by the mentioned experts. Figure 4.7 portrays an exemplary paper prototype, as aspired in Februart 2017, based on collected feedback.

Figure 4.7: Initial NeuroExplore conceptual paper prototype artifact, dating February 2017

Early on, as seen in Figure 4.7, we learned about the possibility of deriving metrics such as Attention, Meditation and Emotional Valence as well as their respective visual idiom presentation.

4.3.4.B High Fidelity Prototype

On January 15th, in a meeting with fellow investigators at IBEB, we had the opportunity to meet Bitalino creator, Dr Hugo Silva15. We discussed Bitalino’s configurations extensively - specifically its channel resolutions, sensors and output rates as well as API documentation so that we could build a High Fidelity, functional prototype. The main motivation for this was the need to start collecting a database through which we could de- liver our InfoVis aspirations. As such, we followed the Python Client-Server API16 guidelines to develop the initial prototype version. The initial NeuroExplore software prototype, depicted in Figure 4.8 featured six charts: two server derived-data qEEGs; and four unprocessed data linecharts: one PPG and three axis-specific accelerometer output. This version allowed the recording and saving of Physiological data, indispensable to the database’s acquisition. To do so, we used a Python back-end and Javascript/HTML front end, Client-Server communication

14http://bitalino.com/en/development/apis 15https://www.linkedin.com/in/hugo-silva-0661763 16http://bitalino.com/en/development/apis,https://github.com/BITalinoWorld/python-serverbit

53 Figure 4.8: Initial NeuroExplore software prototype displaying live data acquisition, April 4th, 2017.

application. We embraced Twisted’s17 event-driven networking engine, following an asynchronous JSON formatted message passing protocol. Finally, this prototype’s VIS implementation was already architectured around HTML, and CSS concepts relying on a graph plotting jQuery library: Flots.js18. Depicted idioms updated their images as new communications where received, each second. This version’s was live-tested in the format of the database acquisition, where we also collected user feedback. As the sessions took place, we started to better understand how the user rationalized the [!InfoVis ] as well as how we could further improve interaction.

4.3.4.C β Version InfoVis

Iterating upon previous prototype versions, this stable version - reproduced in Figure 4.10 - presented the usage of Scalable Vector Graphics (SVG) through the usage of a different Javascript plotting library: D3.js. In addition to real time, this version featured, at the time, separate retrospective InfoVis analysis component of Physiological data. To do so, a small horizontal file selection bar stood at the top of our vis (which allowed CSV file input. Furthermore, a time slider interface, complete with play and pause buttons was located at the bottom of the screen One of the main motivations behind choosing the D3.js library was its proliferation of distinct examples. Through these, we were able to fulfill a common user feedback request: the animation between the time varying data. The β version featured the smooth animation of data instead of flickering every one second. In addition, logarithmic scaling - of base ten - was employed in the qEEG y-axis. This was the result of empirical NeuroExplore usage as, early on, we noticed values respective to smaller frequencies were

17http://twistedmatrix.com/ 18http://www.ﬂotcharts.org/

54 in a signiﬁcantly higher scale than the remaining. In the end, this solution was adopted as a method of visually displaying all spectral data at once.

Figure 4.9: NeuroExplore retrospective mode beta-version InfoVis, featuring the VIS of post-processed, derived physiological metric’s charts. On the left we can see the maximazed menu. On top, its show/hide interface next to the ﬁle selection interface. On the bottom, the previous recording time slider with respective play/pause buttons is visible. The plotted idioms y-axis log-scale usage is also visible.

Finally, this version also supported a lateral, minimizable menu bar. In here the user can ﬁnd Instruc- tions as well as Information and Contacts regarding NeuroExplore support. At this stage, this menu was thought of as a mere assistant whose interaction was mainly destined to intertwine between real-time and retrospective mode and the presentation of instructions for newbie users such as tutorial and other assistance information. The rationale behind was that other interactions - deemed more common - such as the ﬁle opening and recording slider manipulation, should be constantly visible for a quicker overall interaction.

4.3.4.D Final InfoVis

A Beta testing, formative evaluation in which users were asked to use the think aloud protocol resulted in qualitative user data acquisition as well other feedback and suggestions. Three users - unfa- miliar with the project - volunteered for this experiment: two female, one male with ages ranging from [24 − 55]. Speciﬁcally, we decided it was important to improve the interactivity of our VIS in order to increase exploration potential and increase user data perception through visual manipulation. In order to achieve this, we implemented several features, divided in the two following categories:

RetroVis As suggested by users, we decided to increase the visual space for interaction so that we

55 could support the display of six idioms at once. In order to achieve this we moved the ﬁle selection interface into the Menu, with a larger draggable recording upload area. This provided extra comparison capabilities to the user. Additionally, overall interface presentation stood out as cleaner since no chart was only partially visible and the menu integrated - and hid - functions external to the current visual manipulation. Finally, mouse-over interactivity was added so the user could more expeditiously interpret the presented data, instead of relying on axis scales.

RTVis In addition, real-time speciﬁc interaction was enhanced via the display of error messages and red highlights according to incorrect PPG heart rate readings, indicated as a probable sensor misplacement, as well as EEG saturation, in real time. This had been previously requested by system tester which, in some cases, had difﬁculty accomplishing system tasks due to unrecognized saturation.

Figure 4.10: NeuroExplore retrospective mode ﬁnal-version InfoVis, introducing a broader VIS space due to input ﬁle-selection menu integration and mouse-over interaction,

Finally, in anticipation for the usability tests presented in Chapter5, we implemented a information messaging system on both VIS modes. Intended to decrease errors and task completion times, once interaction with NeuroExplore starts, the user is greeted with guiding and informational messages so he can know exactly what to do. This presented signiﬁcant usability increases, as, previously, empty charts occupied most of the screen until connected to BrainBIt (RealTimeVIS) or selected a previous recording ﬁle (RetroVIS)

56 4.4 Discussion

Throughout this chapter, we have extensively detailed the iterative development process which led into the ﬁnal, most current implementation of NeuroExplore. Through the development of this Physiological Computing InfoVis - intended to improve the user’s ability to decipher their mental state - we reached several veredicts:

1. We acquired and preprocessed a massive (5,81GB) biosignals database, featuring approximately 18 hours of recording spanning 250 CSV ﬁles divided into three subjects and four categories associated with speciﬁc, measurable and studied mental patterns.

2. BrainBIT Physiological Computing database acquisition capabilities have been proven. It presented outside electromagnetic noise susceptibility - visible in NeuroExplore - which needs to be managed. Additionally, the PPG sensor signal outputted weak signals in some users. We believe this is related to lightning conditions, which, if correctly handled, work as intended.

3. Our Bitalino sensor specs feature three channels which - thoughtfully - could be reassigned. The accelerometer’s anticipated ﬁltering enabling insight proved fruitless, as EEG signal saturation ﬁltering encompassed it. Currently, we have no use for this output, but have decided to keep it as part of our project to avoid risking future developments.

4. There is extended neuroscientiﬁc research regarding EEG derivation of psycho-physiological metrics. We have meticulously detailed our proposed extraction algorithms - and justiﬁed them accordingly.

5. A ﬁnal version of NeuroExplore is proposed, and its features are widely dissected: from the initial data’s output, through preprocessing, back-end, mechanisms and data derivation alike, to the front-end interactive visual display.

Through the iterative development of project, user - and often expert - feedback was equated and incremented into each surpassing version, thus increasing our goal’s success prospects by repeatedly reality-checking countless preconceived notions In sum, NeuroExplores current prototype varied features warrant it the right to be tested with a larger pool of users and throughly validated, via usability and utility testing alike.

57 58 5 Evaluation

Contents

5.1 Context: European Investigators’ Night 2017...... 61

5.2 Usability Tests...... 62

5.3 Case Studies...... 68

5.4 Discussion...... 72

59 60 Validating an InfoVis is an indispensable external indication of its features successful implementation. We will assess its usability and functionality by testing if our design has succeeded in its purpose [63]. In order to validate our solution, we decided to tackle this challenge in two distinct approaches: Usability Testing and Case Studies. The former was undertook by users without previous knowledge of the system. This was intentional in order to evaluate the system in regards to interactivity and usability. Testers were asked to complete a set of tasks and a quiz while under observation. After which, a discussion where overall feedback and personal suggestion took place. The latter corresponds to two experts in biomedicine engineering interacting with the InfoVis while providing functional feedback.

5.1 Context: European Investigators’ Night 2017

Every detail narrated in this chapter took place at the European Investigators’ Nigth 2017 event. This happened on September 29 at a science museum in Lisbon: Pavilhao˜ do Conhecimento. The event consisted of an open-doors science demonstration with dozens of investigators presenting their experiments from a wide range of scientiﬁc areas to the general public. As participating investigators this provided us an opportunity to validate our InfoVis solution and collect user feedback. Relying on our charged BrainBIT, a laptop for the VIS display and a mouse, our project stood at IBEB’s Physiological Computing table. Other BCI devices and hands-on demos were available as seen in Figure 5.1. Data was systematically collected amidst this events’ six pm to midnight duration. A large audience - of all ages - showed interest in Physiological Computing and in this project. Indeed, a considerable group of people consistently stood either waiting in line or observing our table while we attended visitors.

Figure 5.1: IBEB’s Physiological Computing table at European Investigators’ Night 2017. On the bottom-right we can see a participant ﬁlling the SUS

Amongst the visitors, not everyone we talked to showed availability to be a a formal participant. Some visitors only wanted to talk - enthusiastically - about the technology and its possibilities. Others

61 lacked technical knowledge such as the concept of an operating system’s ﬁle. One visitor just wanted to see how the headband felt like. Regardless, conversations usually skewed towards BrainBIT and our InfoVis’ hypothetical applications such as in Mindfullness an competitive sports training as well as Neuromarketing. Finally, these resulting conversations often provided insightful and interesting feedback. One example was the multi-age interest in BCI, mainly Videogame possibilities’ enthusiasm from children and teenagers and Neuromarketing interest and fears from adults. Additionally, some visitors showed sur- prising knowledge in Physiological Computing wearable devices applications - typically having seen or owned a wristwatch with biosensors.

5.2 Usability Tests

Upon many iterations of development, a ﬁnal version of our VIS must be subjected to a summative evaluation. This is intended to determine our prototype’s usability success as well as assuring a mean of maintaining standards [81] In this section we will detail how the user testing was conducted, depicting each user task and its derived, quantitative data: the amount of time necessary for the user to successfully complete each task; and the number of errors made. Additionally, upon task completion but before feedback took place, participants are asked to ﬁll a SUS, ”a highly robust and versatile tool for usability professional” [82]. Consequently, we could establish a concrete, comparable [0 − 100] usability score. Finally, a detailed account is presented beside statistical analysis of the resulting data.

5.2.1 Methodology

The tests took place in a public event at Pavilhao˜ do Conhecimento science museum. Thousands of visitors attended the event in what soon became a very productive evening, predictable by the constant crowds surrounding investigators’ tables. Overall, the Usability Tests respected the ensuing protocol:

1. Users were selected from this pool of attendees and fellow investigators without any bias towards age, gender or any particular criteria.

2. An introductory and sometimes off-script, contextual chat which subscribed the following pattern:

(a) A brief BCI technologies and applications summary led into a BrainBIT hands-on presentation explaining its functions, speciﬁcally the non-invasive nature of its physiological sensors: the PPG and the EEG and its accelerometer.

62 (b) A mention to Neuroscience’s background on concepts such as Emotional Valence - explained as Emotional attractiveness versus repulse - and its relation with FP1 and FP2 EEG’s alpha brain waves.

3. Users were seated in a chair in front of an battery powered, unplugged laptop and mouse. Inten- tionally, on it, users could see nothing other than a warning message.

4. Testers were aided in BrainBIT placement. Firstly the headband was positioned and tightened so that the electrodes were in direct contact with the forehead. Then the PPG sensor was clipped in the earlobe. Upon proper placement, testers were asked if they were comfortable wearing BrainBIT, and readjusted as needed.

5. At this point a debrieﬁng took place. Testers were told they would have to attempt to complete ﬁve tasks while using the system. The tasks were reformulated from previous evaluation sessions. Other than the tasks’ enunciation, testing was intended to be achieved in succession, without any external intervention. Accordingly, participants told were they would have no assistance during task execution. We made sure to ask our users if they were ready before each task description was read.

6. With BrainBIT powered on, testing began after one ﬁnal check where we made sure to explain there was no need to feel pressured, as it was the system that was being tested, not the user. From the beginning to completion of each task (see Table 5.1) two evaluation metrics are being registered: task completion times and number of errors.

7. One last assessment ensued which followed a meticulous 10-item Likert scale - the SUS - as depicted in sub-section 5.2.1. Participants were asked to record their immediate response to each item, rather than thinking about items for a long time.

8. Finally, a discussion where user suggestions and ideas were annotated took place.

Table 5.1: Planned User Tasks Description

Task Description T1 ”Open a ﬁle from a previous recording.” T2 ”Identify this recordings’ heart rate evolution and current value” T3 ”Change into Real Time Mode” T4 ”Shake your head and identify when the respective signal saturation occurs” T5 ”Identify and Compare the evolution of your Emotional Valence”

63 5.2.2 Results

Initial concerns of not enough participants were soon dismissed thanks to the wide number of attendees present during our demo evening. Our established quota of twenty users was successfully met. This data gathering process took a total of approximately ﬁve hours. Notably, BrainBIT’s battery, which was charged in the precedent evening, endured this test successfully. This is coherent with our database acquisition experience (see subsection. 4.3.1). The continuous Bluetooth connection also proved reliable during the consecutive completion of all the usability testing tasks. Indeed, BrainBIT opportunely sent physiological and physical data to our InfoVis’ real-time mode without communication constraints. On a less positive note, other testing conditions were not as favorable. In appreciation to our participants, whose tests were at times surrounded by a crowded and noisy environment - specially as the evening progressed. Thanks to their collaboration, the situation was made much more manageable. The same can’t be said, regrettably, regarding the electromagnetic noise which consistently enough, saturated our electrodes’ signal. We hypothesize this is credited to the large number of electronic devices both on display and on the participants behalf. Pragmatically, as a consequence, we decided to revise our Tasks. As seen on Table 5.1, Task 4 formerly asked users to ”Shake your head and identify the respective signal’s saturation”. Seeing how, in some cases, the signal was saturated from the moment Real Time Mode started, we decided to exclude this task and used the remaining tasks of Table 5.2 as a guideline instead. Participant’s ages varied from 18 to 51, 60% (12 users) being male and the remaining 40% (8 users) female. Six participants (30%) were fellow investigators and fourteen (70%) were visitors from varied walks of life. A detailed description of how each task, from T1 to T4, was typically executed by our users in presented below:

T1 This tasked asked users to open a file from a previous recording. In order to achieve this, users had to identity the Menu button as they faced a ”Please Select a Recording” message. Upon correct menu option selection, an operating system windows popped up displaying a directory with a recording file to be selected. Relying on intuitive discovery, this task posed users little difficulty.

T2 In here, users had to correctly identify the recordings’ current heart-rate. This could be achieved directly by looking at the valued displayed in the PPG or by looking at the latest value in the Heart- Rate over Time Line Chart. Either way, every one succeeded at this identiﬁcation task.

T3 Users were asked to change into Real Time Mode. Upon exploration, users successfully discovered this was accessible through the menu’s ﬁrst option.

64 Table 5.2: Final User Tasks

Task Description T1 ”Open a ﬁle from a previous recording.” T2 ”Identify the heart rate value” T3 ”Change into Real Time Mode” T4 ”Identify and compare the evolution of your Emotional Valence”

T4 Participants had to identify their Emotional Valence Line Chart as well as to comment its evolution. comparing it with previous seconds. The Identify and Compare nature of this task made it more complex. Despite this, every user completed this task.

5.2.2.A Task Completion Times

The times each user took to complete each of the four tasks seen in Table. 5.2 are presented in Table 5.3. Speciﬁcally, we can see times, in seconds, respective to the duration of each task successful completion. Additionally, statistical data is presenting regarding our results: the minimum, maximum, average time (mean), standard deviation as well as the margin of error with a conﬁdence interval of 95%. Beside Table 5.3 stands a box-plot. Intended for increased data insight, Figure 5.2 portrays the median, upper and lower quartiles and the minimum and maximum data values. After dissecting the data from Table 5.3 and Figure 5.2’s box-plot, several conclusions were reached:

1. Task one and three were the quickest as duration results were the lowest. Respectively, a minimum of 3,75 and 4,06 seconds and an average completion time of 8,55 and 8,05 seconds. Additionally, as perhaps more evident in Figure 5.2, these two tasks also featured the least amount of value dispersion. Indeed, users swiftly identiﬁed and transversed the Menu as these tasks required. We had anticipated such results given the intendedly simple nature of these Discover actions [63]. This leads us to believe that the users group generally found these tasks straightforward and intuitive.

2. Tasks two and three were slower as depicted by the maximum values of 39,18 seconds and 41,69 seconds and the averages of 14,41 seconds and 14,14 seconds, respectively. Comparably, we can clearly see broader value dispersion, as seen in Figure 5.2. This means users had a lower level of agreement with one another. Perhaps explained by the system unfamiliarity, some users struggled with the identiﬁcation nature of these tasks, losing time by attempting to interact with the observing staff. Particularly, errors detailed in sub-section 5.2.2.B and one case of external disturbance amounted to longer performances. This consequently higher completion times, in comparison to task one and three, were previously anticipated due to the increased task difﬁculty.

65 Task Duration (in seconds) Users Task1 Task2 Task3 Task4 U01 12,12 17,25 5,42 12,98 U02 3,90 5,30 6,20 6,79 U03 6,12 7,66 16,18 9,02 U04 4,73 5,73 4,40 11,39 U05 13,45 29,21 4,15 8,74 U06 3,96 4,87 6,87 7,94 U07 6,42 19,31 13,59 5,84 U08 5,76 22,16 5,11 21,13 Figure 5.2: Box Plot for user task completion times. U09 16,95 39,18 6,39 19,22 U10 4,88 4,87 11,52 9,50 U11 9,09 18,22 8,46 24,33 U12 5,91 5,88 12,43 10,88 U13 18,05 26,00 9,74 41,69 U14 4,48 4,92 4,19 8,04 U15 3,75 5,78 4,06 7,82 U16 11,35 10,41 5,83 9,23 U17 5,89 17,67 6,47 8,66 U18 13,54 21,11 9,30 27,51 U19 4,78 8,55 4,98 5,84 U20 18,04 14,02 15,75 26,28 Statistics Min 3,75 4,87 4,06 5,84 Max 18,05 39,18 16,18 41,69 Mean 8,55 14,41 8,05 14,14 SD 4,82 9,74 3,91 9,52 CI 2,10 4,30 1,70 4,20

Table 5.3: Task Completion Times

Notwithstanding, when accounting every user’s performances, tasks two and four still averaged positive times considering they corresponded to Identify and Compare actions [63].

3. Overall, tasks one and three achieved similar mean results - diverging by 0,5 seconds. Both of which averaged 5,59 to 6,39 seconds less than task two and four. Again, similar average times were achieved times - differing 0,27 seconds from each other. Finally we can state with 95% conﬁdence:

T1 took 8,55±2,10 seconds.

T2 took 14,41±4,30 seconds.

T3 took 8,05±1,70 seconds.

T4 took 14,14±4,20 seconds.

66 5.2.2.B Errors

During each task’s execution, user errors such task misinterpretations and incorrect actions were scrutinized. The resulting amounts of errors can be seen next to a brief statistical analysis in Table 5.4.

Number of Errors per Task Users Task1 Task2 Task3 Task4 U01 1 1 0 1 U02 0 0 0 0 U03 0 0 1 0 U04 1 0 0 1 U05 0 1 0 0 U06 0 0 0 0 U07 0 1 1 0 Figure 5.3: Box Plot for number of user errors in each U08 0 1 0 1 Task completion. U09 1 2 0 1 U10 0 0 1 0 U11 0 1 0 1 U12 0 0 1 0 U13 1 1 0 1 U14 0 0 0 0 U15 0 0 0 0 U16 1 0 1 0 U17 0 1 0 0 U18 1 1 0 1 U19 0 0 0 0 U20 1 0 1 1 Statistics Min 0 0 0 0 Max 1 2 1 1 Mean 0,35 0,5 0,3 0,4 SD 0,49 0,61 0,47 0,51 CI 0,21 0,27 0,21 0,22

Table 5.4: User Errors Detailed

Explicitly, Table 5.4 data dissects the minimum, maximum, average time (mean), standard deviation as well as the margin of error with a conﬁdence interval of 95%. Additionally, value dispersion is depicted in Figure 5.3 as well as median, upper and lower quartiles and the minimum and maximum data values. Upon detailed analysis the following verdicts were reached:

1. No task had a minimum number of errors higher than zero. Five testers (25%) completed all of their tasks successfully with no errors. Additionally, ten testers (50%) made one error or less during the entire testing.

2. Only one user, during the second task, made more than one error: two. Accordingly, task two, followed by task four, impaired users the most - respectively averaging 0,5 and 0,4 errors. This

67 was often either a symptom of task description misunderstanding or, on occasion, misidentiﬁcation of the derived biometrics as required by the task.

3. Globally, all tasks were successfully completed. None averaged more than 0,5 errors and as such, NeuroExplore usability is further emphasized. In direct correlation with task completion times visible in sub-section 5.2.2.A, tasks one and three bestowed less errors (0,30 and 0,35 seconds) than tasks two and four (0,4 and 0,5), respectively and on average. Again, these repercussions are a consequence of the Discover versus Identify and Compare nature of the actions performed in each task. Finally we claim, with 95% conﬁdence:

T1 induced 0,35±0,21 errors.

T2 induced 0,50±0,27 errors.

T3 induced 0,30±0,21 errors.

T4 induced 0,40±0,22 errors.

5.2.2.C System Usability Scale (SUS)

NeuroExplore results averaged a total of 85,13 in the SUS. This study provided us a comparable standard to contemplate in distinct ways, such as:

1. The acknowledgement of measurable contrast regarding other current, and future, solutions to the credit of SUS usage in ”a variety of research projects and industrial evaluations”[82].

2. The full extend of the results and their respective statistics are seen in table 5.5. These were the aftermath of the user’s ﬁrst time interacting with the system.

3. The system boasts an ”Excellent” score and a ”B” grade as depicted in Figure 5.4 and deﬁned by Bangor [83]. This achievement is a positive testament to our system overall usability and it correlates with the previously recorded, comprehensive user feedback as detailed in section 5.4.

5.3 Case Studies

Before the European Investigators’ event doors opened to the public, during preparation hours, two case studies were scheduled. The voluntaries were experts in Physiological Computing, IBEB members, respectively, MD/PhD Hugo Ferreira 1 and Bioengineering Msc ﬁnalist Mariana Antas de Barros.

1https://scholar.google.com/citations?user=92iIVdsAAAAJ

68 Table 5.5: SUS scores from each user

Questions Results (in seconds) Final Users Total Q01 Q02 Q03 Q04 Q05 Q06 Q07 Q08 Q09 Q10 (x2.5) U01 4 1 5 2 4 1 5 1 5 1 37 92,5 U02 3 1 5 2 4 2 4 2 4 1 32 80 U03 5 2 4 1 3 2 4 2 5 1 33 82,5 U04 4 2 4 4 5 1 5 2 4 1 34 85 U05 4 2 3 1 4 3 5 1 4 2 31 77,5 U06 3 2 4 1 4 1 5 3 5 3 31 77,5 U07 5 1 4 1 5 1 5 1 5 3 37 92,5 U08 3 2 4 2 4 2 5 1 4 2 29 72,5 U09 5 1 4 1 3 2 5 1 5 2 35 87,5 U10 4 2 4 1 4 2 5 1 4 1 34 85 U11 5 1 5 1 4 2 5 2 5 1 37 92,5 U12 5 1 4 2 4 2 5 1 5 2 34 85 U13 4 2 4 1 4 2 5 1 4 1 34 85 U14 5 1 5 5 5 1 5 1 5 1 36 90 U15 4 1 5 1 5 2 3 1 4 1 35 87,5 U16 4 1 5 2 5 1 5 1 5 1 38 95 U17 5 2 4 1 3 2 4 2 5 1 32 80 U18 5 1 1 1 4 2 2 2 5 1 30 75 U19 4 2 4 1 4 2 5 2 5 1 34 85 U20 5 1 5 2 5 1 4 1 5 1 38 95 Statistics Min 3 1 1 1 3 1 2 1 4 1 29 72,50 Max 5 2 5 5 5 3 5 3 5 3 38 95 Mean 4,3 1,45 4,15 1,65 4,15 1,70 4,58 1,45 4,65 1,40 34,05 85,13

Figure 5.4: Our project’s ﬁnal system usability score presented in Bangors’ Adjective Rating Scale [19]

Participants had previously worked with Bitalino and the BrainBIT prototype in particular but never before interacted with our solution. Interestingly, they also had past experience using Bitalino’s OpenSignals VIS system [20].

Before each session, users were guaranteed that no concrete metric, such as the measurement of time and errors, was being recorded. Indeed, they were explained the purpose of this exercise was to

69 explore the functionality components of our solution as it was transversed by a participants with expert knowledge in the domain of our InfoVis - speciﬁcally Neuroscience, Physiological Computing and BCIs. Both cases followed a less protocular approach. Still, the ensuing proceedings are summarizes below.

5.3.1 Methodology

1. A BrainBIT was mounted on their heads and ears and users were presented NeuroExplore through a demo in which relevant features and common tasks were exempliﬁed (see Table. 5.2).

2. Users were asked to embrace the Think Aloud Protocol as they interacted with our solution. Par- ticularly we asked them to state their current actions’ rationale and intents.

3. During this free exploration and narration, we intentionally interacted with the participant and collected feedback destined to help us validate domain terms as well as understand the most relevant features and how they were being perceived.

4. This process was intended to last past the participant’s exploration of every single feature of our solution.

The results were recorded, as seen in sub-section 5.3.2,

5.3.2 Results

Both participants extensively and enthusiastically transversed our system’s functionalities with ease. Through NeuroExplore, they could visualize raw physiological data as well as the derived affective and nervous-system related metrics both in real time as well as retrospectively. As pointed out, this con- trasted with their current solution which relied on OpenSignals VIS’s restriction to raw values (see ﬁgure 3.31 for reference). Overall, comments were extremely positive. The purposefully minimalistic and sober menuUI, as well as its smooth animation and clean color pallet were praised. Participants appreciated these aesthetics in what they deemed as corresponding to the simple and intuitive user-system interactions. In detail, line-chart mouse interaction resulting in data knowledge without the visual aid of the scales was exalted. What’s more, the location of the remaining interactions inside an extendable menu was pointed out as facilitating both the analysis of a wider area of visible data as well as overall interaction with other relevant system functions by concentrating all these in one place. Interestingly, the employed algorithms and their respective representation were discussed as well as the on-ﬁeld data collection possibilities of our projects’ web-based nature.

70 Additionally, testers stressed recurring connection issues while recording data using OpenSignals. These situations could potentially frustrate user recording sessions. Thus our positive experience when using our solution to collect data was appreciated and successfully put to the test.

5.3.2.A User 1

The ﬁrst session was attended by professor Hugo Ferreira from IBEB 2. Professor Hugo has extensive previous experience working with distinct BCI devices, including Bitalino, and their respective software. Despite previous support in many meetings, our participant had never actually interacted with NeuroExplore. As such, the tests began enthusiastically and at a fast pace with the speak-aloud protocol not always being respected.

Nevertheless, feedback was plenty. After quickly selecting a previous recording, our collaborator pointed out to adequate usage of a logarithmic scale in the yy-axis of the power spectrum. Indeed, a linear scale would be incompatible with the value dispersion, specially at lower frequencies. After- wards, the post-processed, high value outliner in each of the corresponding PPG’s peaks milisecond was inquired upon and explained.

Subsequently, upon looking at all idioms as a whole, the professor inquired about the positioning of each chart. While noticing they were divided in groups according to brain signals, blood volume pressure signals and then derived metrics, the following suggestion ensued: ”Group brain signals next to their derived metrics”.

As the case study progressed, the user moved on to the Real Time Mode of our system. Then, upon realizing inadequate detection of blood volume pressure, the professor swiftly tested different lighting conditions and repositioned the ear-clip until satisﬁed with the resulting PPG. This setback, while regret- table, interestingly depicts the responsiveness and interactions made possible between our BCI and our InfoVis.

Upon further inspection of the power spectrum, our user also suggested:” it would be remarkable if we could add a line representing a previously measured, no user movement baseline to the power spectrum”. Indeed, this feature would allowed to better differentiate our results from the surrounding environment electromagnetic noise.

Most notably, professor Hugo said he was ”intending on using the system”. Emphasizing his interest in using NeuroExplore at IBEB, we were strongly encouraged to develop proper documentation for easier future setups.

2http://ibeb.ciencias.ulisboa.pt/pt/hugo-alexandre-ferreira/

71 5.3.2.B User 2

The second user was fellow colleague Mariana Antas de Barros, who is also finishing her MSc degree thesis using a Bitalino as input for an artistic, audio-VIS. Mariana stated that she was also using FP1-FP2 alpha-wave asymmetry as a means of capturing Emotional Valence used as input in her VIS. Explicitly the sum of one electrode’s power spectrum alpha frequency values was subtracted by other electrode’s respective values. Much to our delight, one of her first remarks was: ”It’s so pretty!”. This happened after our collaborator swiftly opened a previous recording and was promptly presented its respective data time evolution animation. Upon further interaction, the mouse-over highlighting of chart’s values resulted in positive feedback. A brief discussion regarding data processing and other algorithms used in our derived metrics ensued as the user explored each idiom. Eventually, Real Time Mode was clicked. In here, the display of the raw electrode data was appreciated, as these allowed neurosignal experts such as our participant to collect feedback via pattern recognition. As a consequence, our first suggestion was annotated: support an option through which one can permanently close a saturation warning message. In addition to abrupt movements, the signal started to saturate depending on the user’s location, susceptible to external electromagnetic noises. Because of our collaborator’s ability to easily spot saturation on the raw electrode signal, she felt this message should be removable. Otherwise, this message was intentional as any one second recorded deemed saturated was discarded in post-processing and most users would appreciate feedback on saturation other than the raw signal line-chart. Finally, when wondering about possible usages for the unused space at the bottom of the InfoVis such as representing new derived metrics or supporting idioms for other Bitalino sensor configurations, an epiphany moment occurred: support user selection of displayable idioms and respective Bitalino sensor channel configuration. In the end, Mariana admitted personally liking the chosen ”simple and clean” aesthetics. Additionally the broad nature of our project’s possible usages was pointed out as well as interest in the acquired data collection and its applications.

5.4 Discussion

As this section nears completion a retrospective view of our extensively detailed and diversiﬁed system of usability and functionality validation is in order. Through usability testing, we observed expeditious and always successful task completion. As anticipated Discover tasks were considerably faster and less error-prone than Identify and Compare ones.

72 Additionally, our solution stood out as ”Excellent”, scoring a 85,13 rating in the SUS. The case studies with participating IBEB members depicted how the system functions were easily achieved and thus perfectly exemplify the solution’s intuitiveness and ease of use. Furthermore, several usages and genuine interest was displayed in our project’s future. In sum, NeuroExplore was prosperous in our evaluation. Users easily, quickly and successfully identiﬁed and interacted with their own psycho-physiological mental-state data. This is what we aimed for in chapter1’s objectives: ”an interactive visualization so that users better understand one’s mental state and emotions”.

73 74 6 Conclusion and Future Work

Contents

6.1 Future Work...... 77

75 76 Retrospectively, this thesis begins with the introduction of our project and the postulus of comprising objectives which lead into the implementation of NeuroExplore. Given the diversified fields of study our solution transverses, we presented a diversified pallet of background knowledge: this ranged from the more traditional, text derived field of Sentiment Analysis, passed through Affective Computing and its in- crement on this method via multi-modal approaches, to arrive at the complementary fields of Physiologi- cal Computing and - in particular - BCI. This Computer Science meets NeuroScience backdrop then into a related work, InfoVis examples of both proprietary and academical nature. The document progresses by comprehensively entailing the development process leading into our final implementation, namely: the software and hardware’s architecture, the database acquisition process and its preprocessing, the derived metrics rational and protocol, and - finally - the iterative development nature and its procedural growth. Lastly, we vindicate NeuroExplore through a comprehensive functionality and usability, user and expert validation. NeuroExplore is a Physiological Computing InfoVis prototype, through which users can better com- prehend one’s current or previously recorded brain and body status. Achieving a SUS of 85.13, the system was considered excellent by users and experts alike. Coherently, we have fulfilled our main project goal (see Chapter1) and all of its dependencies, namely, a data pipeline which seemingly converts imperceptible biosignals into an insightful VIS, for increased, mind and body, behavior insights.

6.1 Future Work

The developped NeuroExplore system future prospects bode well. As we have detailed in chapter1, several IBEB colleagues displayed interest in working with our solution, typically by further developing it to speciﬁc project contexts. As requested by prof Hugo Ferreira, we are currently writing proper documentation to facilitate this procedure. Additional features have been requested during feedback gathering, notably - but not exclusively - in the case studies:

1. Grouping EEG brain signals next to their derived metrics

2. Graphical represent a previously measured, standing still, user baseline to the power spectrum.

3. Support an option through which one can permanently close a saturation warning messages.

4. Support user selection of displayable idioms and respective Bitalino sensor channel conﬁguration

5. Support the graphical selection of different methodologies - up to user deﬁnition - of data derivation

6. Support the visualization of multi-user physiological data regarding the same object of study or type of activity.

77 On a positive side-note, we have since implemented the ﬁrst request of grouping signals next to their derived metrics.

78 Bibliography

[1] G. A. Miller, “The cognitive revolution: a historical perspective,” Trends in cognitive sciences, vol. 7, no. 3, pp. 141–144, 2003.

[2] P. Ekman and W. V. Friesen, “Constants across cultures in the face and emotion.” Journal of personality and social psychology, vol. 17, no. 2, p. 124, 1971.

[3] B. Guthier, R. Alharthi, R. Abaalkhail, and A. El Saddik, “Detection and visualization of emotions in an affect-aware city,” in Proceedings of the 1st International Workshop on Emerging Multimedia Applications and Services for Smart Cities. ACM, 2014, pp. 23–28.

[4] E. J. Bubrick, S. Yazdani, and M. K. Pavlova, “Beyond standard polysomnography: Advantages and indications for use of extended 10–20 eeg montage during laboratory sleep study evaluations,” Seizure, vol. 23, no. 9, pp. 699–702, 2014.

[5] M.-F. Lee, G.-S. Chen, J. C. Hung, K.-C. Lin, and J.-C. Wang, “Data mining in emotion color with affective computing,” Multimedia Tools and Applications, vol. 75, no. 23, pp. 15 185–15 198, 2016.

[6] J. A. G. Sanchez, A. Shibata, K. Ohnishi, F. Dong, and K. Hirota, “Visualization method of emotion information for long distance interaction,” in Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment and Management (HNICEM), 2014 International Confer- ence on. IEEE, 2014, pp. 1–6.

[7] M. K. Torkildson, K. Starbird, and C. Aragon, “Analysis and visualization of sentiment and emotion on crisis tweets,” in International Conference on Cooperative Design, Visualization and Engineering. Springer, 2014, pp. 64–67.

[8] A. Scharl, A. Hubmann-Haidvogel, A. Jones, D. Fischl, R. Kamolov, A. Weichselbraun, and W. Rafelsberger, “Analyzing the public discourse on works of ﬁction–detection and visualization of emotion in online coverage about hbo’s game of thrones,” Information processing & management, vol. 52, no. 1, pp. 129–138, 2016.

79 [9] U. Krcadinac, J. Jovanovic, V. Devedzic, and P. Pasquier, “Textual affect communication and evocation using abstract generative visuals,” IEEE Transactions on Human-Machine Systems, vol. 46, no. 3, pp. 370–379, 2016.

[10] S. D. Kamvar and J. Harris, “We feel ﬁne and searching the emotional web,” in Proceedings of the fourth ACM international conference on Web search and data mining. ACM, 2011, pp. 117–126.

[11] C. Quan and F. Ren, “Visualizing emotions from chinese blogs by textual emotion analysis and recognition techniques,” International Journal of Information Technology & Decision Making, vol. 15, no. 01, pp. 215–234, 2016.

[12] F. Y. Wang, A. Sallaberry, K. Klein, M. Takatsuka, and M. Roche, “Senticompass: Interactive visualization for exploring and comparing the sentiments of time-varying twitter data,” in Visualization Symposium (PaciﬁcVis), 2015 IEEE Paciﬁc. IEEE, 2015, pp. 129–133.

[13] I. Hupont, S. Baldassarri, E. Cerezo, and R. Del-Hoyo, “Advanced human affect visualization,” in Systems, Man, and Cybernetics (SMC), 2013 IEEE International Conference on. IEEE, 2013, pp. 2700–2705.

[14] M. Gavrilescu and F. Ungureanu, “Enhanced three-dimensional visualization of eeg signals,” in E-Health and Bioengineering Conference (EHB), 2015. IEEE, 2015, pp. 1–4.

[15] D. Cernea, C. Weber, A. Ebert, and A. Kerren, “Emotion-prints: interaction-driven emotion visualization on multi-touch interfaces.” in Visualization and Data Analysis, 2015, p. 93970A.

[16] D. McDuff, A. Karlson, A. Kapoor, A. Roseway, and M. Czerwinski, “Affectaura: an intelligent system for emotional memory,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2012, pp. 849–858.

[17] S. Du, E. Shu, F. Tong, Y. Ge, L. Li, J. Qiu, P. Guillotel, J. Fleureau, F. Danieau, and D. Muller, “Visualizing the emotional journey of a museum,” in Proceedings of the 2016 EmoVis Conference on Emotion and Visualization. Linkoping University, 2016, pp. 7–14.

[18] J. Guerreiro, R. Martins, H. Silva, A. Lourenc¸o, and A. L. Fred, “Bitalino-a multimodal platform for physiological computing.” in ICINCO (1), 2013, pp. 500–506.

[19] A. Bangor, P. Kortum, and J. Miller, “Determining what individual sus scores mean: Adding an adjective rating scale,” Journal of usability studies, vol. 4, no. 3, pp. 114–123, 2009.

[20] H. P. da Silva, A. Fred, and R. Martins, “Biosignals for everyone,” IEEE Pervasive Computing, vol. 13, no. 4, pp. 64–71, 2014.

80 [21] I. B. Mauss and M. D. Robinson, “Measures of emotion: A review,” Cognition and emotion, vol. 23, no. 2, pp. 209–237, 2009.

[22] R. W. Picard, E. Vyzas, and J. Healey, “Toward machine emotional intelligence: Analysis of affective physiological state,” IEEE transactions on pattern analysis and machine intelligence, vol. 23, no. 10, pp. 1175–1191, 2001.

[23] E. Cambria, “Affective computing and sentiment analysis,” IEEE Intelligent Systems, vol. 31, no. 2, pp. 102–107, 2016.

[24] R. W. Picard and R. Picard, Affective computing. MIT press Cambridge, 1997, vol. 252.

[25] W. James, “What is an emotion?” Mind, vol. 9, no. 34, pp. 188–205, 1884.

[26] J. Owyang, “The future of the social web: In ﬁve eras,” Retrieved April, vol. 24, p. 2012, 2009.

[27] V. U. Druskat and S. B. Wolff, “Building the emotional intelligence of groups,” Harvard business review, vol. 79, no. 3, pp. 80–91, 2001.

[28] N. A. Badcock, P. Mousikou, Y. Mahajan, P. de Lissa, J. Thie, and G. McArthur, “Validation of the emotiv epoc® eeg gaming system for measuring research quality auditory erps,” PeerJ, vol. 1, p. e38, 2013.

[29] P. Zikopoulos, C. Eaton et al., Understanding big data: Analytics for enterprise class hadoop and streaming data. McGraw-Hill Osborne Media, 2011.

[30] J. Yi, T. Nasukawa, R. Bunescu, and W. Niblack, “Sentiment analyzer: Extracting sentiments about a given topic using natural language processing techniques,” in Data Mining, 2003. ICDM 2003. Third IEEE International Conference on. IEEE, 2003, pp. 427–434.

[31] B. Fasel and J. Luettin, “Automatic facial expression analysis: a survey,” Pattern recognition, vol. 36, no. 1, pp. 259–275, 2003.

[32] I. A. Essa and A. P. Pentland, “Coding, analysis, interpretation, and recognition of facial expressions,” IEEE transactions on pattern analysis and machine intelligence, vol. 19, no. 7, pp. 757–763, 1997.

[33] S. K. Card, J. D. Mackinlay, and B. Shneiderman, Readings in information visualization: using vision to think. Morgan Kaufmann, 1999.

[34] C. Darwin and P. Prodger, The expression of the emotions in man and animals. Oxford University Press, USA, 1998.

81 [35] P. Ekman, W. V. Friesen, M. O’sullivan, A. Chan, I. Diacoyanni-Tarlatzis, K. Heider, R. Krause, W. A. LeCompte, T. Pitcairn, P. E. Ricci-Bitti et al., “Universals and cultural differences in the judgments of facial expressions of emotion.” Journal of personality and social psychology, vol. 53, no. 4, p. 712, 1987.

[36] H. A. Elfenbein and N. Ambady, “On the universality and cultural speciﬁcity of emotion recognition: a meta-analysis.” Psychological bulletin, vol. 128, no. 2, p. 203, 2002.

[37] B. Pang, L. Lee et al., “Opinion mining and sentiment analysis,” Foundations and Trends® in Infor- mation Retrieval, vol. 2, no. 1–2, pp. 1–135, 2008.

[38] B. Liu, “Sentiment analysis and opinion mining,” Synthesis lectures on human language technologies, vol. 5, no. 1, pp. 1–167, 2012.

[39] W. Medhat, A. Hassan, and H. Korashy, “Sentiment analysis algorithms and applications: A survey,” Ain Shams Engineering Journal, vol. 5, no. 4, pp. 1093–1113, 2014.

[40] E. Cambria, B. Schuller, Y. Xia, and C. Havasi, “New avenues in opinion mining and sentiment analysis,” IEEE Intelligent Systems, vol. 28, no. 2, pp. 15–21, 2013.

[41] K. R. Scherer, T. Banziger,¨ and E. Roesch, A Blueprint for Affective Computing: A sourcebook and manual. Oxford University Press, 2010.

[42] J. Tao and T. Tan, “Affective computing: A review,” in International Conference on Affective computing and intelligent interaction. Springer, 2005, pp. 981–995.

[43] P.E. Ekman and R. J. Davidson, The nature of emotion: Fundamental questions. Oxford University Press, 1994.

[44] S. Hamann, “Mapping discrete and dimensional emotions onto the brain: controversies and con- sensus,” Trends in cognitive sciences, vol. 16, no. 9, pp. 458–466, 2012.

[45] J. Allanson and S. H. Fairclough, “A research agenda for physiological computing,” Interacting with computers, vol. 16, no. 5, pp. 857–878, 2004.

[46] S. H. Fairclough, “Fundamentals of physiological computing,” Interacting with computers, vol. 21, no. 1-2, pp. 133–145, 2008.

[47] A. Holzinger, M. Bruschi, and W. Eder, “On interactive data visualization of physiological low-cost- sensor data with focus on mental stress,” in International Conference on Availability, Reliability, and Security. Springer, 2013, pp. 469–480.

82 [48] S. Amiri, R. Fazel-Rezai, and V. Asadpour, “A review of hybrid brain-computer interface systems,” Advances in Human-Computer Interaction, vol. 2013, p. 1, 2013.

[49] M. Duvinage, T. Castermans, M. Petieau, T. Hoellinger, G. Cheron, and T. Dutoit, “Performance of the emotiv epoc headset for p300-based applications,” Biomedical engineering online, vol. 12, no. 1, p. 56, 2013.

[50] J. van Erp, F. Lotte, and M. Tangermann, “Brain-computer interfaces: beyond medical applications,” Computer, vol. 45, no. 4, pp. 26–34, 2012.

[51] M. Bares, M. Brunovsky, T. Novak, M. Kopecek, P. Stopkova, P. Sos, and C. Hoschl,¨ “Qeeg theta cordance in the prediction of treatment outcome to prefrontal repetitive transcranial magnetic stimu- lation or venlafaxine er in patients with major depressive disorder,” Clinical EEG and neuroscience, vol. 46, no. 2, pp. 73–80, 2015.

[52] M. Brunovsky, M. Viktorinova, A. Bravermanova, J. Horacek, V. Krajca, and C. Hoschl,¨ “Qeeg correlates of emotionally negative state induced by autobiographic script in patients with affective disorder and healthy controls,” Clinical Neurophysiology, vol. 127, no. 3, p. e43, 2016.

[53] S. E. Thompson and S. Parthasarathy, “Moore’s law: the future of si microelectronics,” Materials today, vol. 9, no. 6, pp. 20–25, 2006.

[54] A. Teller, “A platform for wearable physiological computing,” Interacting with Computers, vol. 16, no. 5, pp. 917–937, 2004.

[55] A. E. Hassanien and A. Azar, “Brain-computer interfaces,” Switzerland: Springer, 2015.

[56] M. B. Khalid, N. I. Rao, I. Rizwan-i Haque, S. Munir, and F. Tahir, “Towards a brain computer interface using wavelet transform with averaged and time segmented adapted wavelets,” in Computer, Control and Communication, 2009. IC4 2009. 2nd International Conference on. IEEE, 2009, pp. 1–4.

[57] V. Rozgic,´ S. N. Vitaladevuni, and R. Prasad, “Robust eeg emotion classiﬁcation using segment level decision fusion,” in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE Interna- tional Conference on. IEEE, 2013, pp. 1286–1290.

[58] J. Onton and S. Makeig, “High-frequency broadband modulations of electroencephalographic spectra,” Frontiers in human neuroscience, vol. 3, 2009.

[59] S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras, “Deap: A database for emotion analysis; using physiological signals,” IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 18–31, 2012.

83 [60] E. Cambria, D. Olsher, and D. Rajagopal, “Senticnet 3: a common and common-sense knowledge base for cognition-driven sentiment analysis,” in Twenty-eighth AAAI conference on artiﬁcial intelligence, 2014.

[61] M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic, “Analysis of eeg signals and facial expressions for continuous emotion detection,” IEEE Transactions on Affective Computing, vol. 7, no. 1, pp. 17–28, 2016.

[62] J. A. Russell, “Core affect and the psychological construction of emotion.” Psychological review, vol. 110, no. 1, p. 145, 2003.

[63] T. Munzner, Visualization analysis and design. CRC press, 2014.

[64] A. J. Jerri, “The shannon sampling theorem—its various extensions and applications: A tutorial review,” Proceedings of the IEEE, vol. 65, no. 11, pp. 1565–1596, 1977.

[65] A. Kasamatsu and T. Hirai, “An electroencephalographic study on the zen meditation (zazen),” Psychiatry and Clinical Neurosciences, vol. 20, no. 4, pp. 315–336, 1966.

[66] A. A. Fingelkurts, A. A. Fingelkurts, and T. Kallio-Tamminen, “Eeg-guided meditation: a personalized approach,” Journal of Physiology-Paris, vol. 109, no. 4, pp. 180–190, 2015.

[67] A. T. Pope, E. H. Bogart, and D. S. Bartolome, “Biocybernetic system evaluates indices of operator engagement in automated task,” Biological psychology, vol. 40, no. 1, pp. 187–195, 1995.

[68] C. Berka, D. J. Levendowski, M. N. Lumicao, A. Yau, G. Davis, V. T. Zivkovic, R. E. Olmstead, P. D. Tremoulet, and P. L. Craven, “Eeg correlates of task engagement and mental workload in vigilance, learning, and memory tasks,” Aviation, space, and environmental medicine, vol. 78, no. 5, pp. B231–B244, 2007.

[69] T. McMahan, I. Parberry, and T. D. Parsons, “Evaluating player task engagement and arousal using electroencephalography,” Procedia Manufacturing, vol. 3, pp. 2303–2310, 2015.

[70] P. C. Petrantonakis and L. J. Hadjileontiadis, “An emotion elicitation metric for the valence/arousal and six basic emotions affective models: A comparative study,” in Information Technology and Applications in Biomedicine (ITAB), 2010 10th IEEE International Conference on. IEEE, 2010, pp. 1–4.

[71] R. J. Davidson, G. E. Schwartz, C. Saron, J. Bennett, and D. Goleman, “Frontal versus parietal eeg asymmetry during positive and negative affect,” in Psychophysiology, vol. 16, no. 2. CAMBRIDGE UNIV PRESS 40 WEST 20TH STREET, NEW YORK, NY 10011-4211, 1979, pp. 202–203.

84 [72] E. Harmon-Jones and J. J. Allen, “Anger and frontal brain activity: Eeg asymmetry consistent with approach motivation despite negative affective valence.” Journal of personality and social psychology, vol. 74, no. 5, p. 1310, 1998.

[73] J. Thie, A. Klistorner, and S. L. Graham, “Biomedical signal acquisition with streaming wireless communication for recording evoked potentials,” Documenta Ophthalmologica, vol. 125, no. 2, pp. 149–159, 2012.

[74] S. Debener, F. Minow, R. Emkes, K. Gandras, and M. Vos, “How about taking a low-cost, small, and wireless eeg for a walk?” Psychophysiology, vol. 49, no. 11, pp. 1617–1621, 2012.

[75] D. Lim, P. Condon, and D. DeSteno, “Mindfulness and compassion: an examination of mechanism and scalability,” PloS one, vol. 10, no. 2, p. e0118221, 2015.

[76] A. Howells, I. Ivtzan, and F. J. Eiroa-Orosa, “Putting the ‘app’in happiness: a randomised controlled trial of a smartphone-based mindfulness intervention to enhance wellbeing,” Journal of Happiness Studies, vol. 17, no. 1, pp. 163–185, 2016.

[77] M. M. Bradley and P. J. Lang, “The international affective digitized sounds (; iads-2): Affective ratings of sounds and instruction manual,” University of Florida, Gainesville, FL, Tech. Rep. B-3, 2007.

[78] R. Neelamani, A. I. Baumstein, D. G. Gillard, M. T. Hadidi, and W. L. Soroka, “Coherent and random noise attenuation using the curvelet transform,” The Leading Edge, vol. 27, no. 2, pp. 240–248, 2008.

[79] P. Welch, “The use of fast fourier transform for the estimation of power spectra: a method based on time averaging over short, modiﬁed periodograms,” IEEE Transactions on audio and electroacous- tics, vol. 15, no. 2, pp. 70–73, 1967.

[80] I. H. Gotlib, “Eeg alpha asymmetry, depression, and cognitive functioning,” Cognition & Emotion, vol. 12, no. 3, pp. 449–478, 1998.

[81] Y. Rogers, H. Sharp, and J. Preece, Interaction design: beyond human-computer interaction. John Wiley & Sons, 2011.

[82] J. Brooke et al., “Sus-a quick and dirty usability scale,” Usability evaluation in industry, vol. 189, no. 194, pp. 4–7, 1996.

[83] A. Bangor, P. T. Kortum, and J. T. Miller, “An empirical evaluation of the system usability scale,” Intl. Journal of Human–Computer Interaction, vol. 24, no. 6, pp. 574–594, 2008.

85 86 A Code of Project

estimateHR()

1 function estimateHR(){

2 var aux_queue=[];

3 queue= queue.map(function(x){ returnx+1000;});//update time of previous peak occurance(they are now1sec late)

4 //The~ bitwise not operator returns false if -1 and true if positive.A perfect match for indexOf which returns the index0...n if found and -1 if not.

5 var pos= in2.indexOf(1023);//LOOKFORPEAK OCCURENCEs

7 while (~pos){

8 aux_queue.push(pos);//add peaks occurenceQUEUE

9 pos= in2.indexOf(1023, pos + 1);// use old position incremented to keep searching for peak occurence

10 console.log("peak found");

87 11

12 }

14 while(aux_queue.length>0){//insert peak occurence in adequate order to queue

15 queue.push(aux_queue.pop());

16 }

17 while(queue[0]>10000){//Remove peaks more than 10seconds ' old

18 queue.shift()

19 }

20 var diff=0;

21 for(vari= 0; i

22 diff+=queue[i]-queue[i+1]

23 }

24 //averages the distances and converts this ms value toBPMHR

25 curr_hr= Math.round(60000/(diff*1.0/(queue.length-1)))

26 if(hr_max

27 if(hr_min>curr_hr){ hr_min=curr_hr;}

28 data_HR.push(curr_hr);

29 return curr_hr;

31 }

88 B NeuroTech Artifacts

In this appendix, NeuroTech related work and collected feedback is presented.

B.1 NeuroTech Business Model Canvas

BMC details a developing, new business model as envisioned during the KickUP Sports in which we participated. Figure B.1 captions how the proto-company was envisioned. It is credited to IBEB associate investigator, Susana Navais Santos 1 who was responsible for the project’s application and its subsequent developments.

B.2 Gaming Tournament Conversation Feedback

Flavio,´ 21 anos Joga Hearthstone. O ”bichinho” que o faz jogar e´ o conv´ıvio e a adrenalina. Participa em torneios nacionais, joga competitivamente ha´ 6 meses, comec¸ou a jogar Hearthstone ha´ 1 ano e meio. E´ um jogo de estrategia´ com cartas virtuais que requer muita concentrac¸ao˜ e pensamento

1http://ibeb.ciencias.ulisboa.pt/pt/susana-novais-santos/

89 claro. Os jogadores pro temˆ apoio para ir a torneios internacionais. Estava a participar no seu 3º torneio offline, ja´ participou em +50 online. Os torneios internacionais costumam ter uma fee associada. A Blizzard (criadora do jogo) e´ quem faz a gestao˜ dos torneios mas tem sites parceiros que tambem´ podem organizar. Cada campeonato dura 3 meses de acordo com as estaçoes˜ ao longo do ano. Os jogadores ganham pontos numa ladder a n´ıvel europeu (existem outras para USA, Asia...).´ Quando começou o Flavio´ investiu 50C no jogo para comprar cartas e construir um deck mais forte. Joga 2-3 horas por dia, estuda ao mesmo tempo e costuma deitar-se pelas 2 da manha˜ e acordar as` 10 quando nao˜ tem aulas. Acha que e´ importante dormir bem. Como se prepara: pesquisa no Twitter o que os jogadores famosos estao˜ a fazer/dizer sobre o jogo, tem a ajuda de um amigo mais experiente que lhe da´ dicas, e veˆ os decks mais jogados. Durante o jogo: quando perde em jogos teoricamente faceis´ sente alguma raiva, mas ”tenta nao˜ se enervar, nao˜ pensar muito nisso”. Apos´ cada derrota veˆ onde errou e o que poderia ter feito melhor. Os jogos duram em media´ 10 a 12 minutos. Admite que e´ cansativo jogar de seguida, mas consegue perceber quando parar (começa a fazer erros no jogo - misplays). Para relaxar veˆ filmes, series´ ou joga FIFA. E´ nos´ torneios que sente maior pressao.˜ Como acha que poderia melhorar a sua performance: ter mais tempo para ver outros jogadores (streams). Talvez experimentasse meditaçao˜ mas ”nao˜ acredita muito nisso”. Acha que nao˜ precisa de feedback sobre o seu estado de cansaço porque ja´ percebe quando deve parar de jogar (apos´ 2-3 jogos a cometer erros). Diz que o wearable nao˜ iria incomodar.

Nuno, 35 anos Joga PES e alguns First Person Shooter (FPS) Entrou na GrowUP e comec¸ou por jogar Unreal Tournament. Entrava nos torneios pela diversao˜ e para conhecer pessoas novas. O mais importante para uma boa prestac¸ao˜ e´ o ambiente e o apoio da equipa.

Em casa sente menos pressao,˜ nos torneios ha´ mais ansiedade, sobretudo nos jogos mais im- portantes, mas diz que a ansiedade se controla com a experiencia.ˆ Fazer jogadas erradas causa alguma ansiedade mas para ultrapassar isso pensa que ”a seguir vai correr melhor”.

Meirinhas, 23 anos Utilizou o brainBIT enquanto jogava Hearthstone (perdeu 3-0) Diz que usar o wearable nao˜ inﬂuenciou o seu jogo mas sentiu-se mais observado.

90 91

Figure B.1: NeuroTech BMC