NeuroExplore - Visualizing Brain Patterns A Physiological Computing InfoVis Daniel Jose´ dos Santos Rocha Thesis to obtain the Master of Science Degree in Information Systems and Computer Engineering Supervisors: Prof. Sandra Pereira Gama Prof. Hugo Alexandre Teixeira Duarte Ferreira Examination Committee Chairperson: Prof. Lu´ıs Manuel Antunes Veiga Supervisor: Prof. Sandra Pereira Gama Members of the Committee: Prof. Joao˜ Antonio´ Madeiras Pereira November 2017 Acknowledgments I would like to thank my parents for their friendship, encouragement and caring over all these years, for always being there for me through thick and thin and without whom this project would not be possible. I would also like to thank my sibblings, grandparents, aunts, uncles and cousins for their understanding and support throughout all these years. I would also like to acknowledge my dissertation supervisors Prof. Sandra Gama and Prof. Hugo Fer- reira for their constant technical insight, motivational support and overall enthusiastic sharing of knowl- edge that has made this Thesis possible. Last but not least, to all my friends and colleagues - particularly database participants: Antonio´ Amorim, Pedro Costa and Jose´ Fernandes - that helped me grow as a person and were always there for me during the good and bad times in my life. Thank you. To each and every one of you – Thank you. Abstract The objective of this project is the creation of a Physiological Computing Information Visualization (InfoVis) interface through which, interactively, users can visually decipher one’s intricate emotions and complex mind-state. To this end, we cooperated closely with Neuroscience experts from Instituto de Biof´ısica e Engen- haria Biomedica´ (IBEB) throughout our work. Consequently, we assembled a Brain Computer Inter- face (BCI), from a Bitalino Do It Yourself (DIY) hardware kit, for retrospective and real-time biosig- nals visualization alike. The resulting wearable biosensor was successfully deployed in an extensive Database (DB) acquisition process, consisting of activities with concrete, studied brain-pattern correla- tions. This big-data InfoVis foundation’s magnitude and its, at times saturated, physical signal accredited the development of a data-processing pipeline. Indeed, our solution - entitled NeuroExplore - converts and presents this large number of digitalized, raw biosignal items into more recognizable visual idioms. The system interaction was intentionally designed in order to augment users’ discoveries and rea- soning regarding visually recognizable metrics, as well as subsequently derived trends, outliers and other brain patters. Strengthening this intent, we adopted an iterative development process in which, recurrently, expert needs and user suggestions were equated as orienting guidelines. This all culminated in a final version we deemed worthy of extensive functional and utility user testing and expert validation. In the end, our project achieved both excellent user usability scores as well as expert interest, some already relying on our solution for their own research. Keywords InfoVis, Physiological Computing, Affective Computing, Brain-Computer Interface, biosignals, qEEG iii Resumo O objectivo deste projecto e´ a criac¸ao˜ de uma interface InfoVis para Computac¸ao˜ Fisiologica,´ atraves´ da qual, interactivamente, os utilizadores consigam decifrar visualmente, emoc¸oes˜ intr´ınsecas e com- plexos estado de mente. Para este fim, trabalhamos consistentemente em proximidade com especialistas em Neuroscien- cia do IBEB. Consequentemente, constru´ımos uma Brain Computer Interface (BCI), usando um kit de hardware DIY Bitalino, para a visualizac¸ao˜ retrospectiva ou em tempo-real de biosinais. O biosensor wearable resultante foi empregue, com sucesso, num extenso processo de aquisic¸ao˜ de base de dados, dividido em actividades concretas cujas correlac¸oes˜ com padroes˜ cerebrais foram estudadas. A magnitude destes dados - fundac¸ao˜ do nosso InfoVis - associada a` poss´ıvel saturac¸ao˜ destes jus- tifica a implementac¸ao˜ de preprocessamento. De facto, a nossa soluc¸ao˜ - NeuroExplore - filtra, deriva e apresenta este enorme numero´ de dados fisiologicos´ em idiomas visuais mais facilmente reconhec´ıveis. A interacc¸ao˜ com o sistema foi desenhada, intencionalmente, para aumentar as descobertas e racioc´ınio do utilizador - explicitamente - ao reconhecimento visual de metricas,´ bem como outros padroes˜ - subsequentemente - visualmente derivaveis.´ Fortalecendo esta intenc¸ao,˜ adoptamos um metodo´ iterativo de desenvolvimento onde, recorrentemente, equacionamos as necessidades de ex- perts bem como sugestoes˜ de utilizadores como linhas mestras. Tudo isto culminou num prototipo,´ considerado apto para uma extensa validac¸ao˜ de funcionalidade e utilidade por parte de utilizadores e experts. No final, alcanc¸amos´ notas de usabilidade excelentes bem como interesse por parte de especialistas, alguns dos quais ja´ utilizam a nossa soluc¸ao˜ para a sua propria´ investigac¸ao.˜ Palavras Chave Visualizac¸ao˜ de Informac¸ao,˜ Computac¸ao˜ Fisiologica,´ Computac¸ao˜ Afectiva, Interface Cerebro´ Com- putador, Biosinais, Eletroencefalografia Quantitativa v Contents 1 Introduction 1 1.1 Objectives.............................................5 1.2 Contributions...........................................5 1.2.1 KickUP Sports Accelerator...............................6 1.2.2 GrowUP Gaming recording sessions..........................6 1.2.3 Instituto de Apoio as` Pequenas e Medias´ Empresas e a` Inovac¸ao˜ (IAPMEI) Schol- arship...........................................7 1.3 Document Structure.......................................7 2 Background 9 2.1 Sentiment Analysis........................................ 11 2.2 Affective Computing....................................... 11 2.2.1 Affect Visualization.................................... 12 2.3 Physiological Computing..................................... 12 2.4 Brain Computer Interfaces (BCI)................................ 14 3 Related Work 17 3.1 Emotion’s Visual Representation................................ 19 3.2 Affective Computing....................................... 20 3.2.1 Text Analytics....................................... 20 3.2.2 Video Content Analysis................................. 26 3.2.3 Physiological Computing................................ 30 3.3 Discussion............................................ 35 4 Proposed Solution 37 4.1 Biosignal Input Solution..................................... 39 4.1.1 Device Configuration................................... 41 4.2 Architecture............................................ 42 4.2.1 Users........................................... 43 4.3 Development Process...................................... 44 vii 4.3.1 Biosignal Dataset Development............................. 44 4.3.1.A International Affective Picture System (IADS)................ 47 4.3.2 Data preprocessing................................... 47 4.3.2.A Saturation Filtering.............................. 48 4.3.2.B EEG Power Spectrum Calculus....................... 49 4.3.2.C Photopletismography (PPG) Peak-Finding Algorithm............ 49 4.3.3 Derived Data....................................... 50 4.3.3.A Brain-wave extraction............................. 50 4.3.3.B Heart-Rate metric............................... 51 4.3.3.C Emotional-Valence metric........................... 51 4.3.3.D Meditation metric............................... 51 4.3.3.E Engagement metric.............................. 52 4.3.4 InfoVis Development................................... 52 4.3.4.A Low Fidelity Prototype............................. 52 4.3.4.B High Fidelity Prototype............................ 53 4.3.4.C β Version InfoVis................................ 54 4.3.4.D Final InfoVis.................................. 55 4.4 Discussion............................................ 57 5 Evaluation 59 5.1 Context: European Investigators’ Night 2017......................... 61 5.2 Usability Tests.......................................... 62 5.2.1 Methodology....................................... 62 5.2.2 Results.......................................... 64 5.2.2.A Task Completion Times............................ 65 5.2.2.B Errors...................................... 67 5.2.2.C System Usability Scale (System Usability Scale (SUS)).......... 68 5.3 Case Studies........................................... 68 5.3.1 Methodology....................................... 70 5.3.2 Results.......................................... 70 5.3.2.A User 1..................................... 71 5.3.2.B User 2..................................... 72 5.4 Discussion............................................ 72 6 Conclusion and Future Work 75 6.1 Future Work............................................ 77 A Code of Project 87 viii B NeuroTech Artifacts 89 B.1 NeuroTech Business Model Canvas.............................. 89 B.2 Gaming Tournament Conversation Feedback......................... 89 ix x List of Figures 1.1 Cognitive Science’s comprising fields of study. Each line joining two disciplines represents interdisciplinary synergy [1]....................................3 2.1 Paul Ekman’s 6 categorical emotions: Happy, Sad, Fear, Anger, Suprised, Disgust.[2].. 12 2.2 Valence/Arousal Emotional Space [3].............................. 12 2.3 International 10–20 system. An internationally recognized method to describe and apply the location of scalp electrodes in the context of an EEG test or experiment.A = Ear lobe, C = central,