Towards cognitive brain-computer interfaces : real-time monitoring of visual processing and control using electroencephalography Antoine Gaume To cite this version: Antoine Gaume. Towards cognitive brain-computer interfaces : real-time monitoring of visual process- ing and control using electroencephalography. Cognitive Sciences. Université Pierre et Marie Curie - Paris VI, 2016. English. NNT : 2016PA066137. tel-01397304 HAL Id: tel-01397304 https://tel.archives-ouvertes.fr/tel-01397304 Submitted on 15 Nov 2016 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. i THÈSE DE DOCTORAT DE L’UNIVERSITÉ PIERRE ET MARIE CURIE Spécialité SCIENCES DE L’INGÉNIEUR École doctorale Informatique, Télécommunications et Électronique (Paris) Présentée par Antoine GAUME Pour l’obtention du grade de DOCTEUR DE L’UNIVERSITÉ PIERRE ET MARIE CURIE Sujet de la thèse : TOWARDS COGNITIVE BRAIN-COMPUTER INTERFACES: REAL-TIME MONITORING OF VISUAL PROCESSING AND CONTROL USING ELECTROENCEPHALOGRAPHY Soutenue le 10 juin 2016, devant un jury composé de : Mme. Pascale PIOLINO, Professeur, Université Paris Descartes, Rapporteur M. Jordi SOLÉ-CASALS, Maître de conférences, Universitat de Vic, Rapporteur M. Patrick GALLINARI, Professeur, Université Pierre et Marie Curie, Examinateur Mme. Marion TROUSSELARD, Médecin et Chercheur, IRBA, Examinateur M. Gérard DREYFUS, Professeur émérite, ESPCI ParisTech, Directeur de thèse M. François-Benoît VIALATTE, Maître de conférences, ESPCI ParisTech, Directeur de thèse Abstract Brain-computer interfaces (BCIs) offer alternative communication pathways between the brain and its environment. They can be used to replace a defective biological func- tion or to provide the user with new ways of interaction. Output BCIs, which are based on the reading of biological data, require the measurement of control signals as stable as possible in time and in the population. Identification and calibration of such signals are crucial steps in the conception of a BCI. The first part of this study focuses on BCIs using visual evoked potentials (VEPs) as control signals. A model is proposed to predict steady-state VEPs individually, i.e. to predict the response of a given subject’s brain to periodic visual stimulations. This model uses a linear summation of transient VEPs and an amplitude correction for quantitative prediction of the shape and spatial organization of the brain response to repeated stimulations. The simulated signals are then used as a basis of comparison for real-time decoding of electroencephalographic signals in a BCI. In the second part of this study, a paradigm is proposed for the development of cognitive BCIs, i.e. for the real-time measuring of high-level brain functions. The origi- nality of the paradigm lies in the fact that correlates of cognition are measured contin- uously, instead of being observed on discrete events. An experiment with the purpose of discriminating between several levels of sustained visual attention is proposed, with the ambition of real-time measurement for the development of neurofeedback sys- tems. 3 4 Résumé Les interfaces cerveau-machine (ICM) ouvrent des voies de communication alterna- tives entre le cerveau et son environnement. Elles peuvent être utilisées pour sup- planter une fonction biologique défaillante ou pour permettre de nouveaux modes d’interaction à l’utilisateur. Les ICM de sortie, dont le fonctionnement se base sur la lecture de données biologiques, nécessitent la mesure de signaux de contrôle stables dans le temps et dans la population. La recherche de tels signaux et leur calibration sont des étapes clefs dans la conception d’une ICM. Cette étude s’intéresse en premier lieu aux ICM utilisant les potentiels évoqués visuels comme signaux de contrôle. Un modèle est proposé pour la prédiction indi- viduelle de ces potentiels en régime permanent, c’est-à-dire lorsqu’ils sont issus d’une stimulation périodique. Ce modèle utilise une sommation linéaire corrigée en ampli- tude de la réponse à des stimulations visuelles discrètes pour prédire quantitativement la nature et la localisation spatiale de la réponse à des stimulations répétées. Les sig- naux modélisés sont ensuite utilisés en temps réel comme base de comparaison pour décoder les signaux électroencéphalographiques d’une ICM. Dans une deuxième partie, un paradigme est proposé pour le développement d’ICM cognitives, c’est-à-dire permettant la mesure de fonctions cérébrales de haut niveau. L’originalité du paradigme réside dans la volonté de mesurer la cognition en continu plutôt que son influence sur des événements discrets. Une expérience visant à discrim- iner différents états d’attention visuelle soutenue est proposée, avec l’ambition d’une mesure en temps réel pour le développement de systèmes de neurofeedback. 5 6 Acknowledgements First of all, I would like to express my sincere gratitude to my Ph.D. advisers, Dr. François- Benoît Vialatte and Prof. Gérard Dreyfus. Thank you for your unwavering support throughout my Ph.D. and for your trust, motivation, patience and knowledge. I also want to thank all the members of my Ph.D. committee for their interest in my research, and especially Dr. Jordi Solé-Casals and Prof. Pascale Piolino for their careful reading of my dissertation. Besides my advisers and committee, I would like to thank all the past and present members of the brain-computer interfaces team for the stimulating discussions, the fun we had and the wonderful cultural wealth you brought into the lab. In addition, I would like to give my special thanks to Dr. Pierre Roussel, who always kept his door open, and spent a lot of his time helping me any time I would step into his office. Even though this dissertation only deals with the research part of my Ph.D., I want to mention how grateful I am to Jérôme Coup, Yann Brunel and Benoit Corn for trusting me with the responsibility to teach their class at the Lycée Henri 4. It was a lot of work but a lot of fun and confirmed I could not pursue an academic career without teaching. Furthermore, I would like to thank my teachers at the Conservatoire wholeheart- edly. Thank you Florence Katz, Agnès Watson, Jae-Youn Park-Geiser and Emmanuèle Dubost-Bicalho for welcoming me into your classes and giving me the opportunity to learn a little bit of music while working on my Ph.D. These years were truly amazing. Last but not least, I would like to thank my family and friends for all their love and encouragement. For my parents who raised me to be curious about everything and supported me in all my pursuits. For my sister who helped me find the common ground between science and art. For my friends and among them especially my flat- mates Glen, Coco, Clem, Pierre, Gaïa, Thomas and Clara, who had to endure my tem- per in the harsh times of my Ph.D. Thank you. 7 8 TABLE OF CONTENTS Table of Contents Preamble 3 Abstract...........................................3 Résumé...........................................5 Acknowledgements....................................7 Table of Contents.....................................9 List of Figures....................................... 11 List of Tables........................................ 13 Acronyms......................................... 15 1 Introduction 19 1.1 What are we trying to do?............................. 20 1.2 Thesis overview................................... 21 1.3 List of publications................................. 21 2 Brain-Computer Interfaces: Connecting Brains with Machines 23 2.1 What is a brain-computer interface (BCI) ?................... 24 2.2 How does it work ?................................. 27 2.3 Examples of EEG-based BCIs........................... 39 2.4 Constraints...................................... 43 3 Methods of the Neural Interface Engineer 47 3.1 The nature of EEG signals............................. 48 3.2 Time-domain analysis............................... 49 3.3 Frequency-domain analysis............................ 51 3.4 Filtering EEG signals................................ 56 3.5 Machine Learning.................................. 61 4 Models and Networks of Attention 65 4.1 A history of attention modelling......................... 66 4.2 Integrative model of attention and executive control............. 74 4.3 Vocabulary of attention.............................. 79 4.4 Anatomy of attentional networks......................... 81 4.5 Some neurophysiological effects of attention.................. 86 9 TABLE OF CONTENTS 5 Modelling of steady-state activity from transient potentials 93 5.1 Introduction..................................... 93 5.2 Transient and steady-state visual evoked potentials.............. 94 5.3 Materials and methods............................... 95 5.4 Results........................................ 101 5.5 Discussion...................................... 109 6 Application of SSVEP Modelling to Brain-Computer Interfaces 111 6.1 Introduction..................................... 111 6.2 Materials and methods............................... 112 6.3 Results.......................................