A Multi-Modal Device for Application in Microsleep Detection

A multi-modal device for application in microsleep detection Simon Knopp A thesis presented for the degree of Doctor of Philosophy in Electrical and Computer Engineering at the University of Canterbury, Christchurch, New Zealand. May 2015 ABSTRACT Microsleeps and other lapses of responsiveness can have severe, or even fatal, con- sequences for people who must maintain high levels of attention on monotonous tasks for long periods of time, e.g., commercial vehicle drivers, pilots, and air-traffic controllers. This thesis describes a head-mounted system which is the first proto- type in the process of creating a system that can detect (and possibly predict) these lapses in real time. The system consists of a wearable device which captures multiple physiological signals from the wearer and an extensible software framework for imple- menting signal processing algorithms. Proof-of-concept algorithms are implemented and used to demonstrate that the system can detect simulated microsleeps in real time. The device has three sensing modalities in order to get a better estimate of the user’s cognitive state than by any one alone. Firstly, it has 16 channels of EEG (8 currently in use) captured by 24-bit ADCs sampling at 250 Hz. The EEG is acquired by custom-built dry electrodes consisting of spring-loaded, gold-plated pins. Sec- ondly, the device has a miniature video camera mounted below one eye, providing 320 × 240 px greyscale video of the eye at 60 fps. The camera module includes infrared illumination so that it can operate in the dark. Thirdly, the device has a six-axis IMU to measure the orientation and movement of the head. These sensors are connected to a Gumstix computer-on-module which transmits the captured data to a remote computer via Wi-Fi. The device has a battery life of about 7.4 h. In addition to this hardware, software to receive and analyse data from the head- mounted device was developed. The software is built around a signal processing pipeline that has been designed to encapsulate a wide variety of signal processing algorithms; feature extractors calculate salient properties of the input data and a classifier fuses these features to determine the user’s cognitive state. A plug-in system is provided which allows users to write their own signal processing algorithms and to experiment with different combinations of feature extractors and classifiers. Because of this flexible modular design, the system could also be used for applications other than lapse detection—any application which monitors EEG, eye video, and head movement can be implemented by writing appropriate signal processing plug-ins, e.g., augmented cognition or passive BCIs. The software also provides the ability to iv ABSTRACT configure the device’s hardware, to save data to disk, and to monitor the system in real time. Plug-ins can be implemented in C++ or Python. A series of validation tests were carried out to confirm that the system operates as intended. Most of the measured parameters were within the expected ranges: EEG amplifier noise = 0.14 µVRMS input-referred, EEG pass band = DC to 47 Hz, camera focus = 2.4 lp/mm at 40 mm, and total latency < 100 ms. Some parameters were worse than expected but still sufficient for effective operation: EEG amplifier CMRR ≥ 82 dB, EEG cross-talk = −17.4 dB, and IMU sampling rate = 10 Hz. The contact impedance of the dry electrodes, measured to be several hundred kilohms, was too high to obtain clean EEG. Three small-scale experiments were done to test the performance of the device in operation on people. The first two demonstrated that the pupil localization algorithm produces PERCLOS values close to those from a manually-rated gold standard and is robust to changes in ambient light levels, iris colour, and the presence of glasses. The final experiment demonstrated that the system is capable of capturing all three physiological signals, transmitting them to the remote computer in real time, extracting features from each signal, and classifying simulated microsleeps from the extracted features. However, this test was successful only when using conventional wet EEG electrodes instead of the dry electrodes built into the device; it will be necessary to find replacement dry electrodes for the device to be useful. The device and associated software form a platform which other researchers can use to develop algorithms for lapse detection. This platform provides data capture hardware and abstracts away the low-level software details so that other researchers are free to focus solely on developing signal processing techniques. In this way, we hope to enable progress towards a practical real-time, real-world lapse detection system. Deputy Vice-Chancellor’s Office Postgraduate Office Co-Authorship Form This form is to accompany the submission of any thesis that contains research reported in co-authored work that has been published, accepted for publication, or submitted for publication. A copy of this form should be included for each co-authored work that is included in the thesis. Completed forms should be included at the front (after the thesis abstract) of each copy of the thesis submitted for examination and library deposit. Please indicate the chapter/section/pages of this thesis that are extracted from co- authored work and provide details of the publication or submission from the extract comes: Section 6.1.1 is adapted from a published conference paper: Knopp, S.J., Bones, P.J., Weddell, S.J., Innes, C.R.H. and Jones, R.D. (2012), ‘A miniature head-mounted camera for measuring eye closure’, In Proceedings of the 27th Conference on Image and Vision Computing New Zealand, IVCNZ ’12, ACM, pp. 313–318. Please detail the nature and extent (%) of contribution by the candidate: The PhD candidate did the research and wrote all sections of the published manuscript. The co-authoring supervisory team suggested research directions and offered criticisms which the candidate worked into the manuscript. Certification by co-authors If there is more than one co-author then a single co-author can sign on behalf of all. The undersigned certifies that: • The above statement correctly reflects the nature and extent of the PhD candidate’s contribution to this co-authored work • In cases where the candidate was the lead author of the co-authored work, he or she wrote the text. Name: Richard Jones Signature: Date: 30 April 2015 Deputy Vice-Chancellor’s Office Postgraduate Office Co-Authorship Form This form is to accompany the submission of any thesis that contains research reported in co-authored work that has been published, accepted for publication, or submitted for publication. A copy of this form should be included for each co-authored work that is included in the thesis. Completed forms should be included at the front (after the thesis abstract) of each copy of the thesis submitted for examination and library deposit. Please indicate the chapter/section/pages of this thesis that are extracted from co- authored work and provide details of the publication or submission from the extract comes: Section 6.1.2 is adapted from a published conference paper: Knopp, S.J., Bones, P.J., Weddell, S.J., Innes, C.R.H. and Jones, R.D. (2013), A ‘wearable device for measuring eye dynamics in real-world conditions’, In Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, pp. 6615–6618. Please detail the nature and extent (%) of contribution by the candidate: The PhD candidate did the research and wrote all sections of the published manuscript. The co-authoring supervisory team suggested research directions and offered criticisms which the candidate worked into the manuscript. Certification by co-authors If there is more than one co-author then a single co-author can sign on behalf of all. The undersigned certifies that: • The above statement correctly reflects the nature and extent of the PhD candidate’s contribution to this co-authored work • In cases where the candidate was the lead author of the co-authored work, he or she wrote the text. Name: Richard Jones Signature: Date: 30 April 2015 CONTENTS Abstract iii Acknowledgements xiii Preface xv Nomenclature xvii CHAPTER 1 INTRODUCTION 1 1.1 Motivation1 1.2 Terminology1 1.3 Objectives3 1.4 Structure4 CHAPTER 2 BACKGROUND 5 2.1 Signals for detecting microsleeps5 2.1.1 EEG5 2.1.1.1 Dry electrodes6 2.1.1.2 Commercial wireless EEG devices9 2.1.2 Eye movement 11 2.1.2.1 Remote video of face 11 2.1.2.2 Head-mounted video of eyes 11 2.1.2.3 Electrooculography 13 2.1.3 Head position/movement 14 2.1.4 Other physiological signals 14 2.1.5 Driver-specific signals 15 2.2 Signal analysis techniques 16 2.2.1 EEG 16 2.2.1.1 Measuring drowsiness 16 2.2.1.2 Detecting lapses & microsleeps 17 2.2.2 Eye video 18 2.2.2.1 Remote video of face 18 2.2.2.2 Head-mounted video of eyes 19 2.2.3 Multi-modal techniques 20 2.3 NeuroTech’s lapse research programme 21 2.3.1 Study 1: EEG 21 2.3.2 Study 2: EEG + fMRI 22 2.3.3 Study 3: EEG + fMRI with sleep restriction 22 x CONTENTS 2.3.4 Current and future research 23 2.4 Alerting techniques 24 2.5 Other cognitive monitoring applications 24 2.6 Summary 26 CHAPTER 3 THE ELAPSE PLATFORM 27 3.1 System design overview 27 3.2 Hardware 29 3.2.1 Base board 29 3.2.1.1 Processor 30 3.2.1.2 Power supplies 30 3.2.1.3 USB console 32 3.2.1.4 Inertial measurement unit 33 3.2.1.5 Daughter board interface 33 3.2.2 EEG module 33 3.2.2.1 Requirements 33 3.2.2.2 Implementation 34 3.2.2.3 Safety 36 3.2.3 Dry electrodes 37 3.2.3.1 Requirements

Load more