Development and Evaluation of an Online Transcranial Doppler Ultrasonographic Brain- Computer Interface for Communication

Home , Magnetoencephalography

Development and evaluation of an online Transcranial Doppler Ultrasonographic brain- computer interface for communication

Jie Lu

A thesis submitted in conformity with the requirements for the degree of Master of Applied Science

Institute of Biomaterials and Biomedical Engineering University of Toronto

Abstract

Development and evaluation of an online transcranial Doppler ultrasonographic brain-computer interface for communication

Jie Lu

Master of Applied Science

Graduate Department of Institute of Biomaterials and Biomedical Engineering University of Toronto

2013

We investigated an emerging brain-computer interface (BCI) modality, namely,

transcranial Doppler ultrasonography (TCD), which measures cerebral blood flow

velocity.

We hypothesized that a bilateral TCD-driven online BCI would be able to dichotomously classify a user’s intentions with at least 70% accuracy. To test this hypothesis, we had three objectives: (1) to develop a signal classifier that yielded high

(>80%) offline accuracies; (2) to develop an online TCD-BCI system with an onscreen keyboard; and, (3) to determine the achievable online accuracy with able-bodied participants.

With a weighted, forward feature selection and a Naïve Bayes classifier, sensitivity and specificity of 81.44 ± 8.35% and 82.30 ± 7.39%, respectively, were achieved in the

ii online differentiation of two mental tasks. The average information transfer rate and throughput of the system were 0.87 bits/min and 0.35 ± 0.18 characters/min, respectively. These promising online results encourage future testing of TCD-BCI systems with the target population.

iii Acknowledgments

First off, I would like to thank my wonderful and supportive parents, Yanping Lu and Aixue Jia, without whom I would not be where I am today. Also, great love goes to my younger sister, Luna Lu, who brings joy into my world.

I would like to thank Dr. Tom Chau for his brilliant guidance and supervision along the way. His passion and energy has truly been inspirational. Many thanks to my committee members, Dr. Michelle Keightley and Dr. Elaine Biddiss, and external examiner, Dr. Anne-Marie Guerguerian for their suggestions and comments.

Special thanks to Colleen Smith, Andrew Myrden, Larissa Schudlo, Dr. Saba Moghimi, Dr. Young Don Ko, and Dr. Khondaker Mamun for all of their kind help and wonderful company. I would also like to thank all members of the PRISM lab at Bloorview Research Institute for their support.

Finally, I would like to acknowledge the financial support of donors of the Kimel Family Graduate Student Scholarship in Paediatric Rehabilitation, Holland Bloorview Kids Rehabilitation Hospital, donors of the James F. Crothers Family Fellowships in Peripheral Nerve Damage, and the Institute of Biomaterials and Biomedical Engineering at the University of Toronto.

iv Table of Contents

Abstract ...... i

Acknowledgments...... iii

Table of Contents ...... iv

List of Figures ...... ix

List of Tables ...... xii

List of Appendices ...... xiii

Chapter 1. Introduction ...... 1

1.1 Motivation ...... 1

1.2 Research Question and Objectives...... 2

1.3 Overview ...... 2

Chapter 2. Background ...... 4

2.1 Brain-Computer Interfaces ...... 4

2.2 Existing BCI Modalities ...... 5

2.2.1 Intra-cortical BCIs ...... 5

2.2.2 Cortical BCIs ...... 5

2.2.3 Non-invasive BCIs ...... 5

2.3 Transcranial Doppler Sonography ...... 6

v 2.3.1 Measuring Blood Flow Velocities ...... 8

2.3.2 Hemodynamic Lateralization ...... 8

2.3.3 TCD-BCI Applications ...... 9

2.3.4 Ultrasound Probe Configuration ...... 9

2.4 Communication Systems ...... 10

2.4.1 Existing Virtual Keyboards ...... 10

2.4.2 Dynamic Keyboard ...... 10

2.5 Mental Tasks ...... 12

Chapter 3. Signal Processing and Classification Techniques to Optimize Transcranial Doppler Ultrasonography Performance ...... 14

3.1 Abstract ...... 14

3.2 Introduction ...... 15

3.3 Materials and Methods ...... 17

3.3.1 Participants ...... 17

3.3.2 Instrumentation ...... 17

3.3.3 Mental Tasks ...... 18

3.3.4 Experimental Protocol ...... 19

3.3.5 Pre-processing & Feature Extraction ...... 20

3.3.6 Feature Selection ...... 21

3.3.7 Classification ...... 23

vi 3.4 Results ...... 24

3.4.1 Linear Discriminant Analysis ...... 24

3.4.2 Naïve Bayes ...... 25

3.4.3 Features ...... 27

3.5 Discussion ...... 28

3.5.1 Classification of Word Repetition ...... 28

3.5.2 Feature Selection ...... 29

3.5.3 Classifier ...... 30

3.6 Conclusion ...... 30

Chapter 4. Online Transcranial Doppler Ultrasonographic Control of an Onscreen Keyboard ...... 31

4.1. Introduction ...... 32

4.2. Methods ...... 34

4.2.1. Participants ...... 34

4.2.2. Instrumentation ...... 34

4.2.3. Mental tasks ...... 35

4.2.4. Dynamic Keyboard ...... 36

4.2.5. Experimental Protocol ...... 38

4.2.6. Data Processing and Classification ...... 40

4.2.7. Performance evaluation ...... 41

vii 4.3. Results ...... 44

4.3.1. Feature Selection ...... 44

4.3.2. Inter-participant analysis ...... 45

4.3.3 Inter-session Results ...... 45

4.3.4 Dynamic Keyboard Output & User Feedback ...... 47

4.3.5 User Feedback ...... 49

4.4. Discussion ...... 50

4.4.1 Throughput of the Online TCD-based BCI Communication System ...... 50

4.4.2 Feature Selection ...... 50

4.4.3 Classification of Word Repetition ...... 51

4.4.4 Performance ...... 51

4.4.5 User Feedback Questionnaire ...... 52

4.4.6 Limitations ...... 53

4.5. Conclusion ...... 54

Chapter 5. Conclusions ...... 55

5.1 Contributions ...... 55

5.2 Future Work ...... 56

5.2.1 Evaluate User-Interface effects ...... 56

5.2.2 Evaluating the Effects of Practice on Performance ...... 57

viii 5.2.3 Testing in More Practical Situations ...... 57

5.2.4 Moving Towards the Target Population ...... 57

References ...... 58

Appendix ...... 69

Appendix I: Criteria Questionnaire ...... 69

Appendix II: Survey Template ...... 70

Appendix III: Post Study Questionnaire ...... 72

ix List of Figures

Figure 2.1. Application of BCI system [15] ...... 4

Figure 2.2. Lateral view of transtemporal insonation window [24] ...... 8

Figure 2.3. Axial view of the ultrasound probe directed toward the MCA [36] ...... 8

Figure 2.4. Dynamic keyboard user interface ...... 11

Figure 2.5. An example of dynamic keyboard slide progression ...... 11

Figure 3.1. Lateral and axial view of the ultrasound probe set at the transtemporal insonation window, directed toward the MCA [24,36] ...... 18

Figure 3.2. TCD dynamic feedback signal showing mean CBFV (averaged over every 1.3 seconds with CBFV sampling rate of 100Hz) in white ...... 19

Figure 3.3. Sample training block cues for the activation mental task (left) and the rest mental task (right), respectively ...... 19

Figure 3.4. Schematic diagram of a data collection block. The block began with a 1- minute baseline period, followed by 40 randomized task segments. During each task segment, the screen randomly displayed either an hourglass or a letter. If a letter was presented, the participant must perform the activation mental task for the duration of 15 seconds, followed by a 10 second rest task which was cued by an hourglass. If an hourglass was displayed, the participant continued the rest task...... 20

Figure 3.5. 10-fold cross-validation accuracy for Linear Discriminant Analysis classifier with Fisher criterion, ISF-SFS, and ISF-WSFS feature selection methods...... 25

x Figure 3.6. 10-fold cross-validation accuracy for Naïve Bayes classifier with Fisher criterion, ISF-SFS, and ISF-WSFS feature selection methods ...... 26

Figure 3.7. Fisher criterion Linear Discriminant Analysis (baseline method) accuracy versus WSFS Naïve Bayes (optimal method) accuracy...... 27

Figure 3.8. Average number of features selected by each feature selection and classification method ...... 29

Figure 3.9. Average accuracy from each feature selection and classification method ..... 29

Figure 4.1. Lateral and axial view of the ultrasound probe set at the transtemporal insonation window, directed toward the MCA [24,36] ...... 35

Figure 4.2. TCD dynamic feedback signal showing mean CBFV (averaged over every 1.3 seconds with CBFV sampling rate of 100Hz) in white ...... 36

Figure 4.3. Dynamic Keyboard User-interface ...... 37

Figure 4.4. An example of dynamic keyboard progression ...... 38

Figure 4.5. Schematic diagram of the training block. The training block began with a 1- minute baseline period, followed by 40 randomized task segments. During each task segment, the screen randomly displayed either an hourglass or a letter. If a letter was presented, the participant must perform the activation mental task for the duration of 15 seconds, followed by a 10 second rest task which was cued by an hourglass. If an hourglass was displayed, the participant continued the rest task...... 39

Figure 4.6. Sample recording depicting the three most common features: (1) difference

between left and right mean velocities, µL – µR, at 10-15 seconds; (2) difference between left and right mean velocities, µL – µR, at 5-10 seconds, and, (3) slope of the right MCA

CBFV (mR) at 5-10 seconds. Data shown are normalized and smoothed and represent one trial performed by participant 10. The left graph depicts a rest trial while the right graph

xi portrays an activation trial, showing the difference between left and right mean CBFV at 10 – 15 seconds and at 5-10 seconds...... 40

Figure 4.7. Normalized frequency of features (the number of times a feature has been selected divided by the total number of times all feature have been selected) across all participants ...... 45

Figure 4.8. Average throughput in characters/minute for sessions I, II, and III for all participants...... 47

Figure 4.9. Edit distances for test blocks from Sessions I (left plot), II (middle plot) and III (right plot). The horizontal line on each graph indicates an edit distance of 18 where no input was observed...... 47

Figure 4.10. User feedback of sessions I, II, and III over the 10 participants. Each column represents the question corresponding with the survey from Appendix II. Each row represents each participant...... 49

xii List of Tables

Table 3.1. Number of features selected by each feature selection method ...... 28

Table 4.1. Classification performance within individual sessions ...... 46

Table 4.2 Sample normalized edit distance with corresponding actual and intended outputs ...... 48

xiii List of Appendices

Appendix I: Criteria Questionnaire ...... 68

Appendix II: Survey Template ...... 69

Appendix III: Post Study Questionnaire ...... 71

xiv Chapter 1

Introduction

1.1 Motivation

Individuals who have lost the ability to speak or hear may rely on voluntary physical movements to communicate. However, for those who additionally have severe physical impairments, communication becomes extremely difficult. For such persons, communication and engagement with those around them can become the primary determinant of a meaningful quality of life [1]. Numerous types of communication devices have been developed. Some examples include: eye trackers, which locate the position of an individual’s eye; automatic speech recognition systems, which transcribe spoken language into readable text; motion recognition and gesture systems, which track body movements; and brain-computer interfaces (BCIs), which use physiological signals from the brain to determine user intent [2]. Of these assistive devices, only BCIs require no muscular activity from the user and thus appear to be particularly suited to individuals with the most severe impairments [3].

To date, many non-invasive BCI communication systems have been developed. These BCIs have traditionally used measurement modalities such as electroencephalography (EEG), magnetoencephalography (MEG), or functional magnetic resonance imaging (fMRI) to detect cognitive activity associated with user intent [4,5,6,7]. MEG and MRI systems involve sophisticated equipment that requires specialized personnel, building infrastructure and exorbitant cost. While EEG is widely used, clinical evidence of effectiveness is scant, particularly in pediatrics [8]. These limitations have precluded widespread clinical adoption of these modalities as access solutions [9].

In light of these shortcomings, we considered a recently introduced, non-invasive BCI modality ― transcranial Doppler ultrasonography (TCD). TCD is a medical imaging technique used to monitor cerebral blood flow velocity (CBFV) within the anterior, middle, or posterior cerebral arteries (ACA, MCA or PCA respectively) [10]. TCD is portable and affordable, and generally robust to electrical and magnetic noise [11]. In addition, several offline TCD-BCI studies over the past three years have demonstrated the decoding of selected mental tasks with accuracies often in excess of 80% [11,12,13].

1.2 Research Question and Objectives

This thesis addressed the following question: What level of performance (accuracy and throughput) can be achieved by a TCD-controlled on-screen keyboard with able-bodied participants? Based on recently published offline evidence, we hypothesized that an online TCD-BCI will be capable of detecting a user’s intention with at least 70% average specificity and sensitivity.

The objectives of this thesis were three fold, namely:

1. To determine a signal processing algorithm and classifier that would yield >80% accuracy offline when discriminating between an intuitive mental task (i.e., repetitive spelling and imagined writing of a target word) and a rest task (i.e., visual tracking of a signal on screen).

2. To develop an online BCI system by using TCD to control an on-screen keyboard.

3. To determine the achievable accuracy and throughput of the online BCI system for able-bodied participants.

1.3 Overview

This thesis includes five chapters, including this introductory chapter. Chapter 2 provides background information on existing BCI modalities, TCD, and communication systems.

Chapter 3 details the evaluation of different signal processing and classification techniques for bilateral TCD signal classification. Chapter 4 is a manuscript that focuses on the online testing of the TCD-BCI communication system. Finally, Chapter 5 summarizes the contributions of this thesis and proposes possible future work.

Chapter 2

Background

2.1 Brain-Computer Interfaces

For individuals who are cognitively aware but lacking an effective modality to convey intent, informing caregivers of personal needs can be an insurmountable challenge [14]. An access solution aims to address this challenge, and typically consists of an access technology paired with a user interface to translate some manifestation of intent (e.g., behavioral or physiological) into a functional activity, as depicted in Figure 2.1. A brain- computer interface is a particular type of access technology in which the access pathway is some brain imaging modality that provides physiological data relating from which we can infer changes in cognitive activity [3].

Figure 2.1. Application of BCI system [15]

2.2 Existing BCI Modalities

BCI modalities acquire specific physiological signals from the brain. These signals are then processed to produce control signals for computers or other devices. BCI modalities can be divided into two categories ― invasive and non-invasive modalities [9]. Invasive modalities can be further separated into intra-cortical and cortical BCIs depending on the location of electrodes [15]. Our study will focus on non-invasive BCIs.

2.2.1 Intra-cortical BCIs

Intra-cortical BCIs use signals that are recorded using electrodes directly implanted into the grey matter of the brain during neurosurgery. These signals include single neuron spike trains and extracellular local field potentials [9]. Because of their close contact with the brain tissue, invasive BCIs can produce the fastest and most reliable signals of all BCI devices [9]. However, implants can be high risk as they can cause brain tissue damage and the surgery itself can lead to infections [16]. In addition, the nature and significance of the long-term effects of implanted BCIs remain unclear and has yet to be rigorously tested and configured for clinical use [17].

2.2.2 Cortical BCIs

As the term suggests, cortical BCIs still require a craniotomy but electrodes are placed on the surface of the brain rather than inside the brain. Compared to intra-cortical implants, this approach reduces the risk of infection and the formation of scar tissue [18]. Since the electrodes are not placed within the brain tissue, the signal strength and recognition are weaker than that of invasive BCIs [19].

2.2.3 Non-invasive BCIs

Non-invasive BCIs rely on neuroimaging technologies that record signals from outside of the brain. Some technologies include EEG, MEG, fMRI, and near-infrared spectroscopy (NIRS). However, these modalities all have limitations that hinder their practical

implementation. EEG, for example, is the most commonly used BCI modality [20]. It uses the electrical activity of neurons to derive a control signal. However, EEG signals are easily contaminated by electrical and physiological artifacts such as electromyographic (EMG) activity from cranial muscles or electroculographic (EOG) activity that accompanies eye movement [15]. In addition, some EEG-BCIs also require extensive training [20]. MEG and fMRI on the other hand, measure magnetic fields induced by dendritic currents [21] and blood oxygen level-dependent signals [22], respectively. They both require extremely expensive instrumentation that are bulky and immovable [3]. In addition, these approaches require highly controlled environments that preclude integration in activities of daily living. NIRS is another BCI modality which harnesses changes in cerebral blood oxygenation levels to derive a control signal [23]. Similar to TCD, NIRS is a hemodynamic BCI and is still very early in its developmental stage as a BCI modality. However, NIRS is sensitive to ambient lighting and thus its use is often restricted to illumination-controlled environments. TCD has been proposed as a potential new BCI modality due to its desirable properties such as portability and insensitivity towards environmental noise (electrical, magnetic and optical) [11,12,13].

2.3 Transcranial Doppler Sonography

TCD was first developed in 1982 and has traditionally been used for clinical applications such as screening for sickle cell disease, detection of stenosis and occlusion, and detection of cerebral microemboli [24,25,26,27,28,29]. TCD works by emitting ultrasound waves towards a specific blood vessel and detecting the waves that are reflected by moving red blood cells within that vessel [24]. The difference between the emitted ultrasound wave frequency and the apparent frequency of the echoes can be used to calculate the speed of blood flow. The direction of blood flow can also be identified as well. If the echo frequency is higher than emitted frequency, blood is flowing towards the TCD probe. Conversely, a lower echo frequency indicates that the blood is flowing away from the TCD probe. Because TCD can be used to measure real-time changes in CBFV, functional TCD (fTCD) has been used to examine cerebral lateralization induced by various mental tasks and cognitive activities [30,31, 32,33,34]. From this point on within

this thesis, the acronym TCD will be used to mean fTCD. Over the past few years, TCD has been examined as a BCI modality through a variety of offline studies. In 2011, the inaugral TCD-BCI study conducted by Myrden, Kushki, Guerguerian & Chau used verbal fluency and spatial rotation tasks, activation durations of 45 seconds and simple classifiers to discriminate activity from unconstrained rest with 82.9±10.5% accuracy [11]. The authors subsequently extended classification to 3 classes, achieving greater than 70% accuracy and transmission rates up to 1.2 bits/minute [35]. Aleem & Chau (2013) were able to reduce the task period to 18 seconds while for the first time distinguishing between left and right CBFV lateralizations within a string of successive activations, managing accuracies of 74.6 ±12.6% [12]. Most recently, Faress & Chau (2013) reported an average accuracy of 76.1 ± 9.9% across participants in the automatic differentiation between pre- and post-verbal fluency elicited hemodynamic profiles [13].

Like other BCI modalities, TCD does have its limitations. For example, TCD's spatial resolution is limited because it only targets the main cerebral arteries (within the Circle of Willis). Skill and patience are often needed to properly position TCD probes. Similar to all metabolic BCIs, TCD requires a longer signal acquisition time than EEG and MEG due to the hemodynamic delay, i.e., the neurovascular time lag between neuronal firing accompanying the mental task and consequent CBFV changes. In addition, extensive physical movements may induce similar hemodynamic responses, leading to false detections. However, these spurious responses can usually be controlled by keeping the individual still.

In our study, we used a TCD instrument with two probes, one positioned on each side of the subject's skull. The precise position of each probe will depend on the location of the individual's transtemporal insonation window (Figure 2.2), a relatively thin area on eachlateral side of the skull. These windows provide a clear view of the ACA, MCA and PCA on each side of the brain. Of these arteries, we focussed on the MCAs, as they supply areas of the brain that are implicated in the mental activities of interest (e.g., spelling and motor imagery). Figure 2.3 demonstrates the orientation of the probe as it

measures CBFV within the left MCA. Together, the two probes can simultaneously monitor cerebral blood flow velocity in both the left and right MCAs. Transtemporal Middle Cerebral Window Artery

TCD Probe

Figure 2.2. Lateral view of transtemporal Figure 2.3. Axial view of the ultrasound probe insonation window [24] directed toward the MCA [36]

2.3.1 Measuring Blood Flow Velocities

ACA, MCA and PCA, along with many other arteries, can typically be measured through the transtemporal window [37]. The majority of the brain is perfused either directly or indirectly by the above three arteries, where each artery is usually responsible for supplying a specific part of the brain [38]. Therefore, these arteries are commonly imaged for the purpose of psychological and neuroscience research that aims to identify metabolically active regions associated with specific mental tasks [39]. The change of CBFV as a result of mental activity is known as neurovascular coupling. When a mental task is performed, the neural activity increases, leading to an increased regional demand in glucose and oxygen supply. In addition, the increased neural activity affects the local astrocytes to signal for vasodilation, which causes an increase in CBFV as well [40].

2.3.2 Hemodynamic Lateralization

Certain mental and physical tasks tend to activate specific brain regions in most individuals. The regional activation may not always be symmetrical between left and right sides of the brain, resulting in hemispheric lateralization [31]. If the active region is

perfused by the MCA, ACA, or PCA, the CBFV within the responsible artery will increase. TCD can be used to detect this change. The MCA is the largest of these arteries, and has been implicated in a wide variety of mental tasks [39]. For example, verbal fluency tasks (such as constructing as many words as possible starting with a specific letter) and verbal similarity tasks (such as deciding whether a pair of words is similar or different) usually induce left-hemispheric lateralization, i.e., CBFV increases more in the left MCA than the right MCA. On the other hand, visuospatial tasks (such as solving 3D puzzles) and visual matching tasks (such as visually searching for matching images) usually induce right-hemispheric lateralization [41].

2.3.3 TCD-BCI Applications

To date, TCD has been predominantly utilized for diagnosing and monitoring cerebral vasculature in a clinical setting. Functional TCD has also been used to observe changes in CBFV during cognitive, sensory, and motor tasks. The success of existing studies in depicting cognitive task-elicited CBFV changes has provided the fundamental groundwork for the development of TCD-BCI.

To date, there have been four published papers that report the success of offline TCD- BCI systems [11,12,13,35]. Each system improved upon the previous in terms of the number of features used and the trial-wise data acquisition duration while maintaining the average accuracy. However, an online TCD-BCI has yet to be reported.

2.3.4 Ultrasound Probe Configuration

Within this thesis, the TCD probes were placed on either side of the transtemporal windows. Both probes had the same power setting (spatial peak temporal average intensity) of 300mW/cm2, which is equivalent to a thermal cranial index (TIC) of 1.4 [42]. This setting is used to ensure signal quality while observing participant safety. The setting is also within the limit of 720 mW/cm2 as mandated by the FDA [43]. In addition, according to the British Medical Ultrasound Society, with TIC between 1.0 to 1.5, a maximum of 30 minutes continuous scanning time is acceptable [44].

The low-pass filter cutoff frequency and the sampling volume were set at 150Hz and 11mm, respectively, as in offline TCD studies [11,12,13,35]. The gain was set at either 31, 38, or 44 times, depending on the quality of the participant’s signal.

2.4 Communication Systems

There are many categories of potential user interfaces for BCI systems as illustrated in Figure 2.1. These user interfaces include augmentative and alternative communication (AAC) aids, environmental control units, and computers. In this thesis, we focused on augmentative and alternative communication, which often involves a virtual keyboard.

2.4.1 Existing Virtual Keyboards

Due to the extensive research on EEG BCI, most of the virtual keyboards developed and examined to date have been tailored for EEG or neural input based BCI [45]. For example, the most well-known and utilized keyboard is the P300 speller, which functions by quickly and randomly flashing the rows and columns of a letter matrix to detect the location of the desired letter through a time-locked event related potential. However, the scanning of rows and columns that is inherent in the design of the P300 speller would be very inefficient for a hemodynamic BCI. Sorger et al. proposed a communication system for real-time fMRI-based spelling. The system utilized a combination of BOLD signal location, signal onset delay, and signal duration to encode 27 characters [46]. However, this encoding system would require that participants learn a non-intuitive coding scheme for communication. To date, there is no established user interface for hemodynamic BCIs.

2.4.2 Dynamic Keyboard

In this thesis, we implemented the dynamic keyboard, an innovative communication program developed by researchers of the CanAssist group from the University of Victoria. Compared to other on-screen spellers such as Dasher and On-screen Keyboard, the dynamic keyboard offers higher accuracy and speed [47]. Figure 2.4 illustrates an example of the dynamic keyboard user interface. The software bins the letters of the

English alphabet according to their usage frequencies and presents different bins sequentially.

Figure 2.4. Dynamic keyboard user interface

A bin with a new set of characters is presented every 15 seconds. Figure 2.5 gives an example of the keyboard progression. If the bin containing the desired character is shown, the user can choose the bin by performing the mental activation task. Otherwise, the user can remain in the rest state. If a bin is selected, the user is presented with an undo option to immediately undo the selection, if needed. Otherwise, the user can bypass the undo slide and scan through individual characters within the selected bin. If a specific character is selected, a word prediction slide is shown to increase throughput.

Selection

No Selection

Selection

No Selection

Figure 2.5. An example of dynamic keyboard slide progression

The typed text can be output to computer programs such as Notepad++, Microsoft Office, or Internet Explorer.

We chose the dynamic keyboard because it minimizes the number of choices through which a user has to scan. As a result, the dynamic keyboard can potentially increase communication throughput (characters per minute). At the initiation of the thesis, the available implementation of the dynamic keyboard could only be operated through a computer mouse or a physical switch. We developed a new implementation that simplified the visual interface (i.e., only a single bin was visible at any point in time) and replaced the switch with a TCD-based activation so that no muscle movement would be required.

2.5 Mental Tasks

Numerous neuroscience journals have investigated various types of mental tasks that can elicit a lateralization in blood flow velocity. Previous offline TCD studies have utilized these mental tasks to elicit CBFV lateralization which act as input signals for the BCI system. Some examples include verbal fluency (thinking of as many words as one can within a time frame that starts with a specific letter), which elicits a left-lateralization (higher left MCA CBFV compared to that of the right side); mental rotation (mentally completing a 3D puzzle), which elicits a right-lateralization (higher right MCA CBFV compared to that of the left side) [39]. However, these tasks can be mentally demanding. Verbal fluency, for example, will also require the individual to have a sufficient vocabulary.

To simplify the activation mental, we have considered a new activation task which is word repetition. This task required the participant to repetitively spell the intended word while imagining writing the word throughout the task duration of 15 seconds (i.e. spelling task with associated motor imagery). This task was chosen because it is intuitive while having the potential to elicit a left-lateralized MCA CBFV. The motor imagery component of the activation task has been shown to activate the primary motor and

sensory cortex of the contralateral side [48,49]. Thus a right-hand related motor imagery is likely to elicit a left-sided CBFV lateralization resulting from neurovascular coupling with the activated areas. In addition, single word viewing has also been shown to elicit left lateralization in CBFV [50].

Chapter 3

Signal Processing and Classification Techniques to Optimize Transcranial Doppler Ultrasonography Performance

This chapter describes the data analysis and classification methods considered for an online Transcranial Doppler ultrasonography - brain computer interface system. It is written in paper format and thus may contain information that has been previously stated.

3.1 Abstract

Transcranial Doppler Ultrasonography (TC) is an emerging brain-computer interface (BCI) modality. Several offline studies have demonstrated algorithmic differentiation between two mental tasks with accuracies in excess of chance, but have used computationally sophisticated features and classifiers. A preferred approach for online implementation has not yet been identified. In this study, we conducted an offline analysis of TCD recordings to investigate the potential for increasing accuracy in a TCD- based BCI while adhering to features and classifiers computationally conducive to online implementation. Cerebral blood flow velocities from the left and right middle cerebral arteries were recorded from ten able-bodied participants during the performance of two different mental activities (word generation and visual tracking). Invoking a signal processing method from previous offline studies, we obtained an average accuracy of 73.32 ± 4.09%. We subsequently compared systematic feature selection approaches (Fisher criterion, sequential forward selection, weighted sequential forward selection) and two simple classifiers, namely, linear discriminant analysis (LDA) and a Naïve Bayes (NB) classifier. With the combination of weighted sequential forward selection, which

yielded less than a handful of time domain features, and a Naïve Bayes classifier, a maximum accuracy of 79.69 ± 3.20% was attained. Obtained with computationally simple features and classifier, this result sets the stage for the development of an online TCD-BCI.

3.2 Introduction

Individuals who are cognitively aware but living with severe motor disabilities such as muscular dystrophy, multiple sclerosis, high-level spinal cord injuries or locked-in syndrome may not be able to use conventional means of expression such as speech and gestures for communication. Brain-computer interface (BCI) systems offer an alternative means of communication for these individuals [15]. These BCI systems enable users to generate a control command through mental activity [15]. Many portable brain monitoring modalities have been explored for BCI development. The majority of systems have used electroencephalography (EEG) [51], but hemodynamic-based monitoring modalities such as near-infrared spectroscopy (NIRS) [5,6], and transcranial Doppler (TCD) ultrasonography systems [11] have recently been introduced as BCI alternatives. The cerebral hemodynamic response is inherently slower than the corresponding electrical response measured using EEG. However hemodynamic monitoring systems are not prone to electro-genic artifacts such as muscle contraction or eye-movement. Among the existing hemodynamic monitoring systems, TCD-based systems have recently demonstrated high accuracies in offline studies [11,12,13]. However, TCD has yet to be tested as an online BCI system, a necessary step to justify further exploration of TCD as a viable access pathway.

TCD is a non-invasive ultrasound technology that detects the changes in cerebral blood flow velocity (CBFV). It was first introduced as a medical imaging device in 1982, and has been widely applied clinically [24] for the detection of increased intracranial pressure in neurocritical care, evaluation of subarachnoid haemorrhage, detection of microembolism, and monitoring of cerebral circulation during cardiopulmonary bypass [25,52,53,54].

TCD has recently been used as a functional brain imaging tool to examine the effects of mental tasks on the blood flow velocities. In particular, functional TCD studies have focused on the middle cerebral arteries (MCAs), which perfuse 80% of the brain, and thus measurements of velocities therein reflect cognitive effort levels [39,41]. Blood flow lateralization elicited by mental tasks, such as verbal fluency and visuospatial tasks, has been detected using TCD in many studies [39,41,55,56,57]. The blood flow lateralization is due to the coupling between the cerebral blood flow and oxidative metabolism [58]. Because of the anatomy and the different functional areas of the brain, the left hemisphere exhibits augmented blood flow velocity during verbal fluency tasks while the right hemisphere demonstrates heightened activation during visuospatial tasks [41]. The response latency is rooted in an inherent physiological hemodynamic delay of 5-10 seconds between the onset of mental activation and the manifestation of blood flow changes [59,60]. Despite the slower response rate, recent functional TCD-BCI studies have reported promising rates of differentiation between different mental states [13,11,12]. Myrden et al. (2011) first introduced TCD as a BCI measurement modality and discriminated between word generation and rest (average accuracy of 82.9±10.5%) and between mental rotation and rest (85.7±10%) in 9 able-bodied adults using 45s task periods. The authors later followed up with a 3-class BCI, discerning among word generation, mental rotation and unconstrained rest with over 70% accuracy and reaching transmission rates of 1.2 bits per minute [35]. Subsequently, in a study of 18 adults, Aleem & Chau (2013) reduced the task period to 18s and classified successive left and right lateralizations offline in a user-independent framework with accuracies up to 74.6±12.6%. Most recently, in an offline TCD-NIRS-BCI study, Faress & Chau (2013) achieved an average accuracy of 76.1 ± 9.9% in the automatic differentiation between pre- and post-verbal fluency hemodynamics [13]. The collective evidence from offline studies supports the investigation of an online TCD-BCI [11,12,13].

In this chapter, we have considered an alternative mental task, repetitively spell the intended word while imagining writing the word. Additionally, the duration of the activity was reduced from a previous length of 18 seconds [12] to 15 seconds for throughput improvement. Through this chapter, we have investigated various signal

processing methods and classifiers to enhance the TCD-BCI functionality towards an online system.

3.3 Materials and Methods

3.3.1 Participants

Thirteen able-bodied participants with normal or corrected-to-normal vision were recruited for this study. One participant was excluded after the first session due to the inability to follow study protocol. A second participant was excluded upon disclosing post-study, medical history that violated inclusion criteria. A third participant was excluded due to inadequate transtemporal windows, which precluded the location of the MCAs. The ten remaining participants included for study (aged 18 – 40 years; all female), were all right-handed with no reported history of neurological, metabolic, respiratory, cardiovascular, or drug/alcohol-related conditions. All participants provided written informed consent. This study was approved by the research ethics board of both Holland Bloorview Kids Rehabilitation Hospital and the University of Toronto.

3.3.2 Instrumentation

The Doppler spectra of blood flow through the left and right MCAs were monitored using the MultiDop X-4 TCD (Compumedics Germany) and the accompanying bilateral headgear with two fixed 2 MHz ultrasonic transducers. The data were recorded at a sampling frequency of 100Hz. The probes were positioned over the transtemporal insonation window according to an established insonation procedure [61] as seen in Figure 3.1.

Transtemporal Middle Cerebral Window Artery

TCD Probe

Figure 3.1. Lateral and axial view of the ultrasound probe set at the transtemporal insonation window, directed toward the MCA [24,36]

Ultrasound gel was applied between the probe and the user’s skin to ensure proper signal transduction. Once the probe was placed onto the transtemporal window, the TCD was turned on with an initial depth setting of 50mm. The insonation angle and depth were then adjusted to find the bifurcation of the internal carotid artery into the middle cerebral artery (blood flowing toward the probe) and the anterior cerebral artery (blood flowing away from the probe). The insonation depth was then decreased until the maximum unidirectional flow towards the probe was detected. All participants were given 5 minute breaks per every 15 minutes of TCD usage to provide sufficient time for probe cooling. Throughout the recording process, the thermal cranial index (TIC) of the probes did not exceed 1.5, thus avoiding discomfort or thermal injury to the participants, which is in accordance with the British Medical Ultrasound Society safety guidelines [62].

The MultiDop X-4TCD device was approved by Health Canada’s Medical Devices Directorate for investigational testing.

3.3.3 Mental Tasks

Participants performed two mental tasks throughout the study (i.e. activation and rest). The word repetition was used as an activation mental task and visual tracking was used as a rest state. Word repetition required the participant to repetitively spell the intended

word while imagining writing the word throughout the task duration of 15 seconds (i.e. spelling task with associated motor imagery). Visual tracking required the participant to attend to the time-evolving strip chart (Figure 3.2) of the TCD signal.

Figure 3.2. TCD dynamic feedback signal showing mean CBFV (averaged over every 1.3 seconds with CBFV sampling rate of 100Hz) in white

For the word repetition task, participants were presented with either a single letter or multiple letters forming part of a word (Figure 3.3.). Upon seeing this cue, participants repetitively rehearsed the spelling of the word while imagining writing the word with their right hand. During the visual tracking task, an hourglass was presented to the participants (Figure 3.3.), at which point participants shifted their gaze to the TCD feedback (Figure 3.2) and visually followed the signal while trying to maintain their blood flow velocities as stable as possible. Both tasks were completed without any vocalization to avoid an increase of blood flow due to speech.

Figure 3.3. Sample training block cues for the activation mental task (left) and the rest mental task (right), respectively 3.3.4 Experimental Protocol

Each participant completed three sessions. The first session involved two blocks while subsequent sessions contained one block each. A one minute baseline recoding was obtained before each block for the purpose of normalizing data collected from the block. A five minute rest period was offered between blocks.

For each block, the participants performed a total of forty task segments. Each segment was either an activation task or a rest task. The sequence of task presentation was randomized (Figure 3.4). A ten second recovery period was included after each activation task to allow the participant’s blood flow velocities to return to baseline levels. During the recovery period, the participants performed the rest task to restore basal blood flow velocities. For each mental task, specific cues were presented to the participants for the task duration.

1 minute 13 minutes 20 seconds

Baseline Segment 1 Segment 2 Segment 3 Segment 39 Segment 40

15 s 10 s

or 15 s

Figure 3.4. Schematic diagram of a data collection block. The block began with a 1-minute baseline period, followed by 40 randomized task segments. During each task segment, the screen randomly displayed either an hourglass or a letter. If a letter was presented, the participant must perform the activation mental task for the duration of 15 seconds, followed by a 10 second rest task which was cued by an hourglass. If an hourglass was displayed, the participant continued the rest task. 3.3.5 Pre-processing & Feature Extraction

All data collected from the blocks were used for classifier training. Therefore, for each participant, a total of eighty activation data segments and eighty rest data segments were used to train a user-specific classifier. A total of forty-four features were extracted from each segment. The features were chosen according to previous TCD studies which have investigated lateralized activation tasks [11].

3.3.6 Feature Selection

Three feature selection methods were investigated, including Fisher criterion, sequential forward selection (SFS), weighted sequential forward selection (WSFS).

The Fisher criterion [63] was based on previous offline TCD-BCI studies and thus was considered as the baseline standard [11]. For a given feature, this criterion can be expressed as:

2 − mm J = 10 2 + ss 2 0 1 (1)

where 0and 0 represent the mean and standard deviation of the feature values extracted

from TCD𝑚𝑚 recordings𝑠𝑠 during the resting task, while 1 and 1 are the mean and standard deviation of the same features during the activation task𝑚𝑚 . The𝑠𝑠 value of the Fisher criterion increases as the average separation between groups increases and the average separation within group decreases. This criterion thus yields the features that provide the maximum separability between rest and activation tasks. As in a previous TCD-BCI study, we retained only the top 3 features identified by the Fisher criterion [11].

For the second and third feature selection methods, we used the interclass separability F- score (ISF) as the feature ranking method. The ISF is a normalized measure of discrimination between the features of two classes [64]. The ISF of a feature can be defined as:

2 2 1 0 −+− mmmm )()( ISF = 1 n1 1 n0 2 +− − 2 ∑ = i mx 11, )( ∑ = i mx 00, )( n −1 i 1 n −1 i 1 1 0 (2)

where n1 and n0 represent the number of instances from the activation and rests, task respectively; m , m1 and m0 , are the average feature values for the whole data set, the th activation data, and the rest data, respectively; xi 1, is the i activation feature value, and

th xi 0, is the i rest feature value. Therefore, the numerator is an indication of the separability between the activation and rest class feature values and the denominator reflects the intraclass variability of the feature. The larger the ISF, the more likely the feature under consideration will be discriminative.

The sequential feature selection (SFS) approach consists of a forward step which starts from an initially empty set of features Z0. At each forward step l, a new subset is created + containing all features within the previous subset Z(l-1). In addition, feature a is added to + the new subset Zl, where a is the next available feature from the set of ranked and ordered features [65]. Therefore, the first subset only contains the top ranked feature from the ISF ranking and the last subset contains all 44 extracted features. The number of subsets created is equal to the number of available features. Within this study, we performed a 10-fold cross-validation. For each fold, an optimal subset was obtained according to the validation accuracies of the subsets. The 10 optimal subsets were then collected. A new 10-fold cross-validation was performed with this superset of optimal features and a final set of features was selected as that boasting the highest validation accuracy. These features were utilized to train the final classifier.

The third approach is weighted sequential feature selection (WSFS). WSFS extended the sequential forward search (SFS) approach [65] by explicitly considering feature contributions (i.e. the number of times a specific feature was chosen). For each fold of a 10-fold cross-validation for feature selection, all features were first ranked according to F-score [64] for interclass separability, using the training set. These ranked features were then organized into cumulative subsets such that the first subset contained the top ranked feature, the second subset contained the top two ranked features, and so on. The last subset contained all features. Within each fold, the subset with the highest validation accuracy was selected. Therefore, 10-fold cross-validation yielded 10 such subsets.

3.3.7 Classification

Two classification methods were compared. The first was a Linear Discriminant Analysis and the second, a Naïve Bayes classifier.

Previous offline TCD studies implemented Fisher criterion feature selection and Linear Discriminant Analysis (LDA) classifier methods [11,12,13]. Thus within this section, we have established the baseline accuracy for our TCD data set through LDA:

T )( += wxwxy 0 (3)

Where w is the weight vector for the feature vector x, and w0 is the bias [66]. Not all 44 features were included in the feature vector, x. Within each session, appropriate features were selected according to the methods introduced above. The LDA classified each testing data point as rest when y≤0, and activation when y>0.

From existing studies involving hemodynamic data, the Naïve Bayes (NB) classifier has performed well with a modest feature vector dimensionality [67]. The NB classifier assumes independence between features within each class. However, NB appears to work well in practice even when that independence assumption is not strictly valid [68]. Therefore, we have implemented the NB classifier, assuming independence between features. To decide whether a specific test data belongs to either the rest class or the activation class, the posterior possibilities (Ck) of both classes were calculated. The test

data were assigned to the class with the highest posterior probability. Ck, given by:

k = k ∏ j CxpCpC k )()( j (4)

where Cp k )( is the prior probability for the rest or activation class (k = , 0 or 1 respectively). Because there were equal numbers of rest and activation instances (80 rest, == 80 activation), we have set 0 CpCp 1 5.0)()( . The likelihood, Cxp kj )( , is the class- conditional probabilities for each feature within either or rest activation classes. A

Gaussian distribution was assumed for each feature. Thus for a set of selected features ( ) of the test data, the likelihood of that test data belonging to class k is given as:

𝑋𝑋   − µ   1 1 x ,kjj 2  k = CxpCXp kj )()( =  exp− )(  ∏∏ 2 j j  2πσ  2 σ ,kj   ,kj   (5)

Where µ ,kj and σ ,kj are the mean and variance respectively of the jth feature in class k. Both the mean and the variance were calculated from the training data and the same calculation was done for all features for both classes.

The Fisher criterion, ISF-SFS and ISF-WSFS were compared as feature selection methods for both LDA and NB classifiers.

3.4 Results

3.4.1 Linear Discriminant Analysis

Data from 10 participants were individually analyzed. Each set contained 80 activation and 80 rest instances. A 10-fold cross validation method was applied to each data set so that all data points were randomly arranged into 10 groups. Each group of data contained 8 activation instances and 8 rest instances. For each fold, one group was set aside as testing data while the rest were used as training data. In each fold, a different group was chosen as testing data while the other 9 groups were used as training data. This procedure was repeated until each group participated once as testing data. The 10-fold cross validation method was efficient for small samples of data as it allowed all the data points to contribute as both testing and training data. Additionally, it prevented over-fitting by ensuring that the training and testing data were different within each fold [66].

The performance of the feature selection methods and the classifier was defined by the overall session accuracy. The accuracy for each session was calculated from the averaged accuracies of the 10-fold cross validation. Fisher criterion, SFS, and WSFS resulted in an

average validation accuracy of 73.32 ± 4.09%, 78.69 ± 3.71% and 78.94 ± 3.57%, respectively, using the LDA classifier (Figure 3.5).

Fisher LDA 70 SFS LDA

Accuracy (%) Accuracy 65 WSFS LDA

50 1 2 3 4 5 6 7 8 9 10 Participant Number

Figure 3.5. 10-fold cross-validation accuracy for Linear Discriminant Analysis classifier with Fisher criterion, ISF-SFS, and ISF-WSFS feature selection methods. 3.4.2 Naïve Bayes

The accuracy for each participant using the different feature selection and Naïve Bayes classifier combinations is summarized in Figure 3.6.

75 Fisher NB 70 SFS NB

65 WSFS NB Accuracy (%) Accuracy

50 1 2 3 4 5 6 7 8 9 10 Participant Number

Figure 3.6. 10-fold cross-validation accuracy for Naïve Bayes classifier with Fisher criterion, ISF- SFS, and ISF-WSFS feature selection methods

The Fisher criterion, SFS, and WSFS resulted in an average validation accuracy of 74.56 ± 3.24%, 79.13± 3.85% and 79.69 ± 3.20% respectively. WSFS feature selection yielded the highest accuracy for the NB classifier. Compared to the baseline method (Fisher criterion with LDA classifier), WSFS-NB method yielded higher accuracies across all participants (p<0.001; paired t-test) (Figure 3.7).

70 Fisher LDA WSFS NB 65 Accuracy (%) Accuracy

50 1 2 3 4 5 6 7 8 9 10 Participant Number

Figure 3.7. Fisher criterion Linear Discriminant Analysis (baseline method) accuracy versus WSFS Naïve Bayes (optimal method) accuracy.

The increase in accuracy was especially noticeable for participants who had lower accuracies with the baseline method, namely participants 4, 9 and 10.

3.4.3 Features

The Fisher feature selection method required a set number of three features for classifier training. However, both SFS and WSFS method did not restrict the number of features selected. Overall, SFS and WSFS method selected an average of 5.1 ± 3.0 and 3.2 ± 1.5 features, respectively, for the LDA classifier. For the Naïve Bayes classifier, the SFS and WSFS method selected an average of 4.6 ± 3.5 and 3.0 ± 1.3 features respectively. The detailed number of features selected for each participant is shown in Table 3.1.

Table 3.1. Number of features selected by each feature selection method

SFS SFS WSFS WSFS Participant LDA NB LDA NB 1 6 5 6 5 2 12 14 3 3 3 7 5 1 1 4 3 5 4 2 5 4 2 4 3 6 2 2 1 2 7 2 2 3 2 8 6 3 3 3 9 3 4 3 4 10 6 4 4 5

3.5 Discussion

3.5.1 Classification of Word Repetition

A previous offline TCD-BCI study conducted by Myrden et al. achieved 82.9 ± 10.5% accuracy using a 45 second data collection window, differentiating between verbal fluency mental task and relaxation [11]. A subsequent offline TCD-BCI study conducted by Aleem et al. was able to achieve 80.0 ± 9.6% accuracy using an 18 second data collection window, differentiating between verbal fluency and mental rotation mental tasks [12]. More recently, Faress & Chau achieved an average accuracy of 76.1 ± 9.9% in the automatic differentiation between pre- and post-verbal fluency hemodynamics with a 20 second data collection window [13]. Comparatively, we achieved an average accuracy of 79.69 ± 3.20% with a 15 second window. The chance level for this offline data is 60%, as calculated by Muller-Putz et al. [69] . This offline result exceeds the chance level and compares favorably to findings of previous studies, while considering the shortest data collection window to date and a combination of computationally efficient features and classifier.

3.5.2 Feature Selection

The average number of features used for all methods exceeded the set number of 3 features, suggesting that more than 3 features are needed to achieve class discrimination. Given that ten-fold cross-validation was used, where the training and validation data were mutually exclusive subsets of data, we can be assured that the higher accuracy was not due to overfitting.

Overall, the WSFS method minimized the number of features for each participant as seen in Figure 3.8 while maintaining cross-validation accuracies comparable to those of the SFS method, as seen in Figure 3.9. Therefore, the WSFS feature selection method seemed to eliminate redundant or uninformative features.

8 84.00

7 82.00 6 80.00 5 78.00 4 76.00 3

2 74.00 Number ofFeatures

1 Average Accuracy (%) 72.00

0 70.00 SFS LDA SFS NB WSFS WSFS SFS LDA SFS NB WSFS WSFS LDA NB LDA NB Methods Methods Figure 3.8. Average number of features Figure 3.9. Average accuracy from each selected by each feature selection and feature selection and classification method classification method There was significant difference between the Fisher criterion and SFS feature selection methods with both LDA and NB classifiers (p < 0.001; p = 0.002; paired t-test). Likewise, accuracies significantly differed between Fisher criterion and WSFS feature selection methods with both LDA and NB classifiers (p < 0.001; p < 0.001; paired t-test).

The WSFS and SFS feature selection methods offered similar accuracies with either

classifier (pLDA = 0.401; pNB = 0.134; paired t-test).

3.5.3 Classifier

When keeping the feature selection methods the same, there was no significant difference between the LDA and NB classifiers. However, when taking into account a combination of feature selection methods and classifiers, there were significant differences between both SFS-NB and WSFS-NB methods and the standard Fisher criterion LDA method (p<0.001; p<0.001; paired t-test). In addition, WSFS-NB was significantly different from SFS-LDA (p = 0.026), though SFS-NB was not significantly different from WSFS-LDA (p=0.627).

When comparing the WSFS-NB method with the Fisher-LDA method, the largest improvement in classification was seen in participants 9 and 10 (over 8% increase). When comparing the same participants with SFS-NB method and SFS-LDA method, the improvements were only 1.25% and 4.38% respectively. Collectively, these findings suggest that it was the judicious combination of feature selection method and classifier that improved accuracies, rather than either method alone.

3.6 Conclusion

In this chapter, we compared different feature selection methods and classifiers for a set of TCD data. We demonstrated that although all methods were able to reasonably differentiate the TCD data into activation and rest states, a Naïve Bayes classifier with weighted sequential forward feature selection achieved the best performance. Our feature selection and classifier methods improved upon previously published work by further reducing the data analytic window (to 15s) along with the post-activation window (to 10s) and honing in on algorithms that are conducive to online implementation.

Chapter 4

Online Transcranial Doppler Ultrasonographic Control of an Onscreen Keyboard

This chapter is a manuscript and thus contain information already presented in Chapter 2. In addition, the introduction and method part of this chapter has been described within Chapter 3 and may be repetitive material.

Abstract. Brain-computer interface (BCI) systems exploit brain activity for generating a control command and may be used by individuals with severe motor disabilities as an alternative means of communication. Transcranial Doppler Ultrasonography (TCD) is an emerging brain monitoring modality for BCI development. However, current studies have exclusively used offline analysis. The feasibility of a TCD-based BCI system hinges on its online performance, which has not been documented to date. In this paper, an online TCD BCI system was implemented for the control of a scanning keyboard. Target letters were selected by repetitively rehearsing the spelling of the intended word, while letters were bypassed by performing a visual tracking task. With 10 able-bodied young adults, these two mental tasks were differentiated using a Naïve Bayes classification algorithm and a set of time-domain user-dependent features. The system achieved an average specificity and sensitivity of 81.44 ± 8.35% and 82.30 ± 7.39%, respectively. The level of agreement between the intended and machine-predicted selections was moderate (κ=0.60). The average information transfer rate was 0.87 bits/min with an average throughput of 0.35 ± 0.18 character/min. These results suggest that future studies of online TCD-based BCI systems are warranted.

4.1. Introduction

lateralization elicited by mental tasks, such as verbal fluency and visuospatial tasks, has been detected using TCD in many studies [39,41,55,56,57]. The blood flow lateralization is due to the coupling between the cerebral blood flow and oxidative metabolism [58]. Because of the anatomy and the different functional areas of the brain, the left hemisphere exhibits augmented blood flow velocity during verbal fluency tasks while the right hemisphere demonstrates heightened activation during visuospatial tasks [41]. The response latency is rooted in an inherent physiological hemodynamic delay of 5-10 seconds between the onset of mental activation and the manifestation of blood flow changes [59,60]. Despite the slower response rate, recent functional TCD-BCI studies have reported promising rates of differentiation between different mental states [11,12,13]. Myrden et al. (2011) first introduced TCD as a BCI measurement modality and discriminated between word generation and rest (average accuracy of 82.9±10.5%) and between mental rotation and rest (85.7±10%) in 9 able-bodied adults using 45s task periods. The authors later followed up with a 3-class BCI, discerning among word generation, mental rotation and unconstrained rest with over 70% accuracy and reaching transmission rates of 1.2 bits per minute [35]. Subsequently, in a study of 18 adults, Aleem & Chau (2013) reduced the task period to 18s and classified successive left and right lateralizations offline in a user-independent framework with accuracies up to 74.6±12.6%. Most recently, in an offline TCD-NIRS-BCI study, Faress & Chau (2013) achieved an average accuracy of 76.1 ± 9.9% in the automatic differentiation between pre- and post-verbal fluency hemodynamics [13]. The collective evidence from offline studies supports the investigation of an online TCD-BCI [11,12,13].

In the current study, an online TCD-based BCI system was implemented for operating a spelling system (i.e. scanning keyboard). The spelling interface was controlled via two mental states, namely, rest and activation. The activation task was repetitive mental spelling of the intended word and the rest mental task was the visual tracking of a display of TCD signals.

4.2. Methods

4.2.1. Participants

4.2.2. Instrumentation

Transtemporal Middle Cerebral Window Artery

TCD Probe

Figure 4.1. Lateral and axial view of the ultrasound probe set at the transtemporal insonation window, directed toward the MCA [24,36]

The MultiDop X-4TCD device was approved by Health Canada’s Medical Devices Directorate for investigational testing.

4.2.3. Mental tasks

Figure 4.2. TCD dynamic feedback signal showing mean CBFV (averaged over every 1.3 seconds with CBFV sampling rate of 100Hz) in white

For the word repetition task during the training session, participants were presented with either a single letter or multiple letters forming part of a word. Upon seeing this cue, participants repetitively rehearsed the spelling of the word while imagining writing the word with their right hand. During the visual tracking task, an hourglass was presented to the participants, at which point participants shifted their gaze to the TCD feedback (Figure 4.2) and visually followed the signal while trying to maintain their blood flow velocities as stable as possible. Both tasks were completed without any vocalization to avoid an increase of blood flow due to speech.

During the testing sessions, the participants were asked to perform the activation mental task when the desired letter appeared among the currently available letter choices and to perform the rest mental task when the desired letter was not displayed.

4.2.4. Dynamic Keyboard

A custom on-screen keyboard was developed based on the concept of the dynamic keyboard developed by the University of Victoria. In our implementation, each level of the keyboard hierarchy contained multiple bins, although at any given time, only one bin was displayed to minimize the mental workload and user confusion. Each bin contained multiple letters or words (Figure 4.3).

Output Box

Current bin

TCD signal

Figure 4.3. Dynamic Keyboard User-interface

The letters were grouped together on the basis of letter frequency given existing letter outputs. For example, the top 5 beginning of word letters (t, a, i, s, o) were grouped into the first letter bin only if there had been no letter output. However, if the letter “t” was selected, the first letter bin would contain letters “h, o, r, a, e” instead (Figure 4.4). The words were grouped using word prediction. For every letter bin selected, a bin containing high frequency words based on the existing letter outputs and previous letter bin selection was presented. To select the intended letter or word within a bin, participants had to choose the bin by performing the activation mental task. Once the bin was chosen, an undo button was available to the participant. This button allowed the user to delete the most recent selection and return to the bin from the previous level of the hierarchy. If the participant did not choose to delete the most recent selection, the individual letters or words within the above selected bin were presented one at a time to the participant (Figure 4.4). This latter selection would be at the second level of the keyboard hierarchy. If no selections were made on a given level of the hierarchy, the procedure would restart from the beginning of the previous level.

Selection

No Selection

Selection

No Selection No Selection

Selection

No Selection No Selection

Figure 4.4. An example of dynamic keyboard progression

4.2.5. Experimental Protocol

Each participant completed three sessions. The first session involved two training blocks and one testing block while subsequent sessions contained one training block followed by two testing blocks. A one minute baseline recoding was obtained before each block for the purpose of normalizing data collected from the block. A five minute rest period was offered between blocks.

For each training block, the participants performed a total of forty task segments. Each segment was either an activation task or a rest task. The sequence of task presentation was randomized (Figure 4.5). A ten second recovery period was included after each activation task to allow the participant’s blood flow velocities to return to baseline levels.

During the recovery period, the participants performed the rest task to restore basal blood flow velocities. For each mental task, specific cues were presented to the participants for the task duration. For the first session, participants had a ten minute break while the two blocks of training data were used to train the appropriate classifier. For sessions two and three, the ten minute break occurred after the first training block. During this break, the classifier was trained with data from the current and initial sessions. After each session, the participants' feedback on tiredness levels were collected through a written survey.

1 minute 13 minutes 20 seconds

Baseline Segment 1 Segment 2 Segment 3 Segment 39 Segment 40

15 s 10 s

or 15 s

Figure 4.5. Schematic diagram of the training block. The training block began with a 1-minute baseline period, followed by 40 randomized task segments. During each task segment, the screen randomly displayed either an hourglass or a letter. If a letter was presented, the participant must perform the activation mental task for the duration of 15 seconds, followed by a 10 second rest task which was cued by an hourglass. If an hourglass was displayed, the participant continued the rest task.

For each testing block, the participants were asked to spell a given target phrase to the best of their abilities using the dynamic keyboard. The participants performed the activation task only when the bin containing the intended selection was presented. If a false positive occurred, the participants were instructed to select the undo button and correct the error before continuing the spelling process. If a false-negative occurred, the participants were instructed to simply wait until the keyboard looped back to the intended bin.

4.2.6. Data Processing and Classification

All data collected from the training blocks were used for classifier training. Therefore, for each participant, a total of forty activation data segments and forty rest data segments were used to train a user-specific classifier for session I. For each subsequent session, training data from session I (40 activation and 40 rest segments) and the session at hand (20 activation and 20 rest segments) were pooled for training (i.e. 60 activation and 60 rest data segments). Each segment was 15 seconds in duration. A total of forty-four features were extracted from each segment. The features were chosen according to previous TCD studies which have investigated lateralized activation tasks [11].

µL

µR

µL

µR

Figure 4.6. Sample recording depicting the three most common features: (1) difference between left and right mean velocities, µL – µR, at 10-15 seconds; (2) difference between left and right mean velocities, µL – µR, at 5-10 seconds, and, (3) slope of the right MCA CBFV (mR) at 5-10 seconds. Data shown are normalized and smoothed and represent one trial performed by participant 10. The left graph depicts a rest trial while the right graph portrays an activation trial, showing the difference between left and right mean CBFV at 10 – 15 seconds and at 5-10 seconds.

Weighted sequential feature selection (WSFS) was used to algorithmically select three to five features for each session, for each participant, to train a Naïve Bayes classifier. WSFS extended the sequential forward search (SFS) approach [65] by explicitly considering feature contributions (i.e. the number of times a specific feature was chosen). For each fold of a 10-fold cross-validation for feature selection, all features were first ranked according to F-score [64] for interclass separability, using the training set. These ranked features were then organized into cumulative subsets such that the first subset contained the top ranked feature, the second subset contained the top two ranked features, and so on. The last subset contained all features. Within each fold, the subset with the

highest validation accuracy was selected. Therefore, 10-fold cross-validation yielded 10 such subsets.

We enumerated the occurrence of each feature within these 10 subsets. Certain features appeared consistently across all subsets while others surfaced intermittently. The selected features were regrouped based on their frequency of occurrence, such that the mth group contained all the features that appeared at least m times, where m = {1, 2,…,10}. These new subsets were evaluated through a subsequent constrained 10-fold cross-validation (i.e. only the m pre-determined feature subsets were cross-validated) with newly randomized testing and training sets. The final set of features was then selected as that with the highest average validation accuracy. The chosen features were used to train a Gaussian Naïve Bayes classifier.

Figure 4.6 demonstrates a single trial of a rest task (left) and activation task (right). The three most common features selected across sessions and participants are highlighted. The least common of the three features (slope of the right MCA CBFV) was selected in six out of ten participants.

4.2.7. Performance evaluation

To capture the different nuances of online classification performance, several metrics were invoked as suggested by Thomas et al. and Schlӧgl et al. [70,71]. To gauge the correctness of classification for a biased classifier (i.e., unequal performance for each class), sensitivity and specificity were estimated from the confusion matrix [28]. Specificity is the number of true negatives divided by the actual number of negatives in the test set while sensitivity is the number of true positives divided by the actual number positives in the test set.

To measure the agreement between the predicted and desired selections [72,70] in the presence of unbalanced data (i.e., unequal number of samples per class due to the nature of the experiment), Cohen's kappa (κ) coefficient was estimated. Kappa ranges from 1 (perfect match) to 0 (chance level). If all values of κ within the 95% confidence interval

around the mean are above 0 ( ± 1.96 × ( ) > 0, where ϕ(k) is the standard error), then the average kappa value 𝜅𝜅̅ is significantly𝜑𝜑 𝜅𝜅 above chance [73]. The classification accuracy ACC (overall agreement) was derived from the 2×2 confusion matrix H, as

= 0 = (6) ∑ 𝐻𝐻𝑖𝑖𝑖𝑖𝑖𝑖 𝐴𝐴𝐴𝐴𝐴𝐴 𝑝𝑝 𝑁𝑁 where are the main diagonal elements (i.e., number of correct classifications) of the

confusion𝐻𝐻𝑖𝑖𝑖𝑖 matrix H and = is the total number of trials. The chance expected agreement , is the probability𝑁𝑁 ∑𝑖𝑖 ∑of𝑗𝑗 𝐻𝐻observing𝑖𝑖𝑖𝑖 the current confusion matrix and is given by, 𝑝𝑝𝑒𝑒

+ + = 2 (7) ∑𝑖𝑖 𝑛𝑛 𝑖𝑖𝑛𝑛𝑖𝑖 𝑝𝑝𝑒𝑒 𝑁𝑁 here + and + are the marginal column and row sums, respectively. The estimate of

𝑤𝑤the kappa𝑛𝑛 𝑖𝑖 coefficient𝑛𝑛𝑖𝑖 κ is thus,

κ = 0 (8) 1 𝑝𝑝 −𝑝𝑝𝑒𝑒 −𝑝𝑝𝑒𝑒 while its standard error ϕ(κ) is given by,

2 3 0+ [ + +( + + +)]/ ϕ(κ) = (9) �𝑝𝑝 𝑝𝑝𝑒𝑒 −∑𝑖𝑖 (1𝑛𝑛 𝑖𝑖𝑛𝑛𝑖𝑖 ) 𝑛𝑛 𝑖𝑖 𝑛𝑛𝑖𝑖 𝑁𝑁

−𝑝𝑝𝑒𝑒 √𝑁𝑁 This method of evaluation is preferred for problems with unbalanced classes [74,75], such as sleep classification.

To gauge performance of the system as a communication channel, we estimated the Nykopp information transfer rate (ITR), which is recommended for classification problems with unbalanced class sizes [70]. Letting represent the actual input category

( 0 = rest, 1 = activation) and represent the 𝑥𝑥𝑖𝑖 predicted output ( 0 = rest, 1 = activation𝑥𝑥 ), the𝑥𝑥 ITR was given by 𝑦𝑦𝑗𝑗 𝑦𝑦 𝑦𝑦

1 1 Nykopp = =0 =0 ( ) 2 (10)

𝐼𝐼𝐼𝐼𝑅𝑅 ∑𝑖𝑖 ∑𝑗𝑗 𝑝𝑝 𝑥𝑥𝑖𝑖 𝑝𝑝�𝑦𝑦𝑗𝑗 �𝑥𝑥𝑖𝑖�𝑙𝑙𝑙𝑙𝑙𝑙 �𝑝𝑝�𝑦𝑦𝑗𝑗 �𝑥𝑥𝑖𝑖�� 42

where

1 = =0 ( ) (11)

𝑝𝑝�𝑦𝑦𝑗𝑗 � ∑𝑖𝑖 𝑝𝑝 𝑥𝑥𝑖𝑖 𝑝𝑝�𝑦𝑦𝑗𝑗 �𝑥𝑥𝑖𝑖� = (12) 𝐻𝐻𝑖𝑖𝑖𝑖+ 𝑝𝑝�𝑦𝑦𝑗𝑗 �𝑥𝑥𝑖𝑖� 𝑛𝑛𝑖𝑖 while ( 0) = 0.7 and ( 1) = 0.3 are the prior probabilities of rest and activation tasks,

respectively𝑝𝑝 𝑥𝑥 to be, estimated𝑝𝑝 𝑥𝑥 from the average frequency of occurrence of each task when spelling an intended message with no mistakes. To calculate the bit-rate, we multiplied the Nykopp ITR by the average number of trials per minute [70].

To assess system efficiency, the average throughput, defined as the number of characters output per minute, was determined. Only correct characters were counted while the measured duration included the time required to make error corrections. Since participants were asked to correct mistakes during the spelling process, the estimated throughputs were generally conservative with the low char/min.

To measure the resemblance of the actual output to the intended output, the Levenshtein or edit distance was calculated. The edit distance compares the similarity between two strings of unequal length and is defined as the number of editorial operations required to convert the actual output into the intended output [76]. Each deletion and insertion of a character was given a weight of 1 while a substitution was given a weight of 2, being equivalent to a deletion followed by an insertion [76]. Since the intended outputs were of different lengths for the testing blocks of the three sessions, the edit distances were normalized based on the longest string length of the intended outputs (Equation 13). Other normalization methods more severely penalize a lack of input over an incorrect selection [77,78,79]. However, due to the study design, an incorrect selection should have a higher edit distance than a lack of input since the effort required to correct an incorrect selection is far greater than that needed to produce an intended output with no corrections. The normalized edit distance, , is given by,

𝐷𝐷𝐸𝐸𝐸𝐸 = × | | (13) | | 𝐷𝐷𝐸𝐸 ∗ 𝐷𝐷𝐸𝐸𝐸𝐸 𝑋𝑋 𝑋𝑋 43

where | | is the length of the intended output, is the raw edit distance between

intended𝑋𝑋 and actual output, and | | is the length of𝐷𝐷 𝐸𝐸the longest intended output from all ∗ | | sessions. Given the longest string𝑋𝑋 length in our experiment was 18, i.e. = 18, a ∗ normalized edit distance of 18 indicated no output and 0 meant perfect match𝑋𝑋 between the intended output and the actual output. Any score above 18 indicated that the actual output mismatched the intended output. The larger the normalized edit distance is, the further away the actual output was from the intended output.

4.3. Results

4.3.1. Feature Selection

Bilateral features were more frequently selected (Figure 4.7), which could be due to the left-lateralized nature of the language task. The higher selection frequency of bilateral features was consistent with that reported in a previous offline TCD-BCI study using verbal fluency [11]. Therefore, our modified verbal fluency task (i.e., rehearsing the spelling while imagining the writing of the target word) appeared to elicit machine- discernible left-hemispheric lateralization.

Figure 4.7. Normalized frequency of features (the number of times a feature has been selected divided by the total number of times all feature have been selected) across all participants

4.3.2. Inter-participant analysis

The online performance on the testing blocks in sessions II and III is reported in Table 4.1. The system achieved an average specificity and sensitivity of 81.44 ± 83.5% and 82.30 ± 7.39% respectively, resulting in an average kappa coefficient of 0.60 ± 0.03. All participants exhibited a kappa coefficient that exceeded chance. Seven out of eight participants achieved a kappa coefficient over 0.4, which is equivalent to an accuracy >70% had the classes been balanced [73].

4.3.3 Inter-session Results

Classification performance across all sessions are summarized in Table 4.1. Only four out of ten participants were able to achieve above chance level kappa coefficient ( ± 1.96 × ( ) > 0). Of the four participants, three were able to achieve a moderate 𝜅𝜅̅ agreement 𝜑𝜑 𝜅𝜅 45

within the first session (κ > 0.4). For sessions II and III, all participants achieved accuracies above chance. Moderate agreement between intended and predicted selections (κ > 0.4) was achieved in nine out of ten participants.

Table 4.1. Classification performance within individual sessions

Information # Specificity Sensitivity Kappa transfer rate Bit-rate Features (%) (%) ± ϕ( ) ITRNykopp (bits/min)

Session selected (bits/trial) Participant 𝒌𝒌� 𝒌𝒌 1 I 3 94.23 25.00 0.23 ± 0.19 0.05 0.16 II 3 84.27 70.97 0.53 ± 0.14 0.21 0.65 III 4 70.24 88.89 0.51 ± 0.13 0.23 0.73 2 I 1 78.26 71.43 0.43 ± 0.18 0.16 0.51 II 3 75.00 86.11 0.54 ± 0.13 0.24 0.77 III 2 82.93 94.74 0.72 ± 0.14 0.42 1.33 3 I 2 80.00 46.67 0.26 ± 0.17 0.05 0.16 II 2 82.72 83.78 0.63 ± 0.14 0.30 0.93 III 5 82.72 78.38 0.59 ± 0.14 0.25 0.78 4 I 2 32.65 72.73 0.03 ± 0.08 <0.01 0.01 II 4 77.03 71.88 0.45 ± 0.14 0.15 0.49 III 2 71.62 68.00 0.34 ± 0.13 0.10 0.31 5 I 4 36.84 100.00 0.27 ± 0.13 0.16 0.50 II 1 83.13 91.89 0.69 ± 0.14 0.39 1.22 III 1 71.80 90.91 0.58 ± 0.13 0.26 0.81 6 I 1 94.11 33.33 0.32 ± 0.20 0.09 0.27 II 2 81.18 88.57 0.63 ± 0.14 0.33 1.03 III 3 78.41 74.19 0.47 ±0.13 0.18 0.57 7 I 4 88.89 71.43 0.59 ± 0.21 0.26 0.82 II 3 90.91 71.88 0.63 ± 0.15 0.29 0.91 III 3 93.26 80.65 0.74 ± 0.16 0.41 1.28 8 I 4 85.42 36.36 0.22 ± 0.17 0.04 0.13 II 2 82.98 72.72 0.56 ± 0.16 0.21 0.66 III 3 75.29 77.14 0.47 ± 0.13 0.18 0.56 9 I 4 82.00 40.00 0.20± 0.16 0.04 0.12 II 3 88.64 93.75 0.76 ± 0.15 0.48 1.53 III 3 86.05 81.82 0.64 ± 0.15 0.31 0.99 10 I 4 93.88 54.55 0.52 ± 0.22 0.20 0.64 II 2 83.75 94.87 0.73 ± 0.15 0.43 1.37 III 2 88.37 82.35 0.68 ± 0.15 0.35 1.10 Average online 81.44 82.30 performance 0.60 ± 0.03 0.28 0.87 (sessions II and III) ± 8.35 ± 7.39

4.3.4 Dynamic Keyboard Output & User Feedback

The throughput for all three sessions of all participants are shown in Figure 4.8. The average throughput for session I, II and III across participants were 0.04 ± 0.05, 0.30 ± 0.14, and 0.32 ± 0.10 characters/minute, respectively.

Figure 4.8. Average throughput in characters/minute for sessions I, II, and III for all participants.

Figure 4.9 depicts the edit distances for each session.

Using a paired t-test, we compared the edit distances for the 10 participants at a rigorous significance level of 0.01. In the session I test block, there was no significant difference between edit distances for the no output case ( | | = 18 ) against distances when ∗ something was spelled (p = 0.619). In other words, the𝑋𝑋 composed output was distant from the target output string. In session II, testing blocks 1 and 2 showed significant reduction in edit distances below that achieved in session I (p = 0.001; p = 0.005), though there was no significant difference between the edit distances of the two blocks (p = 0.019). In session III, testing blocks 1 and 2 again showed significant improvement over session I edit distances (p = 0.001; p < 0.001). In addition, there was no significant difference between edit distances of the two testing blocks in session III (p = 0.790). Finally, there was no significant difference between edit distances from the corresponding blocks of sessions II and III (p ≥ 0.114). Table 4.2 gives an example of the normalized edit distances and the actual output along with intended output messages.

Table 4.2 Sample normalized edit distance with corresponding actual and intended outputs

Normalized Actual Output Intended Output Edit Distance 20.8 The to s Good morning 16.0 Wi Winter in January 12.0 Made ae Made it myself 6.5 Having Having fun

The correlation between tiredness levels and performance of all sessions were ascertained

through Spearman's coefficient (rs) [80]. There were no significant correlations between the tiredness levels and edit space or throughput. In addition, there were no significant correlations between tiredness and specificity or sensitivity. However, there was a negative trend on the tiredness of the participant before the session and the specificity of the testing blocks (rs = -0.314, n = 20, p = 0.177).

4.3.5 User Feedback

I A B C D E F II A B C D E F III A B C D E F 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 9 9 9 10 10 10

Very good / Very Good / Somewhat Okay Bad / Not really Very bad / Not at all

Figure 10.10. User feedback of sessions I, II, and III over the 10 participants. Each column represents the question corresponding with the survey from Appendix II. Each row represents each participant.

All participants reported the TCD headset to be very comfortable or somewhat comfortable across all sessions. Participants’ opinions on the dynamic keyboard’s ease of use improved over sessions II and III compared to that of session I, though not significantly (p=0.157, p = 0.059 respectively). Participants’ opinion of the online TCD- BCI system and their feelings after the usage did not differ much between all sessions (p>0.100 for all). Opinions in regard to the online TCD-BCI system were mostly neutral to positive and the user’s feelings after TCD-BCI usage were mostly neutral to positive as well. Participants’ tiredness before and after each session did not differ significantly across all three sessions (p>0.200 for all).

4.4. Discussion

4.4.1 Throughput of the Online TCD-based BCI Communication System

The throughputs for sessions two and three improved beyond those of session one, approaching transmission rates of established BCI spelling devices (0.5 char/min) [81]. The observed combination of low throughput (Figure 4.8) and high kappa coefficient (Table 4.1) can be attributed to the cost (temporal penalty) of a false-negative. If a bin was unintentionally bypassed, the participant had to wait between 4 and 15 additional slides before the target bin would be presented again. This wait time can translate into a temporal penalty of several minutes for missing a selection, and is an inherent limitation of scanning keyboards. Additional practice may help to decrease response latency. Further, the Dynamic Keyboard interface could also be improved (e.g., context-specific word prediction) to enhance the speed and accuracy of letter/word selection.

4.4.2 Feature Selection

Among the unilateral and bilateral features introduced to the feature set, some features were consistently selected by the feature selection algorithm across all participants. For most participants, the left lateralization of the mental task was pronounced. This finding confirms previous reports of left hemispheric lateralization accompanying verbal fluency tasks [11,41]. Due to the inherent lateralization, bilateral features were selected more frequently, as shown in Figure 4.7, particularly features corresponding to differences between the mean velocities of the left and right MCAs. Nine out of ten participants had the two most frequent features (i.e. difference of MCA means at 5-10 seconds and 10-15 seconds) selected at least in one session. The difference in means between 10 and 15 seconds was the most frequently selected feature, followed by the difference in means between 5 and 10 seconds. The selection frequency of the feature representing the difference in the means between 0 and 5 seconds is considerably lower (Figure 4.7). This

is likely due to theinherent5 to 10 seconds hemodynamic delay post-mental activation [59,60].

4.4.3 Classification of Word Repetition

All participants, except participant 3, improved in performance in sessions II and III when compared to session I. This improvement is attributable in part to the increase of training data available to the classifier. In addition, participants may have also become more familiar and comfortable with the study protocol and the user-interface. A longitudinal study of TCD-based BCI may help elucidate the effect of mental practice on functional performance.

Other factors surrounding the experimental paradigm (e.g. fatigue, extended trial duration or head motion) may have also impacted participant performance. For example, participant 4 reported a lack of concentration and physical fatigue, which may explain the lower accuracies for this individual.

4.4.4 Performance

The TCD-BCI was able to achieve an average bit-rate of 0.87 bits/min and a maximum of 1.53 bits/min. If the post-activation task 10 second recovery time was removed, the average bit-rate would improve to 1.10 bits/min. In addition, if we are able to bring a three-task TCD into an online setting, similar to the offline study by Aleem et al. [12], assuming equal priors, we can further increase the bit-rate to 4.38 bits/min. Due to a lack of published online TCD-BCIs at present, we compare our results to those of other hemodynamic BCIs. Recent fMRI BCI studies using two-task algorithm attained an average of 2 bits/min (~80% accuracy). Other fMRI BCI studies with a four-task algorithm attained bit rates between 0.9 bit/min to 1.5 bits/min (~ 90% accuracies) [82,83,84]. In comparison, our study achieved a comparable bit-rate with a much simpler set-up. At present, EEG-BCIs still offer the most compelling bit-rates, typically in the order of 15 – 30 bits/min [85,86].

The average throughput for session I was not significantly different from 0 characters/minute at a significance level of 0.01 (p = 0.022), which could have been due to the lowered specificity and sensitivity across participants in session I. Throughput for sessions II and III were significantly different from session I (p = 0.001; p < 0.001) and from 0 characters/minute (p < 0.001; p < 0.001). This improvement may be due, in part, to the user’s increasing familiarity with the keyboard, facilitating more skilled navigation through the user interface. Given the temporal resolution of TCD and the sequential nature of the dynamic keyboard, the throughput may have approached its theoretical limit at 0.3 characters/minute. Without modification of the user interface and the temporal window of data acquisition, further improvement of the throughput might not be possible. Incidentally, the change in throughput from session II to III was not significant (p = 0.653), but this does not preclude further improvements over extended periods of practice.

Similar to throughput, the edit distances for both session II and III improved significantly beyond session I values. Within sessions II and III, the edit distances for the actual outputs did not differ significantly. This suggested that the duration of TCD usage did not affect the quality of the output as testing block 2 typically occurred an hour after initial TCD set-up. Therefore, prolonged TCD usage may be possible provided that breaks are provided every 15 ~ 20 minutes.

4.4.5 User Feedback Questionnaire

Feedback regarding the performance of the online TCD-BCI system was neutral to positive (except for the first sessions for participants 2 and 5). Participants 4 and 8 both indicated that they were “somewhat tired” prior to and “very tired” after every session. The lack of energy prior to the session may have impacted participant performance with the online TCD-BCI. The live feedback may have further frustrated the participants, exacerbating their fatigue and diminishing their concentration, thus forming a negative feedback loop that further impacted performance. However, the lack of significant overall correlation between tiredness levels and performance in terms of specificity, sensitivity,

edit distance, and throughput suggest that user perceived fatigue did not directly impact overall user performance.

4.4.6 Limitations

Within this study, the participant population was considerably homogeneous. All participants were right-handed females who are naïve to the TCD-BCI system. With such a restricted participant population, it could be difficult to interpret the extendibility of the results into other populations such as a male population, individuals who are not naïve to the TCD-BCI system, or individuals who are not right-handed. However, having such a homogeneous sample will allow for comparison between restrictive factors such as gender and handedness in future studies.

The inefficiency of the scanning keyboard undoubtedly constrained the observed BCI accuracies. Scanning keyboards are frequently used as an interface for assistive technology devices [87,88]. However, the existing keyboard interface was prone to long delays in the event of incorrect selections. For example, for individuals who achieved high accuracies (>85%), it was still difficult to spell the intended phrase within the allotted time. Further improvement of the Dynamic Keyboard is necessary to achieve more efficient communication in future studies. Additionally, the throughput could be lower than reported if one takes into account the five minutes of break that is necessary per every fifteen minutes of TCD usage.

One of the major determinants for participant performance was their motivation and concentration. For participants who reported fatigue during specific sessions, (e.g. session 3, participant 4), the overall accuracy rates were lower compared to that of other participants. For participants who maintained concentration during the testing session, on the other hand, higher accuracies were observed (e.g. participant 7 and participant 9).

Despite the efforts to precisely locate the MCAs, unbalanced left and right CBFV magnitudes were occasionally observed. Probe placement errors may contribute to lower accuracies. Future TCD-BCI studies should endeavor to place the probes flush against the

skin overlying the temporal bone and establish the same physiological insonation depth and sampling volume on either side of the head.

Another potential source of signal contamination for functional TCD studies is motion artifact. Conspicuous facial movements may shift the TCD probes, resulting in momentary or continuous deterioration of the recorded signals. Additionally, extensive body movements e.g. swinging of the arms, crossing and uncrossing the legs, and shifting body in the chair) may also introduce CBFV changes unrelated to the mental tasks at hand. Moderate movements (e.g. moving hands, and shifting feet) were observed in many participants during this experiment. However, the high classification accuracies achieved suggest a level of robustness to these motion artifacts.

4.5. Conclusion

Using an online TCD-BCI system with an onscreen keyboard and combined word repetition-motor imagery as the activation task, an average specificity and sensitivity of 81.44 ± 8.35% and 82.30 ± 7.39% were achieved with 10 able-bodied participants. The agreement between the intended and machine-predicted selections was moderate (κ=0.60 ± 0.03), with an average information transfer rate of 0.87 bits/min. The average throughput was 0.35 ± 0.18 characters/min. These results support further investigation of online bilateral TCD-BCI systems using intuitive language tasks.

Chapter 5

Conclusions

5.1 Contributions

This thesis explored transcranial Doppler Ultrasonography as an online brain-computer interface system for communication purposes. The main contributions are as follow:

1. Compared different signal processing and classification methods for TCD system to determine the best classification scheme for the following activation paradigm:

a. Task duration of 15s

b. Post-activation duration of 10s

Both of these temporal durations represent reductions from values reported in the TCD- BCI literature. The selected signal processing algorithm extracted intuitive time-domain features from TCD signals, selected the most useful of these features based on weighted sequential forward selection based on F-score, and differentiated between the rest and activation mental activities using Naïve Bayes classifier.

2. Explored the possibility of classifying two novel mental tasks in an online study while demonstrating that classification accuracies are not significantly different from previous TCD-BCI studies. The task paradigm was as follows:

a. Activation mental task: verbal repetition with associated motor imagery

b. Rest mental task: visual tracking of a graphical signal display

3. Investigated user-specific feature sets ranging from 1 to 5 features. This low dimensionality and tailoring of the number of features allowed accurate performance with as little data as possible.

4. Implemented the first online TCD-BCI system for communication purposes. The system reached performance levels comparable to that of published off-line systems. In particular, the following performance metrics were attained.

a. Both specificity and sensitivity of the online TCD-BCI exceeded 80% across sessions II and III

b. A moderate level of agreement (κ=0.60 ± 0.03) was reached between the intended and machine-predicted selections.

c. Information transfer rate of the system reached 0.87 bits/min.

d. Throughput of TCD-BCI controlled keyboard was 0.35 ± 0.18 characters/min, which is a conservative estimate as participants were required to correct any mistakes and only the correct outputs were considered for the throughput calculation.

e. Although the edit distance indicated that none of the participants successfully spelled the entire target message, significant improvements in message fidelity were seen between sessions II and I, as well as sessions III and I.

5.2 Future Work

5.2.1 Evaluate User-Interface effects

Currently, there have been many types of user-interfaces designed for BCI signal acquisition systems, none of which are particularly suited to a hemodynamic BCI. Given the hemodynamic delay, any type of hemodynamic BCI will require an extremely efficient user interface. The user interface must be intuitive and be able to disambiguate

the desired character with as few user selections as possible. In addition, the interface should be clean, simple and uncluttered.

5.2.2 Evaluating the Effects of Practice on Performance

The effect of practice on user performance was not explored in this study. To determine the optimal amount of training data and user training to achieve peak performance, two types of longitudinal studies could be considered. First, one may entertain a dynamic classifier that incorporates all previous and incoming online data as training data, retraining the classifier periodically to determine the optimal amount of training data required. Another longitudinal study might involve setting a static amount of training data and analyze the effect of prolonged usage on system performance.

5.2.3 Testing in More Practical Situations

For this thesis, all data were collected under laboratory settings with minimal to no distractions. However, for the system to be practical in everyday life, it is necessary to conduct future tests in out-of-laboratory settings that expose the user to noise, environmental lighting, presence of people, changes in temperature, and other distractions.

5.2.4 Moving Towards the Target Population

While the ultimate goal is to create a novel BCI for individuals with severe motor impairments, it may be helpful to first conduct an intermediate study on individuals who have physical disabilities but retain a means of communication. For example, individuals with upper spinal cord injuries. This type of recruitment will allow us to confirm the participant’s precision in following the experimental protocol while providing some indication of whether or not our results can be generalized to individuals with disabilities. However, caution towards involuntary movements and spasms should be taken when designing the system.

Chapter 6

References

[1] M. Sledz, M. Oddy, and J.G. Beaumont, "Psychological adjustment to locked-in syndrome," Journal of Neurology, Neurosurgery & Psychiatry, vol. 78, no. 14, pp. 1407-1408, 2007.

D. J. Higginbotham, H. Shane, S. Russell, and K. Caves, "Access to AAC: present, [2] past, and future," Augmentative and Alternative Communication, vol. 23, no. 3, pp. 243-257, 2007.

B. Allison, E. Wolpaw, and J. Wolpaw, "Brain-computer interface systems: progress [3] and prospects," Expert Review of Medical Devices, vol. 4, pp. 463-474, 2007.

D.J. Krusienski, E.W. Sellers, D.J. McFarland, T.M. Vaughan, and J.R. Wolpaw, [4] "Toward enhanced P300 speller performance," Journal of Neuroscience Methods, vol. 167, no. 1, pp. 15-21, 2008.

T. H. Falk, M. Guirgis, S. Power, and T. Chau , "Taking NIRS-BCIs outside the lab: [5] towards achieving robustness against environment noise," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 19, no. 2, pp. 136-146, 2011.

R. Sitaram, A. Caria , and N. Birbaumer, "Hemodynamic brain-computer interfaces [6] for communication and rehabilitation," Neural Networks, vol. 22, no. 9, pp. 1320- 1328, 2009.

S. G. Mason, A. Bashashati, M. Fatourechi, K. F. Navarro, and G. E. Birch, "A [7]

comprehensive survey of brain interface technology designs," Annals of Biomedical Engineering, vol. 35, no. 2, pp. 137-169, 2007.

S. Moghimi and T. Chau, "A review of EEG-based brain-computer interfaces as [8] access pathways for individuals with severe disabilities," Assistive Technology, vol. 25, no. 2, pp. 99-110.

N. Birbaumer, A. Murguialday, and L. Cohen, "Brain-computer interface in [9] paralysis," Current Opinion in Neurology, vol. 21, no. 6, pp. 634-638, 2008.

C.H. Tegeler and D. Ratanakorn, "Physics and principles," in Transcranial Doppler [10] ultrasonography. Boston, Massachusetts: Butterworth-Heinemann, 1999, pp. 3-11.

A. J. B. Myrden, A. Kushki, E. Sejdic, A. Guerguerian, and T. Chau, "A Brain- [11] Computer Interface Based on Bilateral Transcranial Doppler Ultrasound," PLoS ONE, vol. 6, no. 9, 2011.

I. Aleem and T. Chau, "Towards a hemodynamic BCI using transcranial Doppler [12] (TCD) without user-specific training data," Journal of Neural Engineering, vol. 10, no. 1, 2013.

A. Faress and T. Chau, "Towards a multimodal brain-computer interface: [13] Combining fNIRS and fTCD measurements to enable higher classification accuracy," Neuroimage, vol. 77, pp. 186-194, August 2013.

S. G. Mason, "Brain-controlled switch for asynchronous control applications," IEEE [14] Transactions on Biomedical Engineering, vol. 47, pp. 1297-1307, 2000.

K. Tai, S. Blain, and T. Chau, "A review of emerging access technologies for [15] individuals with severe motor impairments," Assistive Technology, vol. 20, pp. 204- 219, 2008.

P. Haselager, R. Vlek, J. Hill, and F. Nijboer, "A note on ethical aspects of BCI," [16] Neural Networks, vol. 22, pp. 1352-1357, 2009.

J. R. Wolpaw et al., "BCI meeting 2005 - workshop on signals and recording [17] methods," IEEE Transacrtions on Neural Systems and Rehabilitation Engineering, vol. 14, no. 2, pp. 138-141, 2006.

C. de Negueruela, Broschart M., Menon C., and J.R. Millan, "Brain–computer [18] interfaces for space applications," Personal and Ubiquitous Computing, vol. 15, no. 5, pp. 527-537, 2011.

E.C. Leuthardt, G. Schalk, J.R. Wolpaw, J.G. Ojemann, and D.W. Moran, "A brain- [19] computer interface using electrocorticographic signals in humans," Journal of Neural Engineering, vol. 1, no. 2, pp. 63-71, 2004.

E. Sellers and E. Donchin, "A P300-based brain-computer interface: initial tests by [20] ALS patients," Clinical Neurophysiology, vol. 117, no. 3, pp. 538-548, 2006.

J. Mellinger et al., "An MEG-based brain–computer interface (BCI)," NeuroImage, [21] vol. 36, no. 3, pp. 581-593, 2007.

N. Weiskopf et al., "Principles of a brain-computer interface (BCI) based on real- [22] time functional magnatic resonance imaging (fMRI)," IEEE Transactions on Biomedical Engineering, vol. 51, no. 6, pp. 966-970, 2004.

S. Coyle, T. Ward, C. Markham, and G. McDarby, "On the suitability of near- [23] infrared (NIR) systems for next-generation brain-computer interfaces," Physiological Measurement, vol. 58, no. 4, pp. 815-822, 2004.

R. Aaslid, T.M. Markwalder, and H. Nornes, "Noninvasive transcranial Doppler [24] ultrasound recording of flow velocity in basal cerebral arteries," Journal of Neurosurgery, vol. 57, no. 6, pp. 769-774, 1982.

G. Tsivgoulis, A. Alexandrov, and M. Sloan, "Advances in transcranial Doppler [25] ultrasonography," Current Neurology and Neuroscience Reports, vol. 9, no. 1, pp. 46-54, 2009.

L. Lowe and D. Bulas, "Transcranial Doppler imaging in children: sickle cell [26] screening and beyond," Pediatric Radiology, vol. 35, pp. 54-65, 2005.

R. Panerai, "Transcranial Doppler for evaluation of cerebral autoregulation," [27] Clinical Autonomic Research, vol. 19, no. 4, pp. 197-211, 2009.

M. Saqqur, "Role of transcranial Doppler in neurocritical care," Critical Care [28] Medicine, vol. 35, pp. s216-s223, 2007.

S. Sakar, S. Ghosh, S.K. Ghosh, and A. Collier, "Role of transcranial Doppler [29] ultrasonography in stroke," Postgraduate Medical Journal, vol. 83, pp. 683-689, 2007.

D.V. Bishop, N.A. Badcock, and G. Holt, "Assessment of cerebral lateralization in [30] children using functional transcranial Doppler ultrasound (fTCD)," Journal of Visualized Experiments, vol. 27, no. 43, p. 2161, 2010.

L. Bracco et al., "Cerebral hemodynamic lateralization during memory tasks as [31] assessed by functional transcranial Doppler (fTCD) sonography: effects of gender and healthy aging," Cortex, vol. 47, no. 6, pp. 750-758, 2010.

F. J.C. Artal, C. V. Cabrera, and T. A. Horan, "Lateralization of cerebral blood flow [32] velocity changes during auditory stimulation: a functional transcranial Doppler study," Applied Neuropsychology, vol. 11, no. 3, pp. 167-174, 2004.

P.C. Njemanze, C.R. Gomez, and S. Horenstein, "Cerebral lateralization and color [33] perception: a transcranial Doppler study," Cortex, vol. 28, no. 1, pp. 69-75, 1992.

P.C. Njemanze, "Cerebral lateralization and general intelligence: gender differences [34] in a transcranial Doppler study," Brain and Language, vol. 92, no. 3, pp. 234-239, 2005.

A. Myrden, A. Kushki, E. Sejdic, and T. Chau, "Towards increased data [35] transmission rate for a three-class metabolic brain-computer interface based on transcranial Doppler ultrasound," Neuroscience Letters, vol. 528, no. 2, pp. 99-103, 2012.

Hemodynamics AG, TCD simulator, 2008. [36] P.D. Syme, "The use of Transcranial Doppler ultrasonography as a 'cerebral [37] stethoscope' for the assessment and treatment of acute stroke," The Journal - Royal College of Physicians of Edinburgh, vol. 36, pp. 17-28, 2006.

L. Edvinsson, Cerebral blood flow and metabolism, 2nd ed. New York, NY, USA: [38] Raven Press, 1993.

N. Stroobant and G. Vingerhoets, "Transcranial Doppler ultrasonography monitoring [39] of cerebral hemodynamics during performance of cognitive tasks: a review," Neuropsycology Review, vol. 10, no. 4, p. 213, 2000.

P. Haydon and G. Carmignoto, "Astrocyte control of synaptic transmission and [40] neurovascular coupling," Physiological Reviews, vol. 86, no. 3, pp. 1009-1031, 2006.

G Vingerhoets and N Stroobant, "Lateralization of cerebral blood flow velocity [41] changes during cognitive tasks. a simultaneous bilateral transcranial Doppler study," Stroke, vol. 30, pp. 2152-2158, 1999.

Compumedics, DWL Transcranial Doppler: QL Software, 2009. [42]

T.R. Nelson and J.S. Abramowics, "Ultrasound biosafety considerations for the [43] practicing sonographer and sonologist," Journal of Ultrasound in Medicine, vol. 28, pp. 139-150, 2009.

The Safety Group of the British Medical Ultrasound Society, "Guidlines for the safe [44] use of diagnostic ultrasound equipment," The British Medical Ultrasound Society, November 2009.

T.O. Zander, C. Kothe, S. Jatzev, and M. Gaertner, "Application for Brain-Computer [45] Interfaces," in Human-Computer Interaction Series, J. Karat and J. Vanderdonckt, Eds. London Dordrecht Heidelberg New York: Springer, 2010, ch. 6, pp. 90-93.

B. Sorger, J. Reithler, B. Dahmen, and R. Goebel, "A real-time fMRI-based spelling [46] device immediately enabling robust motor-independent communication," Current Biology, vol. 22, pp. 1333-1338, July 2012.

CanAssist, Dynamic Keyboard, 2008. [47] C.A. Porro et al., "Primary motor and sensory cortex activation during motor [48] performance and motor imagery: a functional magnetic resonance imaging study," The Journal of Neuroscience, vol. 16, no. 23, pp. 7688-7698, December 1996.

A.S.M. Salinet, T.G. Robinson, and R.B. Panerai, "Active, passive, and motor [49] imagery paradigms: component analysis to assess neurovascular coupling," Journal of Applied Physiology, vol. 114, pp. 1406-1412, February 2013.

J. Levy et al., "Piecemeal recruitment of left-lateralized brain areas during reading: [50] A spatio-functional account," NeuroImage, vol. 43, no. 3, pp. 581-591, November 2008.

J.R. Wolpaw, N. Birbaumer, D.J. McFarland, G. Pfurtscheller, and T.M. Vaughan, [51] "Brain-computer interfaces for communication and control," Clinical

Neurophysiology, vol. 113, pp. 767-791, 2002.

S. Sarkar, S. Ghosh, S. Ghosh, and A. Collier, "Role of transcranial Doppler [52] ultrasonography in stroke," Postgraduate Medical Journal, vol. 83, no. 985, pp. 683- 689, 2007.

B. Reinsfelt et al., "Transcranial doppler microembolic signals and serum marker [53] evidence of brain injury during transcatheter aortic valve implantation," Acta Anaesthesiologica Scandinavica, vol. 56, no. 2, pp. 240-247, 2012.

H. White and B. Venkatesh, "Applications of transcranial Doppler in the ICU: a [54] review," Intensive Care Medicine, vol. 32, no. 7, pp. 981-994, 2006.

A Haag, N Moeller, S Knake, A Hermsen, and W Ortel, "Language lateralization in [55] children using functional transcranial Doppler sonography," Dev Med Child Neurol, vol. 52, pp. 331-336, 2009.

A JO Whitehouse, N Badcock, M Groen, and D VM Bishop, "Reliability of a novel [56] paradigm for determining hemispheric lateralization of visuospatial function," Journal of the International Neuropsychological Society, vol. 15, pp. 1028-1032, 2009.

R Aaslid, "Visually evoked dynamic blood flow response of the human cerebral [57] circulation," Stroke, vol. 18, pp. 771-775, 1987.

R.B. Buxton and F.R. Lawrence, "A model for the coupling between cerebral blood [58] flow and oxygen metabolism during neural stimulation," Journal of Cerebral Blood Flow and Metabolism, vol. 17, pp. 64-72, 1997.

I. Szirmai, I Amrein, L. Palvolgyi, R. Debreczeni, and A. Kamondi, "Correlation [59] between blood flow velocity in the middle cerebral artery and EEG during cognitive effort," Cognitive Brain Research, vol. 24, pp. 33-40, 2005.

A.G. Harders, G. Laborde, D.W. Droste, and E. Rastogi, "Brain activity and blood [60] flow velocity changes: a transcranial Doppler study," International Journal of Neuroscience, vol. 47, pp. 81-102, 1989.

A. Alexandrov, M. Sloan, L.K. Wong, C. Douville, and A. Razumovsky, "Practice [61] standards for transcranial Doppler ultrasound: Part I - test performance," Journal of Neuroimaging, vol. 17, no. 1, pp. 11-18, 2007.

BMUS Safety Group, "Guidelines for the safe use of diagnostic ultrasound [62] equipment," Ultrasound, vol. 18, pp. 52-59, 2010.

D. Foley and J. Sammon, "An optimal set of discrimination vectors," IEEE [63] Translactions on Computers, vol. 24, pp. 281-289, 1975.

R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification. New York: John [64] Wiley & Sons, 2012.

K.A. Mamun et al., "A robust strategy for decoding movements from deep brain [65] local field potentials to facilitate brain machine interfaces," Biomedical Robotics and Biomechatronics (BioRob), 2012 4th IEEE RAS & EMBS International Conference on, pp. 320-325, 2012.

C. Bishop, Pattern Recognition and Machine Learning, 1st ed.: Springer, 2006. [66] F. Pereira, T. Mitchell, and M. Botvinick, "Machine learning classifiers and fMRI: a [67] tutorial overview," NeuroImage, vol. 45, pp. S199-209, 2009.

I. Rish, "An empirical study of the naive Bayes classifier," in Proceedings of IJCAI- [68] 01 workshop on Empirical Methods in AI, Sicily, Italy, 2001, pp. 41-46.

G.R. Muller-Putz, R. Scherer, C. Brunner, R. Leeb, and G. Pfurtscheller, "Better [69] than random? A closer look on BCI results," International Journal of

Bioelectromagnetism, vol. 10, no. 1, pp. 52-55, 2008.

E. Thomas, M. Dyson, and M. Clerc, "An analysis of performance evaluation for [70] motor-imagery based BCI," Journal of Neural Engineering, pp. doi:10.1088/1741- 2560/10/3/031001, May 2013.

A. Schlögl, J. Kronegg, J. E. Huggins, and S. G. Mason, "Evaluation criteria in BCI [71] research," in Towards Brain-Computer Interfacing. USA: MIT Press, 2007, ch. 19, pp. 327-342.

J. Cohen, "A coefficient of agreement for nomnal scales," Educational and [72] Psychological Measurement, vol. 20, pp. 37-46, 1960.

E.V.C. Friedrich, R. Scherer, and C. Neuper, "The effect of distinct mental strategies [73] on classification performance for brain-computer interfaces," International Journal of Psychophysiology, vol. 84, no. 1, pp. 86-94, 2012.

H. Danker-Hopfe et al., "Interrater reliability between scorers from eight European [74] sleep laboratories in subjects with different sleep disorders," Journal of Sleep Research, vol. 13, no. 1, pp. 63-69, 2004.

P. Anderer et al., "An E-Health solution for automatic sleep classification according [75] to Rechtschaffen and Kales: Validation study of the Somnolyzer 24x7 utilizing the Siesta Database," Neuropsychobiology, vol. 51, pp. 115-133, 2005.

D. Sankoff and J.B. Kruskal, Time wraps, string edits, and macromolecules: the [76] theory and practice of sequence comparison. Massachusetts, USA: Addison Wesley Publishing Company, April 1983, ch. 1, p. 00.

A. Marzal and E. Vidal, "Computation of normalized edit distance and applications," [77] IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 9, pp. 926-932, September 1993.

A. Weigel and F. Fein, "Normalizing the weighted edit distance," in Proc. 12th [78] IAPR Int'l Conf. Pattern Recognition, vol.2, October 1994, pp. 339 - 402.

Y.J. Li and B. Liu, "A normalized Levenshtein distance metric," IEEE transactions [79] on pattern analysis and machine intelligence, vol. 29, no. 6, pp. 1091-1095, June 2007.

B.W. Brown and M. Hollander, Statistics: A Biomedical Introduction. New York, [80] USA: John Wiley and Sons, Inc, 1977.

N. Birbaumer et al., "A spelling device for the paralysed," Nature, vol. 398, pp. 297- [81] 298, 1999.

S.S. Yoo et al., "Brain–computer interfaceusingfMRI: spatialnavigationbythoughts," [82] Neuroreport, vol. 15, no. 10, pp. 1591-1595, 2004.

S.M. LaConte, S.J. Peltier, and X.P. Hu, "Real-time fMRI using brain-state [83] classificatoin," Human Brain Mapp, vol. 28, no. 10, pp. 1033-1044, 2007.

L. Minati, A. Nigri, C. Rosazza, and M.G. Bruzzone, "Thoughts turned into high- [84] level commands: Proof-of-concept study of a vision-guided robot arm driven by functional MRI(fMRI) signals," Medical Engineering & Physics, vol. 34, pp. 650- 658, 2012.

K. Kansaku, N. Hata, and K. Takano, "My thoughts through a robot's eye: an [85] augmented reality-brain-machine interface," Neuroscience Research, vol. 66, no. 2, pp. 219-222, 2010.

E. Donchin and Y. Arbel, "P300 based brain computer interfaces: a progress report," [86] in Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience. Berlin Heidelberg: Springer, 2009, pp. 724-731.

D. Jans and S. Clark, Augmentative communication in practice: an introduction, A. [87] Wilson, Ed. Edinburgh: CALL Centre, United Kingdom: University of Edinburgh, 1994.

G.W. Lesher, B.J. Moulton, and J.D. Higginbotham, "Techniques for augmenting [88] scanning communication," Augmentative and Alternative Communication, vol. 14, no. 2, pp. 81-101, 1998.

Appendix

Appendix I: Criteria Questionnaire

Participant #: Date:

REB #:

Please answer the following questions to the best of your knowledge:

Do you have or have you had any cardiovascular disease?

a) yes b) used to c) no

Do you have or have you had any respiratory problem? a) yes b) used to c) no

Do you have a history of migraine? a) yes b) used to c) no

Have you had any head injury?

) yes b) no

Appendix II: Survey Template

Participant #: ______

REB #: ______

The following questions are related to your experience with using the online TCD communication system:

A. Overall, what did you think of the online TCD system? a) very good b) good c) okay d) bad e) very bad

B. How comfortable was the TCD apparatus?

a) very b) somewhat c) not really d) not at all

If it was a bit uncomfortable, what was the reason?

C. How easy was the Dynamic Keyboard to use?

a) very b) somewhat c) not really d) not at all

D. How do you feel after using the system?

a) very good b) good c) okay d) bad e) very bad

E. How tired did you feel before the study?

a) very b) somewhat c) not really d) not at all

F. How tired do you feel after the study? a) very b) somewhat c) not really d) not at all

Any comments you would like to make?

Thank you for participating in the study!

Appendix III: Post Study Questionnaire

Participant #: ______

REB #: ______

1.) Have you ever had a systemic rheumatologic illness (e.g. endocarditis - inflammation of the inner layer of the heart, vasculitis - inflammatory destruction of blood vessels)?

2.) Have you ever had a seizure disorder?

3.) Have you ever had a CNS infection (e.g. meningitis)?

4.) Have you ever had a traumatic brain injury?

5.) Do you require a pacemaker or electronic medical implants?

6.) Have you ever received treatment for a psychiatric, behavioural or language disorder (e.g. ADHD, conduct disorder, Tourette Disorder, Obsessive Compulsive Disorder, recurrent major depression, schizophrenia, bipolar disorder, drug dependence, autism, delay in learning to speak, stuttering, dyslexia, problems understanding)?

7.) Do you exhibit a fear of enclosed spaces (not liking elevators, small rooms etc.)?