Santa Clara University Scholar Commons Bioengineering Senior Theses Engineering Senior Theses 6-2021 Classifying Brainwaves for Brain-Computer Interface Technology Derrick Wang Brendan Lawler Follow this and additional works at: https://scholarcommons.scu.edu/bioe_senior Part of the Biomedical Engineering and Bioengineering Commons SANTA CLARA UNIVERSITY Department of Bioengineering I HEREBY RECOMMEND THAT THE THESIS PREPARED UNDER MY SUPERVISION BY Derrick Wang, Brendan Lawler ENTITLED CLASSIFYING BRAINWAVES FOR BRAIN-COMPUTER INTERFACE TECHNOLOGY BE ACCEPTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF BACHELOR OF SCIENCE IN BIOENGINEERING June 10, 2021 Thesis Advisor date June 10, 2021 Department Chair date Classifying Brainwaves for Brain-Computer Interface Technology Santa Clara University Derrick Wang and Brendan Lawler Advisor: Dr. Yuling Yan Table of Contents Abstract 4 Acknowledgements 4 List of Figures 5 List of Tables 5 Chapter 1: Introduction 6 1.1 Problem 6 1.2 Goal 6 Chapter 2: Background and Significance 8 2.1 Electroencephalography 8 2.2 Brain Computer Interface 9 2.3 Current Technology 9 2.3.1 Brain Computer Interface Based Control of Wheel Chair 9 2.3.2 EEG Based Neural Prosthetic 10 Chapter 3: Experimental Methods and Materials 12 3.1 System Overview 12 3.1.1 Programming Platform 12 3.1.2 Mu Wave 12 3.1.3 Project Focus 13 3.1.4 Structure 13 3.2 Data Set 14 3.2.1 EEG Motor Movement/imagery Dataset 14 3.2.2 Electrode Selection 15 3.3 Signal Processing and Analysis 16 3.3.1 Filter 16 3.3.2 Spectrograms 18 3.4 Machine Learning 21 3.4.1 Convolutional Neural Networks (CNNs) 21 3.4.2 Transfer Learning 22 2 3.4.3 ResNet-50 Architecture 23 3.4.4 Training 25 3.4.5 Testing 25 Chapter 4: Results 26 4.1 Trial 1 26 4.2 Trial 2 26 4.3 Discussion 29 Chapter 5: Conclusion 31 Appendices 32 Appendix A: Trial 1 Spectrogram Plotting 32 Appendix B: Trial 2 Spectrogram Plotting 33 Appendix C: Trial 1 CNN Architecture 35 Appendix D: Trial 2 CNN Architecture 42 References 51 3 Abstract The intention of this project is to develop a brainwave classification system that will help restore the independence of those with severe motor function impairments. While current brain computer interface (BCI) technology offers a means of control for those with limited mobility, severely motor disabled individuals represent a population in need of methods to restore independent motor control. Thus, the objective of our project is to utilize neural signals from electroencephalogram (EEG) recordings to develop a machine learning classifier. Since our specific goal is to help those with limited mobility, we are focusing on motion imagery tasks which elicit a specific mu rhythm in the brain wave that occurs over the sensorimotor cortex. Using this principle, we can use EEG recordings of subjects imagining moving their limbs to extract particular features that can be used as motionless commands. The first stage of our project involves identifying a suitable motion imagery data set. This is followed by a pre-processing stage that involves filtering and transforming the signals. After performing necessary processing on our dataset, we train our machine learning model with the goal of developing a classification system in which test data sets can be entered and motion imagery command features can be automatically extracted and eventually utilized for the BCI. Acknowledgements We would like to thank Dr. Yan for her guidance and support throughout the duration of our research and thesis writing. Her assistance was immensely helpful in acquiring the necessary knowledge and skills used in completing this study. We would also like to acknowledge Brent Baculi and Stuart Cansdale for their previous work [1], and for suggesting further research into EEG signal classification which influenced the scope of this project. 4 List of Figures Figure 3.1. Electrode placement and channel locations 16 Figure 3.2. An EEG signal before and after it was filtered 18 Figure 3.3. A spectrogram for the filtered baseline eyes closed signal 19 Figure 3.4. A spectrogram for a sample of a task 2 signal (Trial 1) 20 Figure 3.5. A spectrogram for a sample of a task 2 signal (Trial 2) 21 Figure 3.6: Structure of the ResNet-50 CNN 24 Figure 4.1. Training progress of the model for classification (Four tasks) 26 Figure 4.2. Training progress of the model for binary classification (Task 1 and 3) 27 Figure 4.3. Training progress of the model for binary classification (Task 1 and 4) 27 Figure 4.4. Training progress of the model for binary classification (Task 3 and 4) 28 List of Tables Table 3.1: List of tasks used in this study 15 Table 3.2: The architecture specifications for a ResNet-50 CNN 23 Table 4.1: Summary of experimental results 28 5 Chapter 1: Introduction 1.1 Problem Over 5.4 million people are affected by some form of paralysis in the US alone [2]. Of these cases, 1,462,220 people suffer from spinal cord injuries, with only 1% experiencing full recovery. These individuals who suffer strokes, brain or spinal cord injuries, and neurodegenerative diseases experience a loss in motor function of varying degrees. Those who experience a severe loss in motor function may lose their ability to function independently within society. For example, 41.8% of individuals with paralysis report an inability to work, resulting in almost one third of those with paralysis having an annual household income of less than $15,000 [2]. Overall, the large number of individuals who are impacted by the negative effects of any paraplegia warrants the need for a system in which independent control can be restored despite physical disabilities. 1.2 Goal The objective of our project is to utilize motion imagery EEG data to develop a brainwave classification system that will help restore the independence of those with severe motor function impairments. The central idea is to develop technology that is able to assist individuals who are physically impaired by providing a method of control that enables them to substitute normal physical movement with solely neural commands. Our initial intentions were to gather EEG data in person and eventually utilize the EEG device to operate a motorized system in real time; however, due to the limitations of the COVID 19 pandemic, we decided it would be more feasible to utilize readily available public datasets for developing a deep learning 6 classifier capable of distinguishing between active and resting mental states. By focusing on motion imagery tasks that require no physical movement, the classification system can be applied towards brain computer interface technology to assist those with limited motor function. 7 Chapter 2: Background and Significance 2.1 Electroencephalography The human brain consists of over 100 billion brain cells–called neurons–that function by sending electrical signals to one another in highly complex and interconnected networks. The nature and characteristics of the electrical signals generated by neurons vary across different mental states. To measure and distinguish these variations in the brain's electrical signals, electroencephalography (EEG) devices can be utilized. EEG devices use flat, metal disk electrodes to pick up electrical signals over the surface of the scalp [3]. By placing the electrodes in a particular orientation, specific activity over a given lobe of the brain can be recorded and honed in on using a particular channel of the EEG devices. While raw EEG data itself appears noisy and unstructured, different preprocessing and filtering techniques can be utilized to achieve clearer signals. After applying these techniques, five basic patterns can be distinguished that represent unique brain waves linked to distinct functions. These brain waves are the delta wave (0-4 Hz; linked to deep sleep), theta wave (4-7 Hz; linked to light sleep), alpha wave (8-13 Hz; linked to relaxed awake state), beta wave (14-30 Hz; linked to alertness and concentration), and gamma wave (>30 Hz; linked to peak focus) [4, 5]. In addition to the well known brain rhythms listed above, there exists a variant of the alpha wave known as the mu wave. This brain rhythm is apparent over the sensorimotor cortex, and is closely related to the resting state of motor neurons. The mu wave is suppressed during movement planning, or when individuals think about performing a movement [6]. As such, motion imagery tasks of individuals imagining moving are capable of causing measurable 8 changes in the presence of the mu wave over the sensorimotor cortex. We will discuss the mu wave in more detail in later sections, as its principles are fundamental to our project. 2.2 Brain Computer Interface From the ability to record and distinguish neural activity, brain computer interfaces (BCIs) can be developed. BCIs are devices capable of acquiring EEG signals either non-invasively or invasively by placing electrodes on the scalp or surface of the brain, and transmitting the data to a local device for processing and feature extraction [7]. Once neural signals are collected and processed, machine learning and pattern classification algorithms can be implemented to translate desired features into commands. These commands are then used as feedback to control a local robot device, allowing individuals to perform physical or digital tasks through only neural commands. BCIs of particular interest to our project are those developed using external EEG recording devices. These are particularly attractive because they are the most accessible due to their ability to directly measure neural activity, while also remaining relatively inexpensive, and non-invasive in nature [8]. 2.3 Current Technology BCI technology has made significant advancements over the past few decades.