Applying Block-Based Programming to Neurofeedback Application Development
Total Page:16
File Type:pdf, Size:1020Kb
APPLYING BLOCK-BASED PROGRAMMING TO NEUROFEEDBACK APPLICATION DEVELOPMENT By CHRIS S. CRAWFORD A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2017 ⃝c 2017 Chris S. Crawford ACKNOWLEDGMENTS I would like to thank my Lord and Savior Jesus Christ for carrying me to heights I never imagined. To my loving mother, Dolores Crawford, for always believing in me. My father, Chris S. Crawford Sr., for ensuring I never strayed off a path of success. To my grandparents, Lubertha and O.D Crawford who taught me how to think critically and work hard. To my Aunt, Ida Tyree-Hyche, for providing me love and knowledge that truly changed my life. I would also like to express my extreme gratitude to my advisor, Dr. Juan E. Gilbert, who has guided me through my years as a Ph.D. student. I want to thank Dr. Monica Anderson for introducing me to research. I would like to thank my committee members, Dr. Kyla McMullen, Dr. Christina Gardner-McCune, and Dr. James Oliverio, for their feedback and support of my research. Lastly, I would like to thank Intel Corporation, GEM, and NSF for their financial support. 3 TABLE OF CONTENTS page ACKNOWLEDGMENTS ................................. 3 LIST OF TABLES ..................................... 7 LIST OF FIGURES .................................... 8 ABSTRACT ........................................ 11 CHAPTER 1 INTRODUCTION .................................. 13 1.1 Motivation .................................... 14 1.2 Objective .................................... 16 1.3 Challenges .................................... 16 1.4 Thesis Statement ................................ 18 1.5 Overview of Approach ............................. 19 1.6 Contributions and Implications for Design .................. 21 1.7 Dissertation Organization ........................... 22 2 LITERATURE REVIEW .............................. 23 2.1 BCI Software Platforms ............................ 23 2.1.1 Method Focused BCI Software Platforms ............... 25 2.1.2 Application Focused BCI Software Platforms ............. 31 2.2 Block-Based Programming (BBP) ....................... 36 2.2.1 Block-Based Programming Environments ............... 37 2.2.2 End-User Programming with BBP Environments ........... 44 3 DESIGN AND IMPLEMENTATION OF A BLOCK-BASED NEUROFEEDBACK APPLICATION DEVELOPMENT ENVIRONMENT ............... 47 3.1 Challenges .................................... 49 3.1.1 System Configuration .......................... 50 3.1.2 Text-based Languages .......................... 52 3.1.3 Feedback ................................. 53 3.2 NeuroBlock Design Methodology ....................... 54 3.2.1 EEG Apparatus ............................. 55 3.2.2 EEG Data Communication ....................... 57 3.2.3 Web Application ............................. 59 3.2.3.1 Blocks ............................. 62 3.2.3.2 Neurofeedback ........................ 69 3.3 Implementing NeuroBlock ........................... 71 3.3.1 EEG Data Communication ....................... 71 3.3.2 Feedback ................................. 74 4 3.3.3 Stage and Workspace Management .................. 75 3.4 Pilot Study ................................... 76 3.4.1 Population and Procedure ....................... 76 3.4.2 Observations ............................... 80 3.4.3 Discussions ................................ 84 3.4.4 Pilot Study Conclusion ......................... 85 3.5 Chapter Summary ............................... 85 3.6 Conclusion .................................... 86 4 EVALUATION OF NEUROBLOCK ........................ 87 4.1 Participants ................................... 87 4.2 Procedures .................................... 88 4.3 Methodology .................................. 93 4.4 Results ...................................... 94 4.4.1 Session One ............................... 94 4.4.1.1 Programming efficacy ..................... 94 4.4.1.2 Learning barriers ....................... 95 4.4.1.3 Usability ............................ 96 4.4.1.4 Self-Efficacy, effectiveness, and efficiency .......... 98 4.4.1.5 Interviews ........................... 100 4.4.2 Session Two ............................... 106 4.4.2.1 Programming efficacy ..................... 106 4.4.2.2 Learning barriers ....................... 107 4.4.2.3 Self-Efficacy, effectiveness, and efficiency .......... 108 4.4.2.4 Interviews ........................... 110 4.4.3 Session Three .............................. 114 4.4.3.1 Programming efficacy ..................... 114 4.4.3.2 Learning barriers ....................... 115 4.4.3.3 Self-Efficacy, effectiveness, and efficiency .......... 116 4.4.3.4 Interviews ........................... 119 4.4.4 Summary ................................. 119 5 SUMMARY AND FUTURE DIRECTIONS .................... 121 5.1 Research Questions Revisited ......................... 122 5.2 Contributions .................................. 125 5.3 Limitations ................................... 126 5.4 Future Work ................................... 126 5.5 Conclusion .................................... 127 APPENDIX A STUDY PROTOCOL AND MATERIALS ..................... 128 A.1 Study Procedures ................................ 128 A.2 Recruitment Flyer ............................... 129 5 A.3 Screening Questionnaire ............................ 130 A.4 BCI Self-Efficacy Survey ............................ 131 A.5 Programming Self-Efficacy Survey ....................... 132 A.6 Session One Pre-Task Instructions ...................... 132 A.7 Session One Task Instructions ......................... 133 A.8 Session Two Pre-Task Instructions ...................... 134 A.9 Session Two Task Instructions ......................... 136 A.10 Session Three Pre-Task Instructions ...................... 139 A.11 Session Three Task Instructions ........................ 141 B SELECTED NEUROBLOCK PROGRAMS .................... 145 B.1 Session One Pre-Task Program ........................ 145 B.2 Session One Task Program ........................... 146 B.3 Session Two Pre-Task Program ........................ 147 B.4 Session Two Task Program ........................... 149 B.5 Session Three Pre-Task Program ....................... 151 B.6 Session Three Task Program .......................... 153 REFERENCES ....................................... 155 BIOGRAPHICAL SKETCH ................................ 164 6 LIST OF TABLES Table page 3-1 EEG Frequency Bands related to various mental states. .............. 58 3-2 Stage Components. .................................. 62 3-3 Block Components. .................................. 62 4-1 Learning barriers coding scheme. .......................... 93 4-2 Programming self-efficacy questions. ........................ 94 4-3 SUS questions ..................................... 97 4-4 BCI self-efficacy questions .............................. 98 4-5 Session one selected responses related to the five main categories. ........ 101 4-6 Session two selected responses related to the five main categories. ........ 111 7 LIST OF FIGURES Figure page 2-1 BCI system design. .................................. 24 2-2 Operator, data acquisition, signal processing, and user application programs which are the core of BCI2000. ........................... 26 2-3 Configuration menu for BCI pipeline steps. .................... 27 2-4 Screenshot of Simulink with rtsBCI and feedback application. .......... 28 2-5 BCILAB GUI panels used for visualizing EEG data, calibrating models, scripting, and modifying evaluation approaches. ........................ 30 2-6 Pyff GUI that BCI experimenters may use to select, modify, and control feedback applications. ...................................... 31 2-7 2D visualization of signals and time-frequency dynamics. ............. 33 2-8 OpenViBE visual programing GUI and 3D spatial topography. .......... 34 2-9 Student showing BCI-based science fair project which used OpenViBE to control robotic arm at the United States Nation's capital. ................. 36 2-10 Alice 3 Scene editor, program preview, and block editor. ............. 38 2-11 Scratch interface. ................................... 40 2-12 App Inventor designer interface. ........................... 42 2-13 App Inventor block interface. ............................ 42 2-14 Block-based programming environment designed for clinicians. .......... 44 2-15 Block-based programming environment designed for physical prototyping. .... 45 2-16 A Visual Programming Framework for Wireless Sensor Networks in Smart Home Applications. ..................................... 45 3-1 Interaxon Muse. A) EEG headset, B) Muse electrode positions, and C) User wearing Muse. ..................................... 49 3-2 System design. .................................... 50 3-3 2D signal visualization. ................................ 53 3-4 3D topographic map. ................................. 53 3-5 EEG apparatuses. A) Neurosky Mindwave, B) Interaxon Muse, C) Emotiv Insight, D) Emotiv Epoch, E) OpenBCI Ultra Cortex \Mark IV", and F) g.Nautilus .. 55 8 3-6 User wearing Interaxon Muse EEG apparatus. ................... 56 3-7 User building neurofeedback application with NeuroBlock. ............ 57 3-8 Web application interface with affective state data viewer selected. ........ 60 3-9 Web application interface with sprite viewer selected. ............... 60 3-10 Stage figure. ...................................... 62 3-11 Example of blocks separated. ............................ 63 3-12