Decoding Upper-Limb Kinematics from Electrocorticography Ewan Scott Nurse ORCID: 0000-0001-8981-0074 Submitted in partial fulfillment of the requirements of the degree of Doctor of Philosophy (with coursework component) December 2017 Department of Biomedical Engineering Melbourne School of Engineering The University of Melbourne Abstract Brain-computer interfaces (BCIs) are technologies for assisting individuals with motor impairments. Activity from the brain is recorded and then processed by a computer to control assistive devices. The prominent method for recording neural activity uses microelectrodes that penetrate the cortex to record from localized populations of neurons. This causes a severe inflammatory response, making this method unsuitable after approximately 1-2 years. Electrocorticog- raphy (ECoG), a method of recording potentials from the cortical surface, is a prudent alternative that shows promise as the basis of a clinically viable BCI. This thesis investigates aspects of ECoG relevant to the translation of BCI devices: signal longevity, motor information encoding, and decoding intended movement. Data was assessed from a first-in-human ECoG device trial to quantify changes in ECoG over multiple years. The mean power, calculated daily, was steady for all patients. It was demonstrated that the device could consistently record ECoG signal statistically distinct from noise up to approximately 100 Hz for the duration of the study. Therefore, long-term implanted ECoG can be expected to record movement-related high-gamma signals from humans for many years without deterioration of signal. ECoG was recorded from patients undertaking a two-dimensional center- out task. This data was used to generate encoder-decoder directional tuning models to describe and predict arm movement direction from ECoG. All four patients demonstrated channels that were significantly tuned to the direction of motion. Significant tuning was found across the cortex and was not fo- cused on primary motor areas. Decoding significantly above chance with a population-vector approach was achieved in three of the four patients. Decod- ing accuracy was significantly improved by weighting the population vector by each channel's tuning signal-to-noise ratio. Hence, directional tuning exists in i ii high-frequency ECoG during movement preparation, and movement angle can be decoded using population vector methods. Having confirmed the existence of direction-related information in the recorded data, artificial neural network models were created to decode intended move- ment direction. A convolutional neural network (CNN) model had significantly higher decoding accuracy than a fully connected model for all four patients for decoding movement direction. Training models on data from all patients and testing on a single patient improved decoding performance for all but the best performing patient with the CNN model. Decoding using data from multiple time-points with a CNN model and averaging the results boosted accuracy when using the mode of the outputs. Overall, it was demonstrated that ar- tificial neural network models can decode intended movement direction from ECoG recordings of a two-dimensional center-out task. This thesis presents results that demonstrate ECoG has the desired signal properties for a clinically-relevant BCI. ECoG is shown to be robust over mul- tiple years, encode direction-related information and can be decoded with high accuracy. Declaration I hereby declare that this thesis comprises only my original work towards the degree of Doctor of Philosophy at the University of Melbourne. All work included in this thesis, except where acknowledged in the Statement of Au- thorship, is my original work. All other work has been duly acknowledged. This thesis is fewer than the maximum word limit of 100,000 words exclu- sive of tables, figures, bibliographies and appendices. Signed: Ewan Nurse December 2017 iii Statement of Authorship I hereby declare that this thesis and the work presented in it are original and generated by me as the result of my own investigations. Except where acknowledged below, I was responsible for the experimental design, data col- lection, data analysis, software programming, and generation of images and graphical data. The data analyzed in Chapter 2 was collected as part of a first-in-human device trial conducted by NeuroVista Corporation (Seattle, USA) and the Melbourne University Epilepsy Group. This study is presented in Cook et al. (2013) Due to the multidisciplinary nature of clinical neurosciences, it would not have been possible to collect the data, described in Chapter 3 and analyzed in Chapters 4 and 5, if it were not for the contributions detailed below. • Dr. Dean Freestone, Dr. Alan Lai and Mr. Simon Vogrin assisted with center-out task data collection. • A. Prof. Wendyl D'Souza conducted the clinical functional electrical stimulation • Mr. Simon Vogrin supervised the acquisition of all structural imaging and performed the CT-MRI co-registration. iv v The following chapters contain published works that have resulted from the research presented in this thesis. • Chapter 2 is a modified version of the peer-reviewed published article: Nurse, E.S., John, S.E., Freestone, D.R., Oxley, T.J., Ung, H., Berkovic, S. F., O'Brien, T.J., Cook, M.J. and Grayden, D.B., 2017, Consistency of long-term of subdural electrocorticography in humans, IEEE Transactions on Biomedical Engineering, 10.1109/TBME.2017.2768442. This work is also related to the peer-reviewed published article: Ung, H., Baldassano, S.N., Bink, H., Krieger, A.M., Williams, S., Vitale, F., Wu, C., Freestone, D., Nurse, E., Leyde, K., Davis, K.A., Cook, M. and Litt, B. 2017. Intracranial EEG fluctuates over months after implanting electrodes in human brain. Journal of Neural Engineering, 14(5), p.056011. • Chapter 4 is modified version of the peer-reviewed published article: Nurse, E.S., Freestone, D.R., Oxley, T.J., Ackland, D.C., Vo- grin, S.J., Murphy, M., O'Brien, T.J., Cook, M.J. and Grayden, D.B., 2015, Movement related directional tuning from broadband electrocorticography in humans. In 7th International IEEE/EMBS Conference on Neural Engineering (NER), 22-24 April 2015, (pp. 33-36). • Chapter 5 has been influenced by the following peer-reviewed published articles: Kiral-Kornek, I., Mendis, D., Nurse, E.S., Mashford, B.S., Free- stone, D.R., Grayden, D.B. and Harrer, S., 2017, TrueNorth-enabled real-time classification of EEG data for brain-computer interfacing. In 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 11-15 July 2017, (pp. 1648-1651). vi Nurse, E., Mashford, B.S., Yepes, A.J., Kiral-Kornek, I., Harrer, S. and Freestone, D.R., 2016, Decoding EEG and LFP signals us- ing deep learning: heading TrueNorth. In Proceedings of the ACM International Conference on Computing Frontiers, 16-18 May 2016, (pp. 259-266). Nurse, E.S., Karoly, P.J., Grayden, D.B. and Freestone, D.R., 2015. A generalizable brain-computer interface (bci) using machine learning for feature discovery. PloS One, 10(6), p.e0131328. • A number of conference abstracts have also been published: Nurse, E.S., John, S.E., Freestone, D.R., O'Brien, T.J., Cook, M.J. and Grayden, D.B., Reliability of Long-Term of Subdural Elec- trocorticography in Humans, International Conference on Medical Bionics, 20 - 23 November, 2016. Nurse, E.S., John, S.E., Freestone, D.R., O'Brien, T.J., Cook, M.J. and Grayden, D.B., Spectral features in long-term ambulatory electrocorticography, Melbourne Symposium on Brain Computer In- terfaces and Rehabilitation, 9 October, 2015. Nurse, E.S., John, S.E., Freestone, D.R., O'Brien, T.J., Cook, M.J. and Grayden, D.B., Long-term spectral properties of human electrocorticography, NeuroEng - the Australiasian Workshop on Computational Neuroscience, 26 - 28 August, 2015. Nurse, E.S., Oxley, T.J., Freestone, D.R., Vogrin, S., Murphy, M., Morokoff, A., O'Brien, T.J., Cook, M. J, and Grayden, D.B., Cortically localised directional tuning of arm movements: recording during ECoG monitoring, Annual Meeting of Epilepsy Melbourne, 17 - 18 September, 2014. Acknowledgements It is my pleasure to thank those who have made this thesis possible. Thank you all for your time, encouragement, thoughts and love: I owe my deepest gratitude to Prof. David Grayden and Dr. Dean Free- stone for introducing me to the field of neural engineering and encouraging me to undertake a PhD. There are too many individual things to thank David for, so I will just say thank you for being such an encouraging and thoughtful su- pervisor. Thank you Dean for being my cheer-leader, for pulling me out of my comfort zone when I needed it, and your down-to-earth style of leadership. My profound thanks to Profs. Mark Cook and Terence O'Brien for the privilege of allowing me to work in a clinical setting and see the most important reason we do neuroscience. It has been an honour to work with people whose careers have touched so many lives. Thanks to Dr. Thomas Oxley for the opportunities you've given me, your energy, and for always having the big picture in mind. Thank you to all my supervisors for their unwavering generosity, support, and patience when I wandered. This project was greatly enriched by the mentorship of Sam John and the other members of the Vascular Bionics Laboratory: Nick, Steve, Gil and Amos. It has been a joy to be involved in such a unique project. I am greatly indebted to my co-workers from the NeuroEngineering Lab- oratory and beyond for their friendship, broad expertise, and advise. This includes: Pip, Tim, Kerry, Shannon, Rob, Isabell, Matias, Yan, Elma, Tania, Daniel, Giulia, Peter, Andre, Kat, Miao, William, Colin, Farhad, Pat and Brayden. The clinical aspects of this work were greatly supported by Simon `Vogs' Vogrin, Alan Lai, Wendyl D'Souza and Linda Seiderer. Thank you for all your vii viii help navigating the `wonderful world of EEG' (a direct and deeply concerning quote from Vogs).