Enhancing Upper Limb Prostheses Through Neuromorphic Sensory Feedback
Total Page:16
File Type:pdf, Size:1020Kb
Enhancing Upper Limb Prostheses Through Neuromorphic Sensory Feedback by Luke E. Osborn A dissertation submitted to The Johns Hopkins University in conformity with the requirements for the degree of Doctor of Philosophy. Baltimore, Maryland December, 2018 c Luke E. Osborn 2018 All rights reserved Abstract Upper limb prostheses are rapidly improving in terms of both control and sensory feedback, giving rise to lifelike robotic devices that aim to restore function to amputees. Recent progress in forward control has enabled prosthesis users to make complicated grip patterns with a prosthetic hand and nerve stimulation has enabled sensations of touch in the missing hand of an amputee. A brief overview of the motivation behind the work in this thesis is given in Chapter 1, which is followed by a general overview of the field and state of the art research (Chapter 2). Chapters 3 and 4 look at the use of closed loop tactile feedback for improving prosthesis grasping functionality. This entails development of two algorithms for improving object manipulation (Chapter 3) and the first real-time implementation of neuromorphic tactile signals being used as feedback to a prosthesis controller for improved grasping (Chapter 4). The second half of the thesis (Chatpers 5 - 7) details how sensory information can be conveyed back to an amputee and how the tactile sensations can be utilized for creating a more lifelike prosthe- sis. Noninvasive electrical nerve stimulation was shown to provide sensations in multiple regions of the phantom hand of amputees both with and without targeted sensory reinnervation surgery (Chapter 5). A multilayered electronic dermis (e-dermis) was developed to mimic the behavior of receptors in the skin to provide, for the first time, sensations of both touch and pain back to an am- ii ABSTRACT putee and the prosthesis (Chapter 6). Finally, the first demonstration of sensory feedback as a key component of phantom hand movement for myoelectric pattern recognition shows that enhanced perceptions of the phantom hand can lead to improved prosthesis control (Chapter 7). This work provides the first demonstration of how amputees can perceive multiple tactile sensations through a neuromorphic stimulation paradigm. Furthermore, it describes the unique role that nerve stimula- tion and phantom hand activation play in the sensorimotor loop of upper limb amputees. iii Thesis Committee Primary Readers Nitish V. Thakor, PhD (Primary Advisor) Professor Department of Biomedical Engineering Department of Electrical and Computer Engineering Johns Hopkins School of Medicine Professor and Director Singapore Institute for Neurotechnology Department of Electrical and Computer Engineering Department of Biomedical Engineering National University of Singapore Ralph Etienne-Cummings, PhD Professor and Chair Department of Electrical and Computer Engineering Johns Hopkins University Charles E. Connor, PhD Professor Department of Neuroscience Johns Hopkins University School of Medicine iv Acknowledgments I would like to thank Dr. Nitish Thakor for his guidance, encouragement, and patience over the years. Without his unwavering dogma to focus and finish I wouldn’t be where I am today. Thanks to everyone in the Neuroengineering & Biomedical Instrumentation Lab for their input, criticism, and support. Joseph Betthauser for his friendship and not holding back when critiquing my figures. Gyorgy˝ Lévay for his willingness and patience to be a subject for countless hours of experiments and for his constant desire to eat wings. Guy Hotson for his senior-grad-student wisdom and in- spiringly large bowls of food. Chris Hunt for his coding and Bluetooth wizard skills. Harrison Nguyen and Chris Shallal for being the liaisons to the undergrad world. Mark Iskarous for stepping up and inheriting the defacto leadership role of all things demo, sensors, lab visit, and logistics related. Darshini Balamurugan, Teja Karri, and Avinash Sharma for their curiosity and enthusiasm in picking my brain. To all the undergraduate and high school students who worked in our lab and made sure I didn’t slack off. The entire Infinite Biomedical Technologies team, especially Megan Hodgson, Martin Vilariño, and Damini Agarwal, for their willingness to help out a poor graduate student at a moments notice. Dr. Rahul Kaliki for his longtime friendship, mentorship, and football organization skills. The wonderful people in the Department of Biomedical Engineering at JHU who have supported me over the last several years, but most especially Sam Bourne for his couth and always being a source of insight and relief. I must also thank my parents for teaching me how to read and limiting their questions about when I would graduate and get a job. My siblings Paul (and Alicia), Pamela, John Mark (and Maria), Mary Gail (and Joseph), Timothy, and Laura Ruth for setting the bar so high. Ayushi Sinha for her constant emotional support and understanding all the late nights I spent in the lab. Thanks to all of those close to me who have managed to put a smile on my face and remind me that there is indeed a world outside of these laboratory walls. Thanks to all the researchers, creators, and innovators before me who developed the technology and tools I used in my research. I’d also like to thank you, the reader, for taking an interest in my work. I hope you find this thesis as wildly riveting as I do. v Dedication To my parents who never gave up and taught me the value of dedication, patience, and hard work vi Contents Abstract ii Thesis Committee iv Acknowledgments v List of Tables xii List of Figures xiii 1 Introduction 1 1.1 Motivation . 1 1.2 Original contributions . 1 1.3 Publications . 2 1.4 Thesis organization . 4 2 Neural Prostheses & Literature Review 6 2.1 Overview . 6 2.2 Background . 7 2.3 Prosthesis fundamentals . 8 2.3.1 Neural interface . 10 2.3.2 External hardware interface . 11 2.4 Motor prosthesis . 12 2.4.1 Movement signals & decoding . 13 vii CONTENTS 2.4.2 Targeted muscle reinnervation (TMR) & osseointegration . 15 2.4.3 State of the art . 15 2.5 Sensory prosthesis . 16 2.5.1 Touch sensing in humans . 17 2.5.2 Sensors and advanced materials . 18 2.5.3 Sensory feedback . 19 Tactile . 19 Proprioception . 21 2.5.4 Neuromorphic models . 22 2.5.5 State of the art . 23 3 Closed-Loop Tactile Feedback in Upper Limb Prostheses 25 3.1 Overview . 25 3.2 Introduction . 26 3.3 Materials and ethods . 30 3.3.1 Textile force sensor . 30 3.3.2 Neuromimetic algorithms . 31 Compliant Grasping control . 31 Slip Prevention control . 34 3.4 Experimental methods . 35 3.4.1 Hardware and data collection . 36 3.4.2 Able-bodied experiments . 37 Compliant Grasping . 38 Slip Prevention . 39 3.4.3 Amputee experiments . 39 3.5 Results . 40 3.5.1 Compliant Grasping . 40 Able-bodied subjects . 41 Amputee subjects . 42 3.5.2 Slip prevention . 42 Able-bodied subjects . 43 Amputee subject . 44 viii CONTENTS 3.6 Discussion . 45 3.6.1 Compliant Grasping . 45 Able-bodied subjects . 45 Amputee subjects . 46 3.6.2 Slip prevention . 47 Able-bodied subjects . 48 Amputee subject . 48 3.6.3 Active touch sensing . 49 3.6.4 General considerations . 50 Subjective evaluation . 52 3.7 Conclusion . 52 4 Grip Force Modulation Using Neuromorphic Tactile Sensing 54 4.1 Overview . 54 4.2 Introduction . 55 4.3 Model & Methods . 57 4.4 Experiments & Results . 61 4.5 Discussion . 62 4.6 Conclusion . 64 5 Targeted Sensory Feedback in Upper Limb Amputees 65 5.1 Overview . 65 5.2 Introduction . 66 5.3 Methods & Experiments . 68 5.3.1 Perception Experiments . 70 5.3.2 Neuromorphic Sensor Model . 71 5.4 Results . 72 5.5 Discussion . 73 5.6 Conclusion . 75 6 Multilayered e-Dermis for Perceiving Touch and Pain 77 6.1 Overview . 77 ix CONTENTS 6.2 Introduction . 78 6.3 Results . 82 6.3.1 Biologically inspired e-dermis . 82 6.3.2 Touch and pain perception . 83 6.3.3 Neuromorphic transduction . 86 6.3.4 Prosthesis tactile perception and pain reflex . 89 6.3.5 User tactile perception . 92 6.4 Discussion . ..