SHIFTING GAZES WITH VISUAL PROSTHESES: LONG-TERM HAND-CAMERA COORDINATION by Michael Patrick Barry A dissertation submitted to Johns Hopkins University in conformity with the requirements for the degree of Doctor of Philosophy. Baltimore, Maryland March 2018 This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042 , USA. Purpose: Prosthetic vision is young, and many aspects of its use remain unexplored. Hand- camera coordination, the prosthetic correlate of hand-eye coordination, relies heavily on how the camera is aligned with the eye. It is unknown whether users of prostheses can adapt to using misaligned cameras, or whether requirements for proper alignment remain constant over time. Methods: Four blind subjects implanted with Argus II retinal prostheses participated in this study. Each subject attempted to touch a single 4°–7° white target that was randomly located on an otherwise black touchscreen in a target localization task. Touch response accuracy was used to determine the necessary adjustment to eye-camera alignment, the optimal camera alignment position (OCAP). Subjects attended over 100 sessions across up to 5.3 years. S1–S3 were given misaligned cameras for over 1 year. Adaptation was measured through changes in localization errors. Outside that period of intentional misalignment, cameras were aligned to maximize localization accuracy. During the final year, localization tasks were performed in alternation with eye tracking. S2–S4 also participated in 1-day experiments with simultaneous eye tracking and target localization. Results: Subjects were not able to significantly reduce localization error when cameras were misaligned. When trying to maximize localization accuracy, necessary OCAPs changed significantly over time. OCAP trend directions within days and trial runs matched changes between the beginnings of days and runs. Changes between the end of a day or run and the beginning of the next tended to point in the opposite direction of the previous trend, indicating a reset of OCAP changes. ii Changes in eye orientations correlated significantly with changes in OCAPs. Eye-orientation trends displayed the same reset behavior between days and runs as OCAPs. Simultaneous eye tracking and localization showed agreement between eye-orientation and localization-error trend directions. Adjusting camera alignment with eye-tracking data slowed changes in localization errors. Conclusions: Users of current visual prostheses cannot passively adapt to camera misalignments. OCAPs are not constant with time. Prosthesis users who desire maximum pointing accuracy will require regular camera realignments. Camera alignments based on eye tracking can reduce both transient and long-term changes in localization that are related to eye movements. Readers: Gislin Dagnelie, Robert W. Massof, Xiaoqin Wang iii Acknowledgements While many people directly and indirectly contributed to the work described in this dissertation, none were more essential than the participants who donated their time and effort to the underlying experiments. These participants risked themselves, both physically and emotionally, in agreeing to help the field of visual prosthetics by having retinal prostheses surgically implanted. The prosthesis recipients did not regain anything like normal vision by undergoing the procedure, but generously hoped to provide knowledge and experience necessary to develop better visual prostheses. Beyond the already lengthy demands of participating in the feasibility study for the Argus II retinal prosthesis, these participants volunteered for additional studies that explored finer aspects of prosthetic vision. S1, S2, and S3 each devoted over 100 hours just to this dissertation’s research, and S3 went so far as to dedicate over 400 hours. Their work was exhausting and repetitive, and I cannot thank them enough for their generosity and reliability. Gislin Dagnelie, Ph.D. has been an outstanding mentor throughout this entire process. He introduced me to the realm of visual prostheses as an undergraduate student, and has guided me unwaveringly through the completion of my graduate education. Gislin has been a source of insight for not only the science behind vision restoration and low-vision rehabilitation, but also finer matters of culture and the simplicities of optimism and kindness. Robert Massof, Ph.D. and Xiaoqin Wang, Ph.D. kindly agreed to serve on my thesis committee and provided valuable feedback that improved both the substance and presentation of this dissertation. I did not update them on research progress as often as may have been optimal for debating experiment minutiae, but they graciously still took the time to understand everything iv and point out weak spots in my work. This dissertation’s topic did not directly relate to either of their primary research focuses, but they still found time in their busy schedules to help me improve and complete my project. While working under Dr. Dagnelie, many people affiliated with the lab provided guidance and support through numerous projects. Veronika Mueller was my first direct mentor, who introduced me to prosthetic vision simulations as part of her graduate studies. H. Christiaan Stronks, Ph.D. shared much of the implantee-testing experience, and offered additional perspectives on working through Ph.D. training. Olukemi Adeyemo held everything together by coordinating both subjects and experimenters, making sure everyone was where they needed to be and that they had all the necessary resources. Liancheng Yang kept all of the custom equipment operational and wrote the programs used for the tower-mounted eye-tracking tests. Low-vision rehabilitation specialist Duane Geruschat, Ph.D. was heavily involved with other projects that examined the abilities of Argus II users, and was always very inclusive and supportive during my years of training. Dr. Geruschat helped me appreciate that the artificial vision experienced by prosthesis users is not as good as we tend to hope or desire. Pamela Jeter, Ph.D., while our research areas had minimal overlap, always brought a nice atmosphere to lab meetings and helped me to recognize the place of prosthetic vision in the wider realm of low vision. I also had the pleasure of working with many trainees and students who assisted the lab over shorter time periods, including Lauren Dalvin, M.D., Jordan Matelsky, Sophia Diaz-Aguilar, Ali Abdulkarim, Akudo Umeh, Reem El-Husseiny, Alfred Vinnett, and Neeraj Ochaney. These students each brought a unique perspective as they assisted me with experiments, and they certainly improved the data collection process. Roksana Sadeghi has provided similar assistance v in recent months, and I wish her all the best as she continues her Ph.D. training under Dr. Dagnelie. Second Sight Medical Products, Inc., beyond providing the technological basis for this dissertation’s research, supplied essential training, materials, and experimental support. I learned much about how to test and configure the Argus II from Second Sight’s former and current employees, including Arup Roy, Ph.D., Jessy Dorn, Ph.D., Jianing Wei, Ph.D., Avi Caspi, Ph.D., Varalakshmi Wuyyuru, Uday Patel, Ph.D., Ashish Ahuja, Ph.D., and Punita Christopher, Ph.D. The original target localization program, used as the basis for most measurements in this dissertation, was developed by Second Sight for the Argus II Feasibility Study. Jordan Matelsky customized the program for my studies in 2014, and Varalakshmi Wuyyuru facilitated the development of a more polished research program that has been used for target localization since 2015. Gayatri Kaskhedikar, Ph.D. and Liancheng Yang spent much time determining the best ways to use Dr. Dagnelie’s tower-mounted eye tracker. I had the fortune using the eye tracker after they worked out most of the bugs, which made adaptations for my experiments significantly easier. Dr. Roy and Dr. Caspi collaborated with Paul Rosendall, Jason Harper, Kapil Katyal, and others at the Johns Hopkins University Applied Physics Laboratory to integrate the use of eye-tracking glasses into Argus II stimulation. Although few target localization tests with the eye-tracking glasses were performed, the data from that setup were critical for explaining other results in this dissertation. My entry into Ph.D. training, and the success I found along the way, would not have been possible without the inspiration and guidance I received from numerous educators throughout my academic career. My neuroscience professors, particularly Stewart Hendry, Ph.D., Linda vi Gorman, Ph.D., and Brenda Rapp, Ph.D., exposed me to the wonderful complexities of the nervous system. They fueled my desire to further investigate the nervous system and its workings, particularly how one might interact with it through neuroprostheses. Takashi Yoshioka, Ph.D., as the director of my B.A./M.S. program, expanded my insight on research and publications, as well as bolstered my comfort with research presentations. Long before going college, I also had the benefit of having Ronald Heitmann and Linda Barnfield as middle school teachers. Mr. Heitmann and Mrs. Barnfield were committed to teaching their students about the world and helping them grow as individuals. Through efforts like the “We Didn’t Start the Fire” project and tours of Europe, they contributed greatly to my perseverance and