TOWARDS LOW-COST SPATIAL HAPTICS: BRAKE-BASED SHAPE DISPLAYS AND AUTOMATIC CONTENT AUTHORING A DISSERTATION SUBMITTED TO THE DEPARTMENT OF ELECTRICAL ENGINEERING AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Kai Zhang August 2020 © 2020 by Kai Zhang. All Rights Reserved. Re-distributed by Stanford University under license with the author. This work is licensed under a Creative Commons Attribution- Noncommercial 3.0 United States License. http://creativecommons.org/licenses/by-nc/3.0/us/ This dissertation is online at: http://purl.stanford.edu/kh451yb0256 ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Sean Follmer, Primary Adviser I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Juan Rivas-Davila I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Gordon Wetzstein Approved for the Stanford University Committee on Graduate Studies. Stacey F. Bent, Vice Provost for Graduate Education This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file in University Archives. iii Abstract Haptic technology can significantly enhance the richness and realism of user interaction by providing the experience of physical touch. It is broadly used in various application scenarios including game controllers, surgical robots, design and modeling tools, and accessibility for Blind people. Tactile dis- plays are a class of haptic devices that can render tactile effects to the users, for example reproducing the skin deformation of a user when in contact with a real object, allowing the user to feel vibration, pressure, touch, and texture. However, key challenges currently limit the widespread use of tactile displays including cost, spatial resolution, fabrication complexity, shape rendering flexibility, and refresh rate. Beyond these hardware limitations, authoring of rich tactile content is another chal- lenge. Although manual authoring methods have been developed, methods that can automatically translate vast amounts of information into meaningful tactile stimuli are highly desirable. In this thesis, I present my work on addressing the hardware and content authoring challenges of the tactile displays. The first part of my thesis work focuses on the use of electrostatic brakes and clutches to enable low cost tactile displays. I investigated the use of low-cost electrostatic adhesion in the context of high-resolution, refreshable 2.5D tactile shape display. I modeled, fabricated, and characterized the contact force, refresh rate and robustness of the brakes. A user study conducted using an integrated 4×2 tactile shape display based on the brakes showed similar user shape recogni- tion performance using our device and the 3D printed shapes. In addition to the bed-of-nails tactile displays mentioned above, I also explored the formable-crust 2.5D tactile shape displays using aux- etic materials. A simulation model and algorithms were developed to investigate the characteristics of the shape display and render a target shape. An experimental prototype was constructed to verify the simulation results. The auxetic 2.5D tactile shape display has the advantages of low cost, large displacement range, and roll-to-roll fabrication compatibility. In the second part of my thesis, I investigated methods to address the content authoring challenges of tactile displays. A pipeline was built to automatically generate spatial tactile effects by analyzing cross-modality features in a video. We believe the above results help to improve the accessibility of the haptic technology to a broader audience. iv Acknowledgments Different from most of the other types of occupations, Ph.D. students serve as the brave explorers to expand the border of human knowledge to the new territory. Since my first day in Stanford in 2014, I witnessed my progress from a naive bachelor to a more experienced researcher with a deeper understanding of this world of technology. It was impossible for me to accomplish my Ph.D. journey without the help and encouragement of so many mentors and friends. I would like to thank these people with my sincere gratefulness. First of all, I would like to thank my Ph.D. advisor, Prof. Sean Follmer. In many ways, the life of a Ph.D. student highly depends on his/her Ph.D. advisor. Thus, I feel very fortunate to be a member in Sean's group. No matter it was for building roadmap for the Ph.D. thesis, deciding framework of each research project, or solving the detailed technical challenges in an experimental system, Sean always gave me very helpful advice. Exploring a brand new mechanism in the research is like searching for a path across the ocean, Sean's encouragement, patience and optimism always helped me to overcome the frustration along the way. Besides research, Sean also gave me lots of valuable advice on many important things including career development, classes, and personal health during the COVID-19 pandemic. I believe Sean's advice not only helped me to be a more experienced researcher but also a better and stronger person to deal with the challenges in my future life. I appreciate every member of my defense committee for their valuable feedback: Prof. Gordon Wetzstein, Prof. Juan Rivas-Davila, Prof. Allison Okamura, and Prof. Larry Leifer. Their valuable comments helped me to improve this thesis. I would also like to thank all the collaborators in my Ph.D. research. Lawrence Kim gave me lots of helpful suggestions on the auto haptics project. I learned a lot from his insights in analyzing a research problem. His internet influencer dog, Boba, also serves as a lab mascot. Eric Gonzalez and Jianglong Guo helped me significantly improved my project on electrostatic tactile display. Their rigor in clarifying every detail in the research is impressive. Yipeng Guo is a very effective experi- mental collaborator. My generation gap with younger graduate students was definitely narrowly by working with such an energetic collaborator. Jan Friedrich from Berlin provided very valuable help for the auxetic shape display projects. As a bonus, I also enjoyed many photos of the European v local views shared by Jan. I also appreciate other people for their valuable discussions along the way. Yapeng Tian kindly explained lots of details in his previous project so that I can have a jump start on mine. Jackie Yang and Abe Davis provided many inspiring ideas when I was brainstorming my new project on auto haptics. As a researcher familiar with flexible structure fabrication, Amy Han showed me many handy techniques for building the auxetic shape display. I thank all the labmates in the Shape Lab. This is a fantastic research group with awesome members. Many of them have a best paper award or honorable mention in one or more academic conferences. I always learned a lot from their insightful comments in the group meetings, peer paper reviewing activities and academic discussions everyday. The culture of the Shape Lab is also unique and warm. People held all kinds of interesting events including ceramic painting, board games, pumpkin curving, online drawing, and traditional Thursday tea time at two thirty (TTTTTT). I also appreciate the help from Andrea Brand-Sanchez and Renee Chao. As a student working closely with hardware systems, they provided me with tremendous help on correctly purchasing everything as soon as possible. I appreciate the guidance and support by Prof. Jonathan A. Fan. I did most of my master's research in the Fan Lab. Jon not only gave me tons of sharp advice in the plasmonic research project I was working on at that time, but also patiently showed me many methodology about doing good research. I would also thank my undergraduate research advisor Prof. Zhongfan Liu and Prof. Hailin Peng from Peking University. Although I was just a junior student with very few research experience, they gave me a chance to access a real research lab. Without this opportunity, I will not be able to start building my research skills as a college student. I would like to thank all of my friends in Stanford. Tao Jia, Haoli Guo and Yujie Zheng have been my friends since our first year as graduate students. We have shared so many memorable moments along the Ph.D. journey. I thank Hongquan Li, Shi Dong and Kuan Fang for their support and help during the hard time. It's a treasure to have these people around. It's also a great pleasure to meet other friends in Stanford: He Wang, Zheng Cui, Jinye Zhang, Li Tao, Minda Deng, Yu Miao, Shang Zhai, Zhouchangwan Yu, Sophia Qin. I cannot put everyone's name here due to space limitation. However, I would like to express my gratitude to all my friends in Stanford. Without their accompany and support, I will not be able to be as optimistic during my Ph.D. journey. Last but the most important, I would like to thank my parents for everything they have done to support me for the entire six years of my graduate student life. Although we were separated by the Pacific Ocean for most of the time, I always felt their accompany along the way. vi Contents Abstract iv Acknowledgments v 1 Introduction 1 1.1 Motivation . .1 1.2 Contribution . .2 1.3 Prior Work . .5 1.3.1 2D Tactile Displays .