Human and Mobile Robot Tracking in Environments with Different Scales Dissertation Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By Qiang Zhai, B.S. Graduate Program in Computer Science and Engineering The Ohio State University 2017 Dissertation Committee: Dr. Dong Xuan, Advisor Dr. Chunyi Peng, Co-Advisor Dr. Yuan F. Zheng Dr. Teng Leng Ooi c Copyright by Qiang Zhai 2017 Abstract In the near future, we envision many mobile robots working for or interacting with humans in various scenarios such as guided shopping, policing, and senior care. In these scenarios, tracking humans and mobile robots is a crucial enabling technology as it provides their continuous locations and identities. We study each kind of tracking individually as humans and mobile robots differ in appearance, mobility, capability of computation and sensing, and feasibility of cooperation with tracking systems. In addition to these distinctions, the scale of a tracking system's environment is another key factor that determines the appropriate tracking algorithm. In small scale environments such as halls and rooms, tracking systems need to be lightweight with real-time performance. In large scale environments such as campuses and entire buildings, tracking systems face complex and noisy backgrounds. Tracking systems need to be robust and scalable. This dissertation examines problems for human and mobile robot tracking in both small and large scale environments in order to enable future human-robot applications. First, we study human tracking. In small scale environments like halls, multiple cameras are typically deployed to monitor most areas. We need to track multiple humans across multiple cameras accurately in real-time. We present VM-Tracking, a tracking system that achieves these goals. VM-Tracking aggregates motion sensor information from humans' mobile devices, which accompany them everywhere, and ii integrates it with visual data from physical locations. In large scale human track- ing, determining humans' identities among different visual surveillance scenes is a major challenge, especially with noisy and incorrect data. Since people's mobile de- vices connect to widespread cellular networks, we propose EV-Matching to address these practical challenges. EV-Matching is a large scale human tracking system that matches humans recorded in both cellular network and visual surveillance datasets. It achieves efficient and robust tracking. Next, we study mobile robot tracking. Similar to human tracking, visual cameras are widely available in small scale environments. Mobile robots are capable of compu- tation and cooperation but their on-board sensing capacities have room for improve- ment. Thus, we present S-Mirror, a novel approach using infrastructural cameras to “reflect” sensing signals towards mobile robots, which greatly extends their sensing abilities. S-Mirror is a lightweight infrastructure that is easily deployed as it lever- ages existing visual camera networks, leaving most computation to robots. In large scale mobile robot tracking, scalability is a major challenge as infrastructure does not cover all areas and augmenting it is expensive. Since robots have built-in sensing capabilities, we propose BridgeLoc, a novel vision-based robot tracking system that integrates both robots' and infrastructural camera views. We achieve accurate view bridging via visual tags, images with special patterns. We develop a key technology, rotation symmetric visual tag design, that greatly extends BridgeLoc's scalability. In this dissertation, we study human and mobile robot tracking in small and large scale environments. We design and implement all of the above systems. Our real- world experimental evaluations show the advantages of our work and demonstrate its potential for human-robot applications. iii This is dedicated to my wife and my parents iv Acknowledgments Many people have helped me to reach this point after a long and fruitful journey. Here I would like to express my sincere gratitude to all the people who have helped me during my Ph.D. study. First I would like to express my sincere gratitude to my advisor, Dr. Dong Xuan for his guidance and support throughout my Ph.D. study at The Ohio State University (OSU). In fact, I came to OSU without funding support or master degree. Normally a professor was unlikely to acquire a student without master degree to his/her research group with funding support. However, Dr. Xuan gave me an opportunity to work with him for a quarter and after that quarter he decided to give me continuous funding support to finish my Ph.D. study. I was always grateful to him for the opportunity and trust he gave to me. During my Ph.D. study, Dr. Xuan had been providing me with a lot of insightful advice and helping me out of many difficulties in my research. His diligence and enthusiasm towards research has deeply influenced me. Beyond research, Dr. Xuan is also a good friend in daily life. His suggestions, patience and support helped me overcome many difficult and unexpected situations. All the wonderful Thanksgiving, Christmas, and Chinese New Year parties at Dr. Xuan's home left wonderful memories in my mind. I feel so blessed to have Dr. Xuan as my advisor. v I also would like to greatly thank my co-advisor, Dr. Chunyi Peng. I met Dr. Peng in her faculty talk of Computer Science and Engineering (CSE) Department at OSU. I was impressed by her excellent achievements and huge enthusiasm on academic research. After she became a faculty member in CSE department, I had an opportunity to work with her on a research paper. During this work, she taught me many necessary skills for research including systematic way of defining a problem and solid expression of writing a paper. After our first collaboration, she offered me her co-advising, which was a big affirmation to me. I certainly took this honor and kept learning from her. Beyond research, Dr. Peng is also a good friend to me. She cared about my daily life and gave me many good suggestions to my future career. I am fortunate to have her as my co-advisor. Beside my advisors, I also would like to thank many other faculty members at OSU. Particularly, I would like to thank Dr. Yuan F. Zheng. I have collaborated with Dr. Zheng many times and he has given me a lot of invaluable advice during research. I have learned plenty of knowledge in Computer Vision from Dr. Zheng and my research vision has been broadened. Besides, I need to thank Dr. Kannan Srinivasan for his incisive comments during my candidacy exam. I also thank Dr. Wei Zhao from The University of Macau for his help and insightful advice in my research. It is a great honor to work with Dr. Zhao. In addition, I would like to thank my collaborators, especially, Dr. Boying Zhang, Dr. Jin Teng, Dr. Xinfeng Li, Dr. Junda Zhu, Dr. Adam C. Champion, Dr. Gang Li, Dr. Fan Yang, Dr. Sihao Ding, Guoxing Chen, Cheng Zhang, Ying Li, Xingya Zhao and Quanyi Hu. Without their selfless help, it would be impossible for me to vi finish all the research work in this dissertation. I also would like to thank Qijing Shen for his assistance during my PhD defense. Finally, I am very grateful for the unconditional love and support from my wife and my parents. I cannot finish my Ph.D. study without you. I love you. vii Vita April 9, 1988 . Born - Baotou, China 2011 . .B.S. Information Security, Shanghai Jiao Tong University, Shanghai, China 2011-present . .PhD Candidate, Computer Science and Engineering, The Ohio State University Publications Research Publications - Conference Gang Li (co-primary author), Fan Yang (co-primary author), Guoxing Chen (co- primary author), Qiang Zhai (co-primary author), Xinfeng Li, Jin Teng, Junda Zhu, Dong Xuan, Biao Chen and Wei Zhao. \EV-Matching: Bridging Large Visual Data and Electronic Data for Efficient Surveillance". in Proc. of IEEE International Conference on Distributed Computing Systems (ICDCS), June 2017. Qiang Zhai, Fan Yang, Adam C. Champion, Chunyi Peng, Junda Zhu, Dong Xuan, Biao Chen and Wei Zhao. \S-Mirror: Mirroring Sensing Signals for Mobile Robots in Indoor Environments". in Proc. of IEEE International Conference on Mobile Ad-hoc and Sensor Networks (MSN), December 2016. Fan Yang, Qiang Zhai, Guoxing Chen, Adam C. Champion, Junda Zhu and Dong Xuan. \Flash-Loc: Flashing Mobile Phones for Accurate Indoor Localization". in Proc. of IEEE International Conference on Computer Communications (INFO- COM), April 2016. Qiang Zhai, Sihao Ding, Xinfeng Li, Fan Yang, Jin Teng, Junda Zhu, Dong Xuan, Yuan F. Zheng and Wei Zhao. \VM-Tracking: Visual-Motion Sensing Integration viii for Real-time Human Tracking". in Proc. of IEEE International Conference on Computer Communications (INFOCOM), April 2015. Ying Li, Sihao Ding, Qiang Zhai, Yuan F. Zheng and Dong Xuan. \Human Feet Tracking Guided by Locomotion Model". in Proc. of IEEE International Conference on Robotics and Automation (ICRA), May 2015. Sihao Ding, Qiang Zhai, Yuan F. Zheng and Dong Xuan. \Side-view Face Authen- tication Based on Wavelet and Random Forest with Subsets". in Proc. of IEEE International Conference on Intelligence and Security Informatics (ISI), June 2013. Xinfeng Li, Jin Teng, Qiang Zhai, Junda Zhu, Dong Xuan, Yuan F. Zheng and Wei Zhao. \EV-Human: Human Localization via Visual Estimation of Body Electronic Interference". in Proc. of the mini-conference conjunction with IEEE International Conference on Computer Communications (INFOCOM), April 2013. Adam C. Champion, Xinfeng Li, Qiang Zhai, Jin Teng and Dong Xuan. \Enclave: Promoting Unobtrusive and Secure Mobile Communications with a Ubiquitous Elec- tronic World". in Proc. of the International Conference on Wireless Algorithms, Systems, and Applications (WASA), August 2012. (Best Paper Runner-up) Research Publications - Journal Sihao Ding, Gang Li, Ying Li, Xinfeng Li, Qiang Zhai, Adam C.