Visual Tracking with an Application to Augmented Reality DISSERTATION Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By Changlin Xiao Graduate Program in Civil Engineering The Ohio State University 2017 Dissertation Committee: Dr. Alper Yilmaz, Advisor Dr. Charles Toth Dr. Rongjun Qin Copyrighted by Changlin Xiao 2017 Abstract In simple terms, visual tracking is the process of visually following a given object. This very initial task is one of the fundamental problems in many computer vision applications, like in movement pattern analysis, animal surveillance, robot navigation, and so on. Currently, with the increasing popularity of cameras, large video data are generated every day. However, the algorithms that used to handle this visual information are far from being enough. Besides, the technical progress and increasing demands for unmanned aerial vehicle (UAV) with the automatic pilot capability promote the requirement of visual tracking in many practical applications. Therefore, visual tracking is still one of the most interesting topics in computer vision tasks. However, even various methods have been proposed for visual tracking, it is still an open problem due to its complexity. Tracking means following the target’s motion, and the primary challenge in visual tracking is the inconsistency of target’s appearance. The status of the target may be changed with the illumination change, the non-rigid motion, and occlusions. Additionally, the similar background may cause the drift problem like switching the targeted player to the untargeted ones in the games. In real life, there are more problems such as the scale change, fast motion, lower resolution and out of plane rotation which cause the tracking tasks even more challenging. Therefore, visual tracking, after several decades’ research, is still an active research topic with many unsolved problems. ii In this dissertation, three tracking methods are proposed trying to deal with the tracking problems for different targets and various scenarios. Also, besides the tracking in 2D images, this work further introduces a 3D space tracking model in augmented reality application. For a simple tracking scenario, an efficient tracking method with distinctive color and silhouette is proposed. The proposed method uses colors that most exist on the target to represent and track it. It is a dynamic color representation for the target which is updating with the background changes. This appearance model can substantially reduce the distractors in the background, and the color is constant to the shape change which significantly alleviates the nonrigid deformation problem. Based on the above tracking idea, a unique feature vote tracking algorithm is further developed. This work divides the feature space into many small spaces as storage cells for feature descriptions. And if most of the descriptions in the cell are from the target, the features in the cell are treated as unique features. Besides counting how likely the feature from the target, each feature’s location respect to the target center is recorded to reproject the center in the new coming frames. This voting machine makes the tracker focus on the target against the occlusion and cluster background. Recently, deep learning and neural network show powerful ability in computer vision applications. The neural network, especially the convolutional neural network has been successfully used in object recognition, detection, and tracking. This work uses a pre- trained network that has learned high-level semantic features to represent the target as a iii concept model. This concept is a combination of these high-level features that learned from myriads of objects. With the concept, the network can generate a hot map of the new coming frame that shows the possible distribution of the target. Finally, a Siamese network is used to locate the target location. The high-level semantic features are robust to general appearance changes and can retrieve the target in many complicated scenarios. Besides 2D tracking, 3D space tracking is more useful in many applications. To demonstrate that, this work uses a stereo camera to form a space tracking system for the surgical scalpel. Three LED lights are attached to the top of the scalpel to help the tracking of its tip. After the registration between cameras, operation table, and the augmented reality device, the scalpel’s motion can be displayed in the space by the augmented reality device. This holographic display is advantageous in the surgical operation for education and navigation. The localization experiment also shows the accuracy of the 3D space tracking. In summary, these research efforts have offered different methods for visual tracking at various scenarios. It also demonstrates the visual tracking’s usefulness in practical application that both in 2D and 3D space. iv Acknowledgments It’s an honor to study at the Ohio State University and these years are the most precious time of my life. I would like to thank my Ph.D. advisor, Dr. Alper Yilmaz. He is the most important and kindly person in my academic life. He is not just an advisor but also a mentor, not only shows me how to do research but also shows me how to think about the world. Also, my thanks go to Dr. Rongxing Li who showed me the attitude to learning and work. I am also fortunate to learn from many excellent professors like Dr. Alan Saalfeld, Dr. Jekeli, Dr. Toth and Dr. Qin. Besides, thanks to Dr. Young Jin Lee, Dr. Siavash HosseinyAlamdary, Shirui Li, Nima Ajam Gard, Yujia Zhang, M. Taha Koroglu, and Michael Karnes, the many peers who we had many discussions and happy times. Finally, I would like to thank my family, the source of power and motivation for my Ph.D. study. v Vita 2009................................................................B.S. of Measuring, Testing Technologies and Instruments, University of Electronic Science & Technology of China 2012................................................................M.E. of Measurement Technologies and Instruments, University of Electronic Science & Technology of China 2012 to present ..............................................Graduate Student, Department of Civil, Environmental, and Geodetic Engineering, The Ohio State University Publications Xiao Changlin, Peng Liu*, Zeshu Zhang, Junqi Ma, Jian Gao, Alper Yilmaz, Ronald Xu, “A wearable augmented reality navigation system for surgical telementoring based on Microsoft HoloLens”, Submitted to Annals of Biomedical Engineering, 2017. equal contribution vi Xiao, Changlin, Alper Yilmaz, “A Unique Target Representation and Voting Mechanism for visual tracking.” In processing of IEEE International Conference on Image Processing (ICIP), 2017. Xiao, Changlin, Alper Yilmaz, Shirui Li, “Visual Tracking Utilizing Object Concept from Deep Learning Network.” In processing of ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2017 Xiao, Changlin, and Alper Yilmaz. "Efficient tracking with distinctive target colors and silhouette." In Pattern Recognition (ICPR), 2016 23rd International Conference on, pp. 2728-2733. IEEE, 2016. Fields of Study Major Field: Civil Engineering Geodetic and Geoinformation Engineering vii Table of Contents Abstract ............................................................................................................................... ii Acknowledgments............................................................................................................... v Vita ..................................................................................................................................... vi Table of Contents ............................................................................................................. viii List of Tables ................................................................................................................... xiii List of Figures .................................................................................................................. xiv Chapter 1: Visual Tracking ................................................................................................. 1 1.1 Motivation ................................................................................................................. 2 1.2 Research Scope ......................................................................................................... 3 1.3 Contribution .............................................................................................................. 4 1.4 Dissertation Structure ................................................................................................ 5 Chapter 2: Related Work .................................................................................................... 7 2.1 Overview of Visual Tracking .................................................................................... 7 2.2 Visual Tracking in Images ........................................................................................ 9 2.2.1 Generative Tracking with Appearance Representation ...................................... 9 viii 2.2.2 Tracking with Discriminative Information ....................................................... 12 2.2.3 Tracking Based on Neural Network ................................................................. 14 2.3 Visual Tracking in Space .......................................................................................