University of Connecticut OpenCommons@UConn Doctoral Dissertations University of Connecticut Graduate School 8-24-2018 Three-Dimensional Image Sensing and Visualization with Augmented Reality Xin Shen University of Connecticut - Storrs, [email protected] Follow this and additional works at: https://opencommons.uconn.edu/dissertations Recommended Citation Shen, Xin, "Three-Dimensional Image Sensing and Visualization with Augmented Reality" (2018). Doctoral Dissertations. 1892. https://opencommons.uconn.edu/dissertations/1892 Three-Dimensional Image Sensing and Visualization with Augmented Reality Xin Shen, Ph.D University of Connecticut, 2018 Abstract In recent decades, there have been significant technological advancements in sensors, devices, materials, algorithms, and computational hardware, resulting in extensive improvements for visualization capabilities applied to real world objects. Among these improvements, three-dimensional (3D) imaging technologies have received interest from many research groups and may offer advantages over conventional two-dimensional (2D) approaches. In comparison with 2D sensing techniques, which record only the intensity of the scene, passive 3D imaging also includes depth and directional information. Many techniques for 3D imaging have been proposed, such as holography and interferometry, time-of-flight, two-view based stereoscopy, and multi-view techniques for autostereoscopic 3D imaging, to cite a few. This dissertation will focus on novel aspects of integral imaging based multi-view 3D imaging systems, 3D information processing and visualization in three separate parts. In the first part, two concepts for integral imaging based dynamic 3D imaging are presented. Xin Shen, University of Connecticut, 2018 First, an extended depth-of-focus 3D micro display is presented by using a bifocal liquid crystal lens. Second, a head tracking 3D display is presented by means of proper application of the smart pseudoscopic-to-orthoscopic conversion (SPOC) algorithm. In the second part, novel 3D imaging systems and 3D image processing approaches are proposed. First, the recent progress of integral imaging based Multidimensional Optical Sensing and Imaging Systems (MOSIS) is presented for object recognition, material inspection, and integrated visualization, etc. Second, 3D profilometric reconstruction using flexible sensing integral imaging with automatic occlusion removal is presented. Third, spatial-temporal human gesture recognition under degraded conditions using 3D integral imaging is presented. In the third part, approaches for 3D sensing and visualization with Augmented Reality (AR) are presented. First, an AR based approach for optical visualization and object recognition using 3D axially distributed sensing (ADS) is presented. Second, we present an eye fatigue-free 3D augmented display using lenslet based integral imaging. Lastly, a dynamic 3D imaging system based optical see-through augmented reality display with enhanced depth range of a 3D augmented image is presented to reduce the accommodation- convergence mismatch problem. Three-Dimensional Image Sensing and Visualization with Augmented Reality Xin Shen B.S., Xidian University, Xi’an, China, 2010 M.S., Xidian University, Xi’an, China, 2013 M.S., Doshisha University, Kyoto, Japan, 2013 M.S., University of Connecticut, Connecticut, USA, 2016 A Dissertation Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy at the University of Connecticut 2018 i Copyright by Xin Shen 2018 ii APPROVAL PAGE Doctor of Philosophy Dissertation Three-Dimensional Image Sensing and Visualization with Augmented Reality Presented by Xin Shen, B.S., M.S. Major Advisor ______________________________________________ Bahram Javidi Associate Advisor ______________________________________________ Rajeev Bansal Associate Advisor ______________________________________________ Helena Silva University of Connecticut 2018 iii ACKNOWLEDGMENTS While I am writing this dissertation, the memory of my first week starting this Ph.D program looks like it just happened yesterday. Five years have passed since the beginning of the program, and over that time I have gained knowledge and wisdom along the way which I will always be thankful. I would not be able to write this dissertation without the encouragement, guidance, and collaborations of my advisors, colleagues, family and friends. My deepest gratitude goes first and foremost to my advisor, Prof. Bahram Javidi, for his constant encouragement and guidance. I am very fortunate and proud to have joined Prof. Javid’s group for my academic study and research. His insightful vision and foresight guided me to navigate the research field deeply. I was deeply inspired by his rigorous and earnest research attitude. His collaborations with worldwide outstanding researchers and scientists help broaden my horizon. With his illuminating instruction, I am able to keep improving. I would like to extend my sincerest appreciations to my committee members, Prof. Rajeev Bansal, Prof. Helena Silva, Prof. John Chandy, and Prof. Monty Escabi. Their advice and support helped my research and study at UConn a lot. I would also like to extend my gratitude to Prof. Artur Carnicer, Prof. Pedro Latorre Carmona, Prof. Arun Anand, and Prof. José Manuel Rodríguez Ramos for their guidance and help during their visits at UConn. I will always remember the valuable discussions with them on my study, research, and concerns. I also enjoyed conversations with Prof. Manuel Martinez-Corral, Prof. Yi- iv Bai Huang, Prof. Adrian Stern, Prof. Hua Hong and Prof. Myungjin Cho in international conferences. I wish to truly thank my current lab mates, Adam Markman, Satoru Komatsu, Siddharth Rawat, Hee-Seung Kim, Timothy O'Connor, and Hyun-Woo Kim for their hard work, collaboration, and help. We have built a great lab culture to support each other and improve together. Thanks to my former colleagues, Xiao Xiao, Jingang Wang, Kaleel Mahmood, and Gang Yao for their guidance, which made me be engaged into my research quickly. I would like to have a thank to my international research mates: Yu-Jen Wang and Simon Jen. I could not forget the summer and winter during their visit at UConn. With a limited period, we collaborated efficiently to achieve our goals. I thank other visiting scholars, Alba Peinado Capdevila, Prof. Xiaoxi Chen, Chen Yang, Zhiyuan Shen, Faliu Yi, Juan Trujillo, and all the friends I have met at UConn, who made my life colorful. I am indebted to my parents, Ling Shen and Lianxiang Wang. None of this would be possible without their unconditional encouragement and support. Although we just had very limited time to get together for the past years, I can always feel their love deeply from my heart. This dissertation is also dedicated to all my beloved family, my dear grandparents who I am always missing, my aunts & uncles, cousins, and my lovely niece. They have always been there for me and I am thankful to have their continuous love which makes me keep going. v Table of Contents List of Figures................................................................................................................ ix List of Tables .................................................................................................................xv Part I ................................................................................................................................1 Three-Dimensional Imaging and Dynamic Three-Dimensional Optical Display .....1 Chapter 1 Overview of Three-dimensional Integral Imaging Technologies .............2 1.1 Introduction .........................................................................................................2 1.2 Principle of 3D Integral Imaging .........................................................................3 1.2.1 Pickup Stage of Integral Imaging ...............................................................4 1.2.2 Reconstruction Stage of Integral Imaging ..................................................8 1.3 Main Characteristics of 3D Integral Imaging Display ......................................13 1.3.1 Display Modes of Integral Imaging ..........................................................13 1.3.2 Depth of Focus and Spatial Resolution ....................................................14 1.3.3 Viewing Angle .........................................................................................15 1.3.4 Viewing Quality .......................................................................................17 1.4 Organization of Thesis ......................................................................................19 Chapter 2 Extended Depth-of-focus Three-Dimensional Micro Integral Imaging Display using a Bifocal Liquid Crystal Lens ..............................................................21 2.1 Introduction .......................................................................................................21 2.2 Operating Principle of the Micro Integral Imaging Display with Liquid Crystal Lens .........................................................................................................................22 2.3 Experimental Results .........................................................................................27 2.4 Conclusion .........................................................................................................32 Chapter 3 Head Tracking Three-dimensional Integral Imaging Display using Smart Pseudoscopic-to-orthoscopic